Skip to content

feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction - #411

Open
LeeroyHannigan wants to merge 22 commits into
mainfrom
feat/partiql
Open

LeeroyHannigan wants to merge 22 commits into
mainfrom
feat/partiql

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

This PR adds the three PartiQL operations: ExecuteStatement, BatchExecuteStatement and ExecuteTransaction. They run SELECT, INSERT, UPDATE and DELETE statements, and EXISTS checks inside a transaction. The server authorizes each statement by itself, as Amazon DynamoDB does.

The engine translates each statement into operations that it already has. A statement runs as the point reads, Queries, Scans and item writes that the service makes for it, through the existing handlers. The PR changes no storage backend and no handler of a classic operation.

  • crates/core/src/partiql/ holds the lexer, the parser, and the planner that maps a WHERE to key reads.
  • crates/engine/src/partiql/ holds the executors and the code that authorizes each statement.
  • Both modules compile only with the default-on partiql Cargo feature. With the feature off, the three operations answer UnknownOperationException, as on main.
  • Outside the two modules and the tests, the PR changes 17 files, with 468 lines added and 6 lines deleted. The commit messages name them.

Reads

  • A SELECT reads by key when every disjunct of its WHERE names a key. Any other SELECT scans the table or the index.
  • A read by key splits into at most 50 disjuncts. A page visits at most 10 of them, in the order that the service uses. A page stops at Limit items or at 1 MB of item bytes read.
  • An index read takes one Query for each disjunct.
  • ORDER BY, projections, positional parameters and the NextToken follow the rules and messages of the service.

Writes, batch and transaction

  • INSERT creates an item and answers DuplicateItemException for a key that exists. UPDATE and DELETE need the full key. The other terms of the WHERE become the condition expression.
  • The service sizes an UPDATE by its finished item alone, and sizes a standalone UpdateItem by its statement. At a three-byte key an UPDATE writes an item of exactly 409,600 bytes, while UpdateItem stops 17 bytes short. The UPDATE executor asks the storage layer for the finished item rule.
  • BatchExecuteStatement takes 1 to 25 statements, all reads or all writes, with one response entry for each statement.
  • ExecuteTransaction takes 1 to 100 statements and runs all of them or none. It gives positional cancellation reasons and honours ClientRequestToken.

Authorization

  • The server authorizes each statement with dynamodb:PartiQLSelect, PartiQLInsert, PartiQLUpdate or PartiQLDelete. The resource is the table, or the index alone for a read of an index. The operation itself needs no action.
  • The engine asks an ItemAuthorizer in its OperationContext. The server's authorizer evaluates the caller's policies with check_authorization, and allows every access in developer mode. Without an authorizer, the engine refuses every statement.
  • The engine authorizes each disjunct of a read by key with its own dynamodb:LeadingKeys value. A refused statement in a batch gets its own AccessDenied entry. A refused transaction member cancels the transaction.
  • The condition keys LeadingKeys, FullTableScan, EnclosingOperation, Attributes, Select and ReturnValues hold the values that the service gives them.
  • Each check stays on the side of authorization where the service makes it. So a caller with no permission gets AccessDeniedException for a missing table, a missing index or a key of the wrong type.

Differences from DynamoDB

docs/differences-from-dynamodb.md records one difference: ExtendDB does not throttle a PartiQL request by table. The throttle reads the top-level TableName of a request, and a PartiQL request names its table in the statement.

Known gaps

The measurements do not cover these orderings for a caller that has no permission. In each case the code follows the measured refusals beside it, and the module documentation names the choice.

  • In a read transaction, ExtendDB refuses a member that reads an index, or whose WHERE lacks the full primary key, before authorization.
  • ExtendDB refuses two batch statements on one item before it authorizes them.
  • A page visits only a disjunct that the policy refuses, while another held-back disjunct fails a check. That page gets the refusal of the check.
  • A statement fails an ORDER BY check or has a key of the wrong type, and also has overlapping disjuncts or more than 50 disjuncts. ExtendDB authorizes that statement first.
  • ExtendDB refuses a NextToken that another statement gave before it authorizes the statement.

The measurements do not cover these orderings for a caller with permission either:

  • A WHERE that does not read by key, against a key value of the wrong type.
  • On an index, a path into a key attribute against the ORDER BY refusals.
  • Two maps in one statement that each repeat a name. ExtendDB reports the map whose repeated name comes first.

The refusal message for a role session differs from the service's message. ExtendDB names the session with an arn:aws:iam:: ARN and says that no identity-based policy allows the action. The service uses an arn:aws:sts:: ARN and names the session policy. Every operation uses this message, so one test stays a strict expected failure until the shared code changes.

The NextToken is not signed. The server authorizes each request that presents a NextToken again, from the statement in that request.

Commits

  1. The parser and the planner, in core.
  2. The executors and the docs, in the engine.
  3. The dual-target wire tests.
  4. The authorization of each statement.
  5. The authorization tests.
  6. The clippy pedantic fixes in the new code.
  7. The item size of an UPDATE.

Each commit message lists the rules that its part implements.

Why

On main, the server answers the three operations with UnknownOperationException, so a client that uses PartiQL cannot run against ExtendDB. The tests expect the values and messages that Amazon DynamoDB gave in us-east-1, measured between 2026-10-19 and 2026-11-03.

Testing done

  • Wire tests: 35 files, tests/test_partiql_*.py, with 1,976 cases.
    • 32 files with 1,919 cases are dual-target. On 2026-11-04 against Amazon DynamoDB, the new tests/test_partiql_item_size.py gave 13 passed. On 2026-11-03 the other 31 files gave 1,902 passed, 3 skipped and 1 failed. The failed case makes an eventually consistent read right after a PutItem, and the service returned no item. The case passed on a rerun, and so did the rest of its file.
    • 3 files with 57 cases test authorization on ExtendDB only, because they make users and roles through the management API.
    • On ExtendDB with SQLite, 1,967 passed, 3 skipped and 6 are expected failures:
      • 4 divergences in code that PartiQL shares with the classic operations, where the fix belongs. They are set_add on a list element (classic ADD), and the 32-level nesting check on 3 parameter values that a statement does not store.
      • The role session message above.
      • The order in which ExtendDB checks the members of a parameter map. The order depends on whether the build includes the MongoDB crate, so this expected failure is not strict.
    • The pytest jobs of CI run the 35 files on PostgreSQL, MongoDB and Cassandra. On MongoDB and Cassandra the cases of tests/test_partiql_item_size.py that need a refusal skip, because those backends size updates differently (MongoDB applies no item size limit to updates, and Cassandra sizes updates on the finished item against a fixed 400 KiB with the PutItem wording #818).
  • The authorization tests on main (test_fgac_leading_keys.py, test_fgac_attributes.py, test_auth_permissions.py and test_authz_resource_arn.py): 143 passed and 1 skipped.
  • The item size and index key tests on main (test_item_size_at_limit.py and test_index_key_validation_on_update.py): 122 passed.
  • Unit tests of the core, engine, server and auth crates: 1,972 passed. The PartiQL unit tests also pass in the build that keeps JSON maps in request order, which includes the MongoDB crate.
  • The test jobs of CI run cargo test --workspace and cargo +1.88.0 check --workspace --all-targets.
  • cargo fmt --all -- --check, cargo +1.97.0 clippy --all-targets -- -D warnings and cargo check --no-default-features of the core and engine crates are clean.
  • Clippy pedantic reports no new warning other than doc_markdown. Its 123 doc_markdown warnings flag names such as PartiQL and NextToken in doc comments, which main also leaves unquoted, with more than 540 such warnings. The sixth commit clears the other 24.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The PR adds the three operations as the service runs them, and authorizes each statement as the service does. docs/differences-from-dynamodb.md records the one difference.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

Adds crates/core/src/partiql, compiled only with the new default-on
`partiql` Cargo feature. It turns a statement string into an AST and
then into a plan made of the operations the engine already has.
Nothing else in core depends on it.

The lexer and parser take SELECT, INSERT, UPDATE (also in its FROM-first
form), DELETE and, inside ExecuteTransaction, EXISTS, with positional
parameters. They refuse what the service refuses, with its messages: a
statement longer than 8192 characters, a WHERE of more than 300
operators, an IN list of more than 100 members, a value nested inside 32
maps or lists, a number with more than 38 significant digits or outside
the service's magnitude range, and reserved words used as names. A
statement that nests deeper than 32 levels is parsed on a thread with a
64 MiB stack, and nesting past the service's limits (1000 groups, 1000
literal levels) is refused.

The planner reads a WHERE the way the service does. OR and IN on key
attributes expand into disjuncts, and each disjunct is a point read, a
Query of one partition, or part of a scan. Overlapping key conditions,
key values of the wrong type and paths into key attributes are refused
in the order the service checks them.

DuplicateItemException is a new error variant. Only a PartiQL INSERT of
a key that already exists returns it.

Each rule and message was measured against Amazon DynamoDB in us-east-1
between 2026-10-19 and 2026-11-03, and the code cites the measurement
where it implements the rule.
…eTransaction

Adds crates/engine/src/partiql behind the same `partiql` feature. It
translates each statement onto the handlers the engine already has (get,
query, scan, put, update and delete, and their batch and transaction
forms) and changes no storage backend and no classic handler. The
dispatch gains three arms and is_known_operation three entries, each
behind the feature. With the feature off the server answers the three
operations with UnknownOperationException as before, and removing
PartiQL is deleting the two partiql modules, those arms and entries, the
DuplicateItemException variant and the feature.

SELECT reads by key when every disjunct of its WHERE is keyed, and scans
the table or index otherwise. A read by key that splits into more than
50 disjuncts is refused with "Too many decomposed read operations for a
given query." A page visits at most 10 disjuncts, the first page in
key order (or the ORDER BY order) and later pages in the order the
service uses, and stops at Limit items or at 1 MB of item bytes read. A
page of a read by key has a LastEvaluatedKey only when it holds Limit
items, and every page reports the capacity of its own reads. An index
read takes one Query for each disjunct. A disjunct that fails a check
made when it is read (a key over the size limit, a key attribute in its
filter) is held back: it takes a visit and reads nothing, and the
request whose page would visit only held-back disjuncts gets its
refusal. The NextToken carries where the next page goes on from and
digests of the statement and its parameters, and a token sent with
another statement or other parameter values is refused.

INSERT creates an item and answers DuplicateItemException for a key that
exists. UPDATE needs the full key, does not create a missing item, and
supports SET (with arithmetic, list_append, set_add and set_delete),
REMOVE and the RETURNING modes. DELETE needs the full key, succeeds on a
missing item and allows RETURNING ALL OLD *. Every other term of a
write's WHERE becomes its condition expression.

BatchExecuteStatement takes 1 to 25 statements, all reads or all
writes, and answers each statement in its own response entry.
ExecuteTransaction takes 1 to 100 statements, all reads or all writes
and EXISTS checks, cancels as a whole with positional cancellation
reasons, and honours ClientRequestToken.

The requests a statement is translated into run with the classic
expression size and token limits raised to 131,072 (16 for each of the
8192 characters a statement may hold), so the translation of a statement
the service accepts is not refused for its length.

docs/differences-from-dynamodb.md lists the three operations as
supported and adds a PartiQL section. The difference it records: a
PartiQL request is not throttled by table, because the throttle reads
the request's top-level TableName and a PartiQL request names its table
in the statement.

Behaviour was measured against Amazon DynamoDB in us-east-1 between
2026-10-19 and 2026-11-03.
Adds 30 files, tests/test_partiql_*.py, with 1,883 test cases. Each case
runs against an ExtendDB server when EXTENDDB_TEST_ENDPOINT is set and
against Amazon DynamoDB otherwise, with the same assertions. Every
expected value was measured on the service in us-east-1, and on
2026-11-03 the whole set ran against the service: 1,880 passed and 3
were skipped.

Where the service gives more than one answer to the same request, the
tests accept each answer it gave: whether a LastEvaluatedKey or an empty
page follows the last item of a partition, which depends on where the
partition sits in storage, and the two forms in which a validation
message echoes a statement list (its fields, or the name of the
service's request object). A test that reads an index first waits until
the index shows every item it wrote.

Expected failures on ExtendDB:
- The order in which parameter map members are checked: request order
  when the build includes the MongoDB crate, name order otherwise. Not
  strict.
- Two divergences in code PartiQL shares with the classic operations,
  to be fixed there: set_add and set_delete on a list element (classic
  ADD and DELETE), and the 32-level nesting check on parameter values a
  statement does not store. Strict.
Until this commit the server authorized the three PartiQL operations
once, before dispatch, with the operation's action against table/*. A
Deny on one table did not apply, a dynamodb:LeadingKeys condition had no
values to check, and a policy that allows dynamodb:PartiQLSelect and the
other PartiQL actions allowed nothing. The service authorizes each
statement by itself, with dynamodb:PartiQLSelect, PartiQLInsert,
PartiQLUpdate or PartiQLDelete against the table it reads or writes, or
against the index alone for a read of an index. It needs no action for
the operation itself, and the classic actions grant nothing.

The engine gains an ItemAuthorizer trait, carried in OperationContext.
The server's implementation evaluates the caller's policies with
check_authorization against this deployment's table or index ARN, with
the request-level condition keys every other operation carries and the
item-level keys of the statement. In developer mode it allows every
access, as authorize_request does for the other operations.
authorize_request no longer authorizes the three operations, and the
engine refuses every statement when its context has no authorizer. The
trait is not behind the partiql feature, and only PartiQL uses it.

Each executor makes its checks in the order a permitted caller gets
them and authorizes the statement before it reads or writes an item.
The checks of the request and the refusals made at parse time come
first, as before. Among the checks of a statement, these also come
before authorization: a parameter shortfall, SELECT COUNT(*),
ConsistentRead on a global secondary index, a path into a key attribute,
overlapping disjuncts, more than 50 disjuncts, a NextToken that does not
decode or that another statement gave, a DELETE RETURNING clause other
than ALL OLD *, a BatchExecuteStatement read without the full primary
key, and an EXISTS member of the wrong form or on an index. Every other
check comes after it, so a caller with no permission gets
AccessDeniedException for a missing table, a missing index, a key of the
wrong type or a duplicate INSERT. A check made before authorization is
answered first only when no check ahead of it fails. Otherwise the
statement is authorized first.

Three refusals are made before authorization by analogy with the
measured refusals beside them, and none is measured for a caller with no
permission: a read transaction member that reads an index, one that
does not name the full primary key, and two batch statements on one
item. The module documentation says so where each is made.

A read by key authorizes each disjunct with its own partition key value,
at most 50 values, the MAX_DECOMPOSED_READS limit. A refused disjunct is
held back like one that fails a check made when it is read: it takes a
visit and reads nothing, and the request whose page would visit only
held-back disjuncts gets AccessDeniedException. Across the disjuncts of
one read, an authorization refusal ranks below the refusals of the other
checks. A refused statement in BatchExecuteStatement is its own response
entry, {"Error": {"Code": "AccessDenied", ...}, "TableName": ...}. A
refused member of ExecuteTransaction cancels the transaction before any
condition is evaluated, with AccessDenied for each refused member and
None for the others.

The condition keys:
- dynamodb:LeadingKeys holds the partition key value of each disjunct or
  written item. A number is compared in the form Java's new
  BigDecimal(s).toString() prints for the text the statement or the
  parameter gives it (NumberForm::Decimal): under ForAllValues ["10"],
  pk = 10 and pk = 010 run and pk = 10.0 is refused.
- dynamodb:FullTableScan is "true" for a scan and "false" for a read by
  key in ExecuteStatement, and absent in BatchExecuteStatement,
  ExecuteTransaction and every write. RequestParams in the auth crate
  gains full_table_scan to carry "false".
- dynamodb:EnclosingOperation is BatchExecuteStatement or
  ExecuteTransaction inside those operations, and absent in
  ExecuteStatement.
- dynamodb:Attributes holds the table's key attributes and every
  top-level name the statement uses: projection, WHERE, SET values,
  REMOVE targets and conditions.
- dynamodb:Select is ALL_ATTRIBUTES for * on a table,
  ALL_PROJECTED_ATTRIBUTES for * on an index, and SPECIFIC_ATTRIBUTES
  for a projection list.
- dynamodb:ReturnValues is NONE for a write without RETURNING and
  follows RETURNING otherwise.
An EXISTS member is authorized as a read. An index read of a missing
table is authorized against the table ARN.

Two refinements outside authorization, measured with a permitted caller:
- A map literal that names a key twice is refused at parse time with
  "Duplicate keys in tuple. Key: <name> under <path> at
  <line>:<column>:<length>". At the top level of an INSERT VALUE the
  position is the second occurrence of the name, with the length of the
  quoted name. In any other map it is that map's opening brace, with
  length 1. Before this commit ExtendDB accepted the literal and kept the
  last value. common::insert_values is the one place that reads an
  INSERT's values, for authorization, the write and the duplicate check
  of a batch or transaction.
- A SELECT makes its key checks in the service's order: a path into a
  key attribute, an ORDER BY term that is not a key attribute, the key
  values, the hash-key ORDER BY rule, overlapping disjuncts, and then
  the number of disjuncts. ExtendDB checked the key values and overlap
  first, so 8 measured statements got another refusal, for example
  (pk = 'a' OR pk = 'a') ORDER BY n.

Outside the two partiql modules this commit changes 10 files, with 346
lines added and 3 removed:
- crates/engine/src/item_authorizer.rs, new: the trait and ItemAccess,
  with the field and re-export in crates/engine/src/lib.rs.
- crates/server/src/item_authz.rs, new: the server's authorizer, with
  its module line in crates/server/src/lib.rs and its construction in
  crates/server/src/handler.rs.
- crates/server/src/request_helpers.rs: authorize_request returns early
  for the three operations.
- crates/auth/src/policy/context.rs: RequestParams.full_table_scan, with
  a unit test.
- crates/server/src/leading_keys.rs: extract_typed_value is pub(crate).
- crates/engine/Cargo.toml and Cargo.lock: extenddb-storage-sqlite as a
  dev-dependency, for the unit tests that build an OperationContext.

Removing PartiQL now also means deleting item_authz.rs and its lines in
handler.rs and lib.rs, the early return in authorize_request, and the
trait module, field and re-export in the engine. full_table_scan and the
visibility of extract_typed_value change no other operation and can
stay.

Each rule was measured against Amazon DynamoDB in us-east-1 on
2026-11-03.
Adds ExtendDB-only wire tests for the authorization of each statement,
and dual-target tests for the two refinements of the previous commit.

The authorization tests, 57 cases in three files, are ExtendDB-only.
They create users, roles and policies through the management API, so
they need the builtin auth provider and the admin login in
EXTENDDB_ADMIN_USER and EXTENDDB_ADMIN_PASSWORD. Each test gives an IAM
user the permissions the measured role session had, in its identity
policy, and expects the answer the service gave that session on
2026-11-03.
- test_partiql_authorization.py: the action of each statement type,
  table and index resources, the answers of the three operations,
  dynamodb:LeadingKeys for each disjunct and each write,
  dynamodb:FullTableScan, dynamodb:EnclosingOperation and session
  policies. Two tests send every statement path, and a NextToken a
  permitted caller got, from a caller without a PartiQL action.
- test_partiql_authorization_keys.py: EXISTS members,
  dynamodb:ReturnValues, dynamodb:Attributes, dynamodb:Select, number
  partition keys, writes without dynamodb:FullTableScan, and an index
  read of attributes the index does not hold.
- test_partiql_authorization_order.py: the checks on each side of
  authorization, and the pages of a read by key with refused disjuncts.
- partiql_authz_helpers.py holds the principals, policies and refusal
  checks the three files share.

One case is a strict expected failure: the refusal of a role session.
The shared authorization message names the session with an
arn:aws:iam:: ARN and says that no identity-based policy allows the
action. The service names it with an arn:aws:sts:: ARN and names the
session policy. Every operation uses that message, so the fix belongs
there.

The dual-target cases, 23 in two files, ran against Amazon DynamoDB in
us-east-1 on 2026-11-03, and all 23 passed.
- test_partiql_repeated_names.py: a map literal that names a key twice,
  at the top level of a VALUE, in a nested map, a list, a SET value and
  a WHERE literal. Also names that differ only in case, a syntax error
  after the value, and the refusal in BatchExecuteStatement and
  ExecuteTransaction.
- test_partiql_order_by.py, 12 new cases: the ORDER BY refusals among
  the key checks, with ORDER BY on a key and on another attribute.
`cargo clippy -- -W clippy::pedantic` reported 144 warnings in the code
this PR adds that main does not have. This commit clears the 24 that are
not doc_markdown, with no change in behaviour:
- single_match_else, manual_let_else, if_not_else, match_same_arms,
  needless_for_each and many_single_char_names take the form clippy
  suggests.
- next_prefix in plan/ranges.rs returns Option<Vec<u8>> instead of
  Option<Option<Bound>>, and the BeginsWith arm maps the result back to
  the type of its bound.
- shape_response and write::reshape return DispatchResult instead of a
  Result that was always Ok. The internal helpers of base_join.rs and
  returning.rs take the serde_json error by reference.
- detect_duplicate_keys keeps a HashSet instead of a HashMap with unit
  values.
- The server boxes the check_authorization future, as large_futures
  suggests.
- split in plan/split.rs moves the cross product of an AND into its own
  function. lex, batch::run and select::run keep their length, and
  Parser keeps its six flags, each with an allow and its reason.

The 120 doc_markdown warnings stay. They flag names such as PartiQL and
NextToken in doc comments, which main also leaves unquoted: main has 542
such warnings.
Since #416, the SQLite and PostgreSQL storage layers size a standalone
UpdateItem by its statement: a base of 3, 18 per SET clause, and the
written paths with their values, checked against the limit beside the
finished item. A PartiQL UPDATE runs as an UpdateItem, so it inherited
that rule. At a three-byte key it refused a finished item of 409,600
bytes, which the service writes.

The service sizes a PartiQL UPDATE by its finished item alone, as it
sizes a transacted Update (measured in us-east-1 on 2026-11-04):
- UPDATE ... SET p = ? writes a finished item of exactly 409,600 bytes
  at a key where UpdateItem with SET p = :p stops 17 bytes short.
- One byte more is refused with "Item size to update has exceeded the
  maximum allowed size".
- A second SET clause or an if_not_exists adds nothing to the size.
- BatchExecuteStatement gives the refusal as the statement's
  ValidationError entry. ExecuteTransaction cancels with a
  ValidationError reason with the same message.
- A WHERE term that fails, and the implicit check that the item exists,
  are answered first: an UPDATE over the limit that would not run gets
  ConditionalCheckFailedException.

ExpressionMaps gains finished_item_sizing. When it is set,
validate_update_item_size checks only the finished item, as
validate_transact_update_item_size does. handle_update_item now calls
handle_update_item_with(body, ctx, false), and the PartiQL UPDATE
executor calls handle_update_item_with with true, in ExecuteStatement
and BatchExecuteStatement. ExecuteTransaction already ran through the
transacted Update and its sizing.

Outside the two partiql modules this commit changes:
- crates/core/src/expression/resolver.rs: the finished_item_sizing field.
- crates/core/src/validation/mod.rs: the branch in
  validate_update_item_size, with a unit test.
- crates/engine/src/update_item.rs: handle_update_item_with.
- docs/differences-from-dynamodb.md: the Item Size section names the
  PartiQL UPDATE beside the transacted Update.
Removing PartiQL now also means deleting the field, the branch and
handle_update_item_with.

tests/test_partiql_item_size.py adds 13 dual-target cases. All 13 passed
against the service on 2026-11-04. The cases that need a refusal skip
on MongoDB and Cassandra, whose update sizing differs from the service's,
as docs/differences-from-dynamodb.md records.
@navareamol navareamol added the release-blocker Must ship for its milestone's release label Nov 4, 2026
@navareamol navareamol added this to the 1.0 milestone Nov 4, 2026
@hitesh96db hitesh96db self-assigned this Nov 5, 2026

@hitesh96db hitesh96db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How these differences were measured

Every inline comment in this review reports results from one experiment, described here once.

When and where. On 2026-11-05 the same requests were sent to two endpoints:

  • Amazon DynamoDB in us-east-1
  • this PR at af0fac76, built with the default PostgreSQL backend and served at https://127.0.0.1:18443

The requests were sent with boto3, with client retries off, so each case is exactly one request. The commands in the comments are the AWS CLI form of those requests. The same command was run against both endpoints; for ExtendDB it adds --endpoint-url https://127.0.0.1:18443.

Table. Both endpoints had the same table, created with CreateTable and seeded with PutItem:

  • table partiql_probe_t: hash key pk (S), range key sk (S), on-demand
  • GSI gsi: hash key g (S), range key gs (S), projection ALL
  • seed items {pk, sk, a: 1, b: 2, x: 0, m: {k: 1, 'my key': 1}, g: <pk>, gs: '1'} for (pk, sk) = ('a','b'), ('a','c') (with gs: '2') and ('b','b')

Write cases. Each write case first puts a fresh item of the same shape under its own pk with sk = 'b' (an INSERT case deletes it instead). After the request it reads that item back with a consistent GetItem, which is where "stored" and "unchanged" in the comments come from.

Names. The comments write the table as t. Positions inside error messages (at line:column:length) are shifted to match the shorter name. Every such position is on line 1, so the shift is exact.

How to read the comments. Each experiment is labelled:

  • [Difference]: DynamoDB and ExtendDB answered differently. The suggestion is to change ExtendDB to match DynamoDB and add the experiment as a dual-target test.
  • [Same]: both answered the same, usually a control next to a difference. No code change is needed, but the experiment is worth adding as a dual-target test so the behaviour stays pinned.

Every experiment ends with that suggestion. All of them could go into the existing tests/test_partiql_*.py files, which already run against both targets.

Batch and transaction requests. These are written as a req.json here-doc followed by aws dynamodb batch-execute-statement --statements file://req.json or aws dynamodb execute-transaction --transact-statements file://req.json.

Some(KeyEquality::Partition(value)) => (&mut pk_value, value, pk),
Some(KeyEquality::Sort(value)) => (&mut sk_value, value, sk.unwrap_or_default()),
None => {
residual.push(term);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A path into a key attribute in a write's WHERE becomes part of the condition; DynamoDB refuses the statement.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: DELETE with a map lookup on the hash key

aws dynamodb execute-statement --statement "DELETE FROM t WHERE pk = 'k1' AND sk = 'b' AND pk.c = 1"
Endpoint Result
DynamoDB ValidationException Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key pk; item unchanged
ExtendDB ConditionalCheckFailedException The conditional request failed; item unchanged

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: UPDATE with a list index on the range key

aws dynamodb execute-statement --statement "UPDATE t SET x = 1 WHERE pk = 'k2' AND sk = 'b' AND sk[0] = 'z'"
Endpoint Result
DynamoDB ValidationException Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key sk; item unchanged
ExtendDB ConditionalCheckFailedException The conditional request failed; item unchanged

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

SELECT already gives the DynamoDB refusal. In the write path the term goes to residual here and becomes part of the condition expression.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now refuses both statements before it checks the condition, with DynamoDB's message: the DELETE with pk.c = 1 gets "Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key pk", the UPDATE with sk[0] = 'z' gets the same message naming sk, and the item is unchanged. A path into a key in a condition that does not compare it with values, such as NOT pk.c = 1, gets the message DynamoDB gives at its later check, with "Key:" before the name, after the key type checks and also for an item that does not exist. Fixed in 095190c. The dual-target tests are test_a_write_refuses_a_path_into_a_key_tested_against_values and test_a_write_refuses_a_path_into_a_key_in_its_condition in tests/test_partiql_key_conditions.py.

pub fn check_number(text: &str) -> Result<(), NumberProblem> {
let decimal = read_decimal(text).ok_or(NumberProblem::Unconvertible)?;
let significant = decimal.digits.trim_start_matches('0');
if significant.is_empty() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Negative-zero decimals are stored as 0; DynamoDB refuses them, in values and in WHERE.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: INSERT a negative-zero decimal

aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z1','sk':'b','x':-0.0}"
Endpoint Result
DynamoDB ValidationException Negative zero decimal is not supported under key root at 1:46:3 (the position is the 0.0 after the sign)
ExtendDB OK; GetItem returns x = 0

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: UPDATE SET with an exponent form

aws dynamodb execute-statement --statement "UPDATE t SET x = -0e0 WHERE pk = 'z2' AND sk = 'b'"
Endpoint Result
DynamoDB ValidationException Negative zero decimal is not supported under key root at 1:19:3
ExtendDB OK; GetItem returns x = 0

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: negative zero inside a list

aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z4','sk':'b','x':[-0.0]}"
Endpoint Result
DynamoDB ValidationException Negative zero decimal is not supported under key root[0] at 1:47:3
ExtendDB OK; GetItem returns x = [0]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: negative zero in WHERE

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND x = -0.0"
Endpoint Result
DynamoDB ValidationException Negative zero decimal is not supported under key root at 1:41:3
ExtendDB OK, 2 items

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Same]: an integer zero with a sign

aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z3','sk':'b','x':-0}"
Endpoint Result
DynamoDB OK; GetItem returns x = 0
ExtendDB OK; GetItem returns x = 0

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

The location in DynamoDB's message names the value (root, root[0]), not the attribute.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now refuses a negative zero decimal literal with DynamoDB's message and position: -0.0 in the INSERT, -0e0 in the SET and -0.0 in the WHERE get "Negative zero decimal is not supported under key root" at the number after the sign, and [-0.0] names root[0]. Two minus signs that cancel and a sign before parentheses, as in - -0.0 and -(0.0), are refused too, as DynamoDB refuses them. The integer -0 and a parameter whose N value is -0.0 are still stored as 0. Fixed in eb92f2f. The dual-target tests are test_a_negative_zero_decimal_in_an_insert_is_refused, test_a_negative_zero_decimal_in_a_set_or_a_condition_is_refused and test_an_integer_zero_and_a_negative_zero_parameter_are_zero in tests/test_partiql_grammar.py.

}
}
TokenKind::QuotedIdent(_) => Ok(Operand::Path(self.path()?)),
_ => Err(self.term_error()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One value or path in parentheses is refused here; DynamoDB treats the parentheses as grouping and runs the statement.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: a value in parentheses

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = ('a')"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Statement wasn't well formed, can't be processed: Unexpected term

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: a key path in parentheses

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE (pk) = 'a'"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Statement wasn't well formed, can't be processed: Expected a comparison operator

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: a parameter in parentheses

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = (?)" --parameters '[{"S":"a"}]'
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Statement wasn't well formed, can't be processed: Unexpected term

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: a non-key path in parentheses

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND (a) = 1"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Statement wasn't well formed, can't be processed: Expected a comparison operator

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Difference]: a signed number in parentheses in SET

aws dynamodb execute-statement --statement "UPDATE t SET x = -(5) WHERE pk = 'p4' AND sk = 'b'"
Endpoint Result
DynamoDB OK; GetItem returns x = -5
ExtendDB ValidationException Statement wasn't well formed, can't be processed: Unexpected term; x stays 0

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now reads one value, parameter or path in parentheses as grouping, as DynamoDB does: pk = ('a'), (pk) = 'a', pk = (?) and pk = 'a' AND (a) = 1 each return the 2 items, and SET x = -(5) stores -5. A comma inside the parentheses makes a list, and an operand alone in parentheses keeps the refusal it gets without them. Fixed in b156d29, and 90aea5f gives DynamoDB's syntax error when a token that does not continue the operand follows it, as in (a b), which gets "Expected RIGHT_PAREN" in place of "Expected a comparison operator". The dual-target tests are test_parentheses_around_one_value_or_path_group_it, test_a_parameter_in_parentheses_and_a_key_in_parentheses_read_by_key, test_a_set_value_in_parentheses and test_a_token_after_an_operand_ends_it in tests/test_partiql_grammar.py.

// anything else that is not a name, after a dot.
_ => return Err(ParseError::well_formed("Invalid path dot component")),
}
} else if self.eat(&TokenKind::LBracket) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A string or name subscript is refused here; DynamoDB reads m['k'] and m[k] as the map step m.k.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: a string subscript in WHERE

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m['k'] = 1"
Endpoint Result
DynamoDB OK, 2 items (same as m.k = 1)
ExtendDB ValidationException Unexpected path component at 1:38:3

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: a string subscript that is not a valid bare name

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m['my key'] = 1"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Unexpected path component at 1:38:8

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: a bare name in brackets

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m[k] = 1"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB ValidationException Unexpected path component at 1:38:1

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: a string subscript as a SET target

aws dynamodb execute-statement --statement "UPDATE t SET m['k'] = 5 WHERE pk = 's6' AND sk = 'b'"
Endpoint Result
DynamoDB OK; GetItem returns m = {k: 5, 'my key': 1}
ExtendDB ValidationException Unexpected path component at 1:16:3; item unchanged

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Difference]: an empty string subscript

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m[''] = 1"
Endpoint Result
DynamoDB ValidationException Path component cannot be an empty string
ExtendDB ValidationException Unexpected path component at 1:38:2

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 6 [Same]: the dotted form

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m.k = 1"
Endpoint Result
DynamoDB OK, 2 items
ExtendDB OK, 2 items

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now reads a string or a name in brackets as a map member, as DynamoDB does: m['k'] = 1, m['my key'] = 1 and m[k] = 1 return the same 2 items as m.k = 1, SET m['k'] = 5 sets the member k, and m[''] gets "Path component cannot be an empty string". Other contents of brackets keep the refusals DynamoDB gives them. Fixed in 94e6936. The dual-target tests are test_a_string_or_a_name_in_brackets_is_a_map_member, test_a_member_in_brackets_in_an_update and test_other_contents_of_brackets_are_refused in tests/test_partiql_grammar.py.

table: bool,
value_of: &dyn Fn(&Operand) -> Option<AttributeValue>,
) -> Result<(), DynamoDbError> {
let Some(order) = order.filter(|_| table) else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ORDER BY on an index skips the ORDER BY checks, and a path term isn't checked on a table either. DynamoDB refuses all three.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: ORDER BY on an index with no WHERE

aws dynamodb execute-statement --statement 'SELECT * FROM "t"."gsi" ORDER BY g'
Endpoint Result
DynamoDB ValidationException Must have WHERE clause in the statement when using ORDER BY clause.
ExtendDB OK, scans the index and returns its items

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: ORDER BY on an index without an equality on the index hash key

aws dynamodb execute-statement --statement "SELECT * FROM \"t\".\"gsi\" WHERE g > 'a' ORDER BY g"
Endpoint Result
DynamoDB ValidationException Must have at least one non-optional hash key condition in WHERE clause when using ORDER BY clause.
ExtendDB OK, returns items

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: ORDER BY a path on the table

aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' ORDER BY m.x"
Endpoint Result
DynamoDB ValidationException Argument to ORDER BY must be of type VariableReference. Type: PATH at 1:41:1 (the position of m)
ExtendDB OK, 2 items

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Same]: ORDER BY on an index with the hash key

aws dynamodb execute-statement --statement "SELECT * FROM \"t\".\"gsi\" WHERE g = 'a' ORDER BY gs DESC"
Endpoint Result
DynamoDB OK
ExtendDB OK

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

Experiment 5 [Same]: ORDER BY on the table with no WHERE

aws dynamodb execute-statement --statement "SELECT * FROM t ORDER BY pk"
Endpoint Result
DynamoDB ValidationException Must have WHERE clause in the statement when using ORDER BY clause.
ExtendDB ValidationException Must have WHERE clause in the statement when using ORDER BY clause.

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

The table path already gives the first message; filter(|_| table) here drops the checks for an index.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now makes the ORDER BY checks on an index as on a table, with DynamoDB's messages: the index read with no WHERE gets "Must have WHERE clause in the statement when using ORDER BY clause.", the one with g > 'a' gets "Must have at least one non-optional hash key condition in WHERE clause when using ORDER BY clause.", and ORDER BY m.x gets "Argument to ORDER BY must be of type VariableReference. Type: PATH" at the position of m, on a table and on an index. Fixed in c8c51ed. The dual-target tests are test_an_order_by_on_an_index_needs_a_where_that_reads_it_by_key, test_a_path_in_the_order_by_list_is_refused_at_its_first_token and test_a_path_in_the_order_by_list_of_an_index_read in tests/test_partiql_order_by.py.

Comment thread crates/engine/src/partiql/batch.rs Outdated
}

// Two statements naming the same item fail the whole request.
detect_duplicate_keys(&parsed, &key_infos, is_write_batch)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duplicate-key check counts a statement that failed validation, so the whole batch is refused. On DynamoDB the failed statement gets its own error and the other statement runs.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: an INSERT with an unbound ? and a DELETE of the same key (item cb3 exists beforehand; no Parameters)

cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO t VALUE {'pk':'cb3','sk':'b','x':?}"
  },
  {
    "Statement": "DELETE FROM t WHERE pk = 'cb3' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json
Endpoint Result
DynamoDB HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Number of parameters in request and statement don't match."}}, {TableName: t}]; GetItem afterwards: cb3 was deleted
ExtendDB ValidationException Provided list of item keys contains duplicates; GetItem afterwards: cb3 still there

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Same]: two well-formed statements on one item

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'bwc' AND sk = 'b'"
  },
  {
    "Statement": "DELETE FROM t WHERE pk = 'bwc' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json
Endpoint Result
DynamoDB ValidationException Provided list of item keys contains duplicates
ExtendDB ValidationException Provided list of item keys contains duplicates

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c7e67c, the same change as the mix check. Only a statement that translates and passes the checks of its request counts toward the duplicate refusal now, so the INSERT with the unbound ? gets its own error and the DELETE runs. An UPDATE of a key attribute, or an index key of the wrong type, is left out too, and two writes that reach the item are still duplicates even when the first one's condition fails. The tests are test_a_write_refused_before_its_item_is_not_a_duplicate and test_two_writes_that_reach_one_item_are_duplicates in tests/test_partiql_batch.py.

other => Self {
code: "ValidationError",
message: other.message().to_owned(),
include_table_name: false,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A per-statement ValidationError in a batch has no TableName; DynamoDB includes it when the error comes from running the statement.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: an UPDATE that sets a key attribute

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET pk = 'z' WHERE pk = 'cb4' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json
Endpoint Result
DynamoDB Responses: [{TableName: t, Error: {Code: ValidationError, Message: "One or more parameter values were invalid: Cannot update attribute pk. This attribute is part of the key"}}]
ExtendDB Responses: [{Error: {Code: ValidationError, Message: "One or more parameter values were invalid: Cannot update attribute pk. This attribute is part of the key"}}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Same]: an error raised while reading the statement (a ? with no Parameters)

cat > req.json <<'EOF'
[
  {
    "Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
  },
  {
    "Statement": "SELECT * FROM t WHERE pk = ? AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json
Endpoint Result
DynamoDB the second entry is {Error: {...}} with no TableName
ExtendDB the second entry is {Error: {...}} with no TableName

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c7e67c. An error the item operation raises now carries TableName: your key attribute update, a REMOVE of a key attribute, an index key of the wrong type, an operand of the wrong type and a path the item doesn't have. Errors from reading or translating the statement still carry none, the unbound ? included. The tests are test_an_error_of_the_item_operation_names_the_table and test_an_error_of_reading_the_statement_names_no_table in tests/test_partiql_batch.py.

// A statement of another type is left to the executors.
Some(_) => return,
};
let constraint = if statement.is_empty() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A statement of only whitespace is read as a syntax error; DynamoDB refuses it as an empty statement.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: BatchExecuteStatement with a whitespace-only statement

aws dynamodb batch-execute-statement --statements '[{"Statement":"   "}]'
Endpoint Result
DynamoDB HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "The statement was empty."}}]
ExtendDB HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Statement wasn't well formed, can't be processed: Unexpected term"}}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: ExecuteTransaction with a whitespace-only statement

aws dynamodb execute-transaction --transact-statements '[{"Statement":"   "}]'
Endpoint Result
DynamoDB ValidationException ExecuteTransaction can not have empty statement. (no TransactStatements[0] prefix)
ExtendDB ValidationException Validation failed in TransactStatements[0]: Statement wasn't well formed, can't be processed: Unexpected term

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

The check here only looks for an empty string.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ExtendDB now refuses a statement of only whitespace as an empty statement, as DynamoDB does: in BatchExecuteStatement its response is the ValidationError "The statement was empty.", and ExecuteTransaction gets "ExecuteTransaction can not have empty statement." with no TransactStatements prefix. The whitespace characters are those DynamoDB treats as whitespace, and the tests cover tab, newline, U+001F, U+2028 and U+3000 among them. Fixed in a80f39e. The dual-target tests are test_a_blank_statement_in_a_batch_fails_that_statement and test_a_blank_statement_in_a_transaction_fails_the_request in tests/test_partiql_requests.py.

}
}

let reads = parsed.iter().filter(|(_, s)| s.is_read()).count();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a transaction has several faults, ExtendDB reports a different one than DynamoDB. ExtendDB parses every member, then checks RETURNING on every member, then the read/write mix, then each member's form. DynamoDB checks each member fully, in order, and stops at the first member that fails, so a fault in an earlier member wins over any fault in a later one.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: read, write, then a syntax error

cat > req.json <<'EOF'
[
  {
    "Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
  },
  {
    "Statement": "INSERT INTO t VALUE {'pk':'ct5','sk':'b','a':1}"
  },
  {
    "Statement": "SELECT * FROM"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException ExecuteTransaction API does not support both read and write operations in the same request.
ExtendDB ValidationException Validation failed in TransactStatements[2]: Statement wasn't well formed, can't be processed: Unexpected term

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: a RETURNING clause, then a syntax error

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET x = 1 WHERE pk = 'ct6' AND sk = 'b' RETURNING ALL NEW *"
  },
  {
    "Statement": "SELECT * FROM"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: RETURNING clause is not supported in ExecuteTransaction.
ExtendDB ValidationException Validation failed in TransactStatements[1]: Statement wasn't well formed, can't be processed: Unexpected term

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: an index read, then a write

cat > req.json <<'EOF'
[
  {
    "Statement": "SELECT * FROM \"t\".\"gsi\" WHERE g = 'a' AND gs = '1'"
  },
  {
    "Statement": "INSERT INTO t VALUE {'pk':'ct7','sk':'b','a':1}"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: Reads on indices are not supported within transactions.
ExtendDB ValidationException ExecuteTransaction API does not support both read and write operations in the same request.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: a read without the sort key, then a read of a missing table

cat > req.json <<'EOF'
[
  {
    "Statement": "SELECT * FROM t WHERE pk = 'a'"
  },
  {
    "Statement": "SELECT * FROM nosuch_tbl WHERE pk = 'a' AND sk = 'b'"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: Select statements within ExecuteTransaction must specify the primary key in the where clause.
ExtendDB TransactionCanceledException, CancellationReasons: [{Code: None}, {Code: ResourceNotFound, Message: "Requested resource not found"}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Difference]: an EXISTS with only the key, then a read

cat > req.json <<'EOF'
[
  {
    "Statement": "EXISTS(SELECT * FROM t WHERE pk = 'a' AND sk = 'b')"
  },
  {
    "Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: EXISTS() must contain a single item read with additional condition
ExtendDB ValidationException ExecuteTransaction API does not support both read and write operations in the same request.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 6 [Difference]: an EXISTS on an index of a missing table

cat > req.json <<'EOF'
[
  {
    "Statement": "EXISTS(SELECT * FROM \"nosuch_tbl\".\"idx\" WHERE pk = 'a' AND x = 1)"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: Operations on indices are not supported in transactions
ExtendDB TransactionCanceledException, CancellationReasons: [{Code: ResourceNotFound, Message: "Requested resource not found"}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 7 [Difference]: a read, then an INSERT with an unbound ? (no Parameters)

cat > req.json <<'EOF'
[
  {
    "Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
  },
  {
    "Statement": "INSERT INTO t VALUE {'pk':'ct11','sk':?}"
  }
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[1]: Number of parameters in request and statement don't match.
ExtendDB ValidationException ExecuteTransaction API does not support both read and write operations in the same request.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Where one of these refusals was measured on its own (the read/write mix, RETURNING, a partial key, a key-only EXISTS, a syntax error), the message is the same on both sides.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 26695e3. The executor checks each member in order and stops at the first that fails: its statement and parameters, then its form against its table, then the read and write mix, with a missing table left to cancel the request at the end. The seven cases match DynamoDB now, along with 19 more orderings measured on 5 Nov, and a read transaction with a missing table and a wrong key type cancels with both reasons. The tests are test_the_first_member_that_fails_is_reported and test_a_read_cancellation_gives_every_member_its_reason in tests/test_partiql_transaction.py.

Comment thread crates/engine/src/partiql/batch.rs Outdated
obj.insert("TableName".to_owned(), Value::String(table.clone()));
obj.insert("CapacityUnits".to_owned(), json!(units));
if let Some(tu) = table_units {
obj.insert("Table".to_owned(), json!({ "CapacityUnits": tu }));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ConsumedCapacity leaves out GSI writes, and a batch leaves out the units of a failed write.

Setup: see the experiment described in the review summary.

Experiment 1 [Difference]: BatchExecuteStatement, INDEXES, one UPDATE that changes the GSI key

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET g = 'x' WHERE pk = 'cb17' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity INDEXES --statements file://req.json
Endpoint Result
DynamoDB [{TableName: t, CapacityUnits: 3.0, Table: {CapacityUnits: 1.0}, GlobalSecondaryIndexes: {gsi: {CapacityUnits: 2.0}}}]
ExtendDB [{TableName: t, CapacityUnits: 3.0, Table: {CapacityUnits: 1.0}}] (no GlobalSecondaryIndexes)

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: ExecuteTransaction, INDEXES, one UPDATE that changes the GSI key

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET g = 'x' WHERE pk = 'ct17' AND sk = 'b'"
  }
]
EOF
aws dynamodb execute-transaction --return-consumed-capacity INDEXES --transact-statements file://req.json
Endpoint Result
DynamoDB [{TableName: t, CapacityUnits: 4.0, WriteCapacityUnits: 4.0, Table: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}, GlobalSecondaryIndexes: {gsi: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}}}]
ExtendDB [{TableName: t, CapacityUnits: 2.0, WriteCapacityUnits: 2.0, Table: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: ExecuteTransaction, TOTAL, two UPDATEs of a (projected into the GSI)

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'twr' AND sk = 'b'"
  },
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'tws' AND sk = 'b'"
  }
]
EOF
aws dynamodb execute-transaction --return-consumed-capacity TOTAL --transact-statements file://req.json
Endpoint Result
DynamoDB [{TableName: t, CapacityUnits: 6.0, WriteCapacityUnits: 6.0}]
ExtendDB [{TableName: t, CapacityUnits: 4.0, WriteCapacityUnits: 4.0}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: BatchExecuteStatement, TOTAL, one UPDATE that succeeds and one on an absent item (fails its condition)

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'cb15' AND sk = 'b'"
  },
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'cb16' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity TOTAL --statements file://req.json
Endpoint Result
DynamoDB [{TableName: t, CapacityUnits: 3.0}]
ExtendDB [{TableName: t, CapacityUnits: 2.0}]

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Same]: BatchExecuteStatement, TOTAL, two UPDATEs that both succeed

cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'bwh' AND sk = 'b'"
  },
  {
    "Statement": "UPDATE t SET a = 9 WHERE pk = 'bwi' AND sk = 'b'"
  }
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity TOTAL --statements file://req.json
Endpoint Result
DynamoDB [{TableName: t, CapacityUnits: 4.0}]
ExtendDB [{TableName: t, CapacityUnits: 4.0}]

Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.

The batch merge here keeps Table but drops GlobalSecondaryIndexes (experiment 1), and only sums statements that succeeded (experiment 4). The transaction numbers (experiments 2 and 3) presumably come from the shared TransactWriteItems path; that wasn't measured separately here.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The batch part is fixed in 251490d. The batch merge keeps GlobalSecondaryIndexes and LocalSecondaryIndexes, and a write that fails at its item adds the stored item's units, 1 when it's missing, to its table's CapacityUnits, so experiments 1, 4 and 5 match; those units only count when another statement on the same table succeeded, as on DynamoDB. The transaction numbers come from TransactWriteItems, which reports no index units for the classic API either, because the engine has no old or new item for an Update or Delete member. Matching that needs the storage layer to return those items on all four backends, so it's in #866, and experiments 2 and 3 are held as strict expected failures in test_a_transaction_reports_its_index_units and test_a_transaction_total_counts_its_index_units.

crates/engine/Cargo.toml: main added futures to the engine's
dependencies. This branch added a dev-dependency on the SQLite storage
crate and the default-on partiql feature. The merged file keeps all
three.

Cargo.lock: the merged file is main's, with extenddb-storage-sqlite
added to the dependencies of extenddb-engine for this branch's
dev-dependency.

crates/engine/src/partiql/snapshot.rs and
crates/engine/src/partiql/test_context.rs: main added the backup
runtime to OperationContext. The context a batch or a transaction runs
in now carries the request's runtime, and the unit test context holds a
runtime of one slot with no configured store.
The service translates each statement of a BatchExecuteStatement into
the read or the write it runs as, then sorts the statements that
translate into reads and writes. A statement whose translation fails
gets its own error entry and counts neither as a read nor as a write,
nor in the check for two statements on one item: a missing table or
index, a key value of the wrong type, an INSERT without a key attribute,
a write whose WHERE does not give the full key, a ? with no parameter,
and a DELETE returning clause other than RETURNING ALL OLD *. A
statement that translates and then fails the check of its request counts
as a read or a write, but not in the duplicate check: a read that is not
of one item by its full primary key, an UPDATE that sets or removes a
key attribute, and a write that gives a secondary index key a value of
the wrong type. The batch counted every statement that parsed, so one
statement that failed refused the whole request with the mixed read and
write refusal or the duplicate key refusal.

The errors of the item operation a statement runs as now carry the
statement's TableName: an UPDATE of a key attribute, a secondary index
key of the wrong type, an update the item cannot take (an operand of
the wrong type, a document path the item does not have), a failed
condition and a duplicate key. The errors of reading and translating a
statement carry none. Before, a key value past its size limit, a failed
condition and a duplicate key were the only errors that named the table.

Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests
are the new cases in tests/test_partiql_batch.py, on a table with a
global secondary index from tests/partiql_gsi_table.py.
A BatchExecuteStatement sums the ConsumedCapacity of its statements per
table. At INDEXES the sum now keeps the units of each global and local
secondary index, as it kept the Table units. The batch's per-table sum
dropped GlobalSecondaryIndexes and LocalSecondaryIndexes, so an UPDATE
that moved an item in its index reported a total of 3 with a Table arm
of 1 and no index arm.

A write that fails at its item is charged the write units of the item
stored under its key, or one unit when there is none: a failed
condition, a duplicate INSERT, and an update the item cannot take (an
operand of the wrong type, a document path the item does not have).
Those units count in the CapacityUnits of the write's table alone, and
only when another statement on that table succeeded; a table on which
every write failed reports nothing. An UPDATE of a key attribute and a
secondary index key of the wrong type are refused before the item is
read and are not charged. The batch counted the statements that
succeeded alone, so an UPDATE of a missing item beside one that
succeeded reported 2 where the service reports 3.

ExecuteTransaction still reports no index units. ExtendDB's
TransactWriteItems has no old or new item of an Update or Delete
member to rate an index write from, for the classic operation as for
PartiQL. Two tests in tests/test_partiql_transaction.py hold the
service's answer as expected failures against ExtendDB.

Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests
are the ConsumedCapacity cases in tests/test_partiql_batch.py and the
unit tests of the capacity list.
The service checks the members of an ExecuteTransaction in order and
stops at the first one that fails, so a refusal of a member wins over a
refusal of any later member. For each member it reads and parses the
statement, checks its parameters, then checks the member against its
table, and then refuses a read beside a write. The executor parsed
every member, then checked every RETURNING clause, then the mix of
reads and writes, then the form of each member, so a syntax error in a
later member, or the mix, could win over a refusal of an earlier
member.

The checks of a member against its table, each with the member's
position: a read of an index, existing or not, then a read whose WHERE
does not give the full primary key; an EXISTS through an index, ahead
of a missing table, then an EXISTS that reads one item with no further
condition; a RETURNING clause. A member on a missing table gets none of
them and cancels the request once every member is checked. A `?` with
no parameter is checked with the statement, ahead of the mix.

In a read transaction a missing table and a key value of the wrong type
now cancel the request together, each member with its own reason, in
either order. The missing table was reported alone, with None for the
other members.

Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests
are test_the_first_member_that_fails_is_reported and
test_a_read_cancellation_gives_every_member_its_reason in
tests/test_partiql_transaction.py.
A read member of an ExecuteTransaction returned the whole item whatever
its projection list named: the TransactGetItems member it was
translated into carried only the table and the key. The executor now
reads each item whole and gives it the member's projection, keeping the
shape of each path as a ProjectionExpression gives it, so SELECT a, m.k
returns {a, m: {k}}. An item that has none of the paths is returned
empty, and a member whose item is missing returns no item. ExtendDB's
TransactGetItems leaves out an item that its projection leaves empty,
where the service returns it empty in a PartiQL transaction, so the
projection is applied here.

A projection list that names an output identifier twice, or whose paths
overlap, is refused on all three operations, the identifier refusal
first. A transaction refuses it with the member's position, with the
checks of the statement, ahead of a missing table, a partial key and an
index read. A batch statement gets it as its own error and counts as
neither a read nor a write. ExecuteStatement refused a repeated
identifier and now also refuses two paths that overlap, which it
returned items for. The overlap message names the later path first, as
an UPDATE's does, and the two share the check.

Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests
are the projection cases in tests/test_partiql_transaction.py and
tests/test_partiql_batch.py, and the unit tests of check_projection.
A BatchExecuteStatement member whose statement holds only whitespace now
gets the ValidationError "The statement was empty." in its entry, and an
ExecuteTransaction with such a member fails with "ExecuteTransaction can
not have empty statement." and no member position. The check comes
before the member's parameters are converted and before its statement is
parsed, so a transaction reports a syntax error of an earlier member
first. ExecuteStatement keeps its syntax error for such a statement.

The whitespace set follows the service: tab to carriage return, the
information separators, space, the line and paragraph separators and the
Unicode spaces other than the no-break ones. A comment, U+0085 and
U+00A0 are not whitespace here. The dual-target tests in
tests/test_partiql_requests.py pin each case.
An ORDER BY on an index skipped the checks of the WHERE it reads, so a
read of an index with no WHERE, or with no equality on the index
partition key, ran as a scan. The index now gets the refusals a table
gets: no WHERE, a WHERE that does not read by key, and more than one
partition key value without the partition key in the list.

A path with steps in the list, such as m.x, m[0] or pk.x, was not
checked and the statement ran. It is now refused with "Argument to ORDER
BY must be of type VariableReference. Type: PATH at" the position of the
path's first token. The parser keeps that position on each term. The
terms are checked in list order together with the rule that a name must
be a key attribute.

A statement with no WHERE is refused for it ahead of the list, so ORDER
BY n with no WHERE gets the missing WHERE refusal on a table and on an
index. The dual-target tests in tests/test_partiql_order_by.py pin each
case.
A path into a key attribute in the WHERE of an UPDATE or a DELETE, such
as pk.c = 1 or sk[0] = 'z', went into the condition the write carries,
so the statement failed its condition or ran. The service refuses the
statement in two places, and ExtendDB now does the same.

A condition that tests the path against values alone (a comparison with
a value on either side, BETWEEN, IN with values, begins_with, contains,
attribute_type, an IS test of a type, or one of these compared with a
boolean) is refused while the WHERE is read for the key, with "Key
attributes must be scalars; ...: Key pk". The refusal and the one of two
values for one key attribute come in statement order, ahead of the full
key and the key types. A transaction member reports the key schema
mismatch reason for it, and an EXISTS check fails the request naming its
position.

Any other condition with a path into a key attribute (OR, NOT, a
function of the path, IS MISSING, IS NOT, the path compared with a
path) is refused after the key types and after the refusal of a SET of a
key attribute, also for an item that does not exist. Its message has
"Key:" before the name, which is the sort key when a path into it is
among the conditions. A batch entry for it names the table, and a
transaction member has it as its cancellation reason.

The dual-target tests in tests/test_partiql_key_conditions.py pin each
form.
A path step in brackets took only a number, so m['k'] and m[k] were
refused with "Unexpected path component". A string, a quoted name or a
name in brackets is now a map member, as after a dot: m['k'], m["k"]
and m[k] are m.k in a condition, a SET target, a REMOVE, a SET value and
the select list, and m['my key'] names a member a dot cannot. An empty
string gets "Path component cannot be an empty string".

Other contents of brackets parse and are refused once the statement
parses, at their first token: a parameter, a number that is not a whole
number, a literal word and a path. A minus before a whole number is
refused as an index out of range. A reserved word in brackets is a
syntax error, and so is a bracket left open, which now reads "Expected
RIGHT_BRACKET" with the syntax error prefix.

The steps after a path root are read by a function of their own, which
the next change uses for a path in parentheses. The dual-target tests
in tests/test_partiql_grammar.py pin each case.
Parentheses around one operand were refused: pk = ('a') and pk = (?)
got "Unexpected term", and (pk) = 'a' got "Expected a comparison
operator". They now only group what they hold, as the service reads
them.

In an operand position, (x) is x: a value, a parameter, a path, a call
or an arithmetic operand. A path in parentheses takes the steps after
them, so (m).k is m.k, and a comma of their own makes a list, so SET x =
('a', 'b') stores a list. Signs before a number fold through
parentheses: SET x = -(5) stores -5 and a = -(-1) compares with 1. A
sign before anything else in parentheses keeps its refusal. INSERT
values and literal elements take parentheses the same way.

Where a condition starts, a group that holds one operand and nothing
else is that operand, and the predicate goes on after the group:
(pk) = 'a' reads by key, and (a) BETWEEN 0 AND 2, (a) IN [1] and NOT (a)
= 2 run. An operand alone in parentheses is refused as the operand alone
is. That refusal now also applies inside a group and after a NOT, as the
service gives it: (a AND b = 1) is refused for the arguments of AND, and
NOT a gets "The argument to NOT must be a Type or Expression". Before
anything else such an operand keeps its syntax error.

The dual-target tests in tests/test_partiql_grammar.py pin each case.
A zero written with a decimal point or an exponent and a minus sign
before it, such as -0.0, -0e0 or -.0, was stored or compared as 0. The
service refuses such a literal with "Negative zero decimal is not
supported under key root at 1:46:3", which names where the number sits
in its value (root, root[0], root.y) and the span of the number after
its signs. ExtendDB now gives the same answer.

The parser reads the signs before a number literal, also through
parentheses, so -(0.0), (-0.0) and +-0.0 are refused, and two minus
signs that cancel (- -0.0) keep the refusal. A zero written as an
integer (-0), +0.0 and a parameter whose N value is -0.0 are still
stored as 0, as the service stores them.

The refusal comes when the statement is read, before the table is
resolved and before the key type checks. In a condition, the walk meets
it where it meets a number that breaks an item number rule. In an
INSERT value or a SET clause it comes in the order the statement is
written with those numbers and with a parameter inside a literal: for
{'x':1e126,'y':-0.0} the overflow is reported, and for
{'x':{'b':-0.0,'a':1e126}} the zero. It comes ahead of a sign before a
SET value that is not a number, and after what the condition meets.

The dual-target tests in tests/test_partiql_grammar.py pin each case.
The rustfmt check of toolchain 1.97.0 failed on eight files that the six
commits before this one change. This commit applies rustfmt to them,
with no change in behaviour.
An operand in a condition followed by a token other than a comparison,
AND, OR, the end of the statement or a ) that closes a group got
"Expected a comparison operator", which the service does not answer.
The service ends the operand there and reads the token in the part of
the statement around the operand, and ExtendDB now does the same.

WHERE a b and WHERE pk = 'a' AND a b get "Unexpected token after
expression", in a SELECT and in a DELETE. (a b) and (pk = 'a' AND a
LIMIT 1) get "Expected RIGHT_PAREN". WHERE a LIMIT 1 is refused for the
LIMIT clause at its value, and WHERE a ORDER BY pk is refused for the
operand with "WHERE clause must contain an expression that resolves to
a boolean". A syntax error after the operand still comes ahead of the
operand's own refusal. A size call with no comparison after it keeps
"Expected a comparison operator", which is not measured.

The dual-target tests in tests/test_partiql_grammar.py pin each form.
crates/engine/src/partiql/common.rs: feat/partiql renamed
StatementError::from_key_error to from_item_error, which names the
table for every error of the item operation a statement runs as, and
removed is_key_size_error. The parser fixes added the refusal of a path
into a key attribute among a write's other conditions to
from_key_error. The result keeps from_item_error and drops that
addition. The merged batch answers that refusal once the statement is
authorized, through from_item_error, so the entry names the table, and
test_a_batch_names_the_table_only_for_a_path_in_the_condition passes.

crates/engine/src/partiql/common/tests.rs: feat/partiql renamed the key
size test to an_error_of_the_item_operation_names_the_batch_table. The
parser fixes added three tests beside it. The result keeps the renamed
test and the tests of where_key and condition_key_refusal, and drops
the test of from_key_error together with that function.

crates/engine/src/partiql/transaction.rs: feat/partiql moved the read
and the parse of each member into parse_member, so each member is
checked in full before the next one is parsed. The parser fixes refused
a statement of only whitespace in the old parse loop. The result makes
that refusal the first check in parse_member, ahead of the member's
parameters and its parse. DynamoDB reports an earlier member that fails
its checks against its table, and a read and a write in one request,
ahead of the blank member, and the result does the same.

crates/engine/src/partiql/batch/translation.rs: an UPDATE or a DELETE
with a path into a key attribute among its other conditions gets that
refusal from its translation, after the refusals of the UpdateItem
request. Such a statement counts as a write and stays out of the check
for two statements on one item, which is how DynamoDB counts it.
Without this change the merged code refused such a statement and a
second write of the same item as duplicates.

Tests: dual-target cases in tests/test_partiql_batch.py,
tests/test_partiql_key_conditions.py and
tests/test_partiql_transaction.py check these answers, each measured
against DynamoDB: the two batch classes of a path into a key attribute,
a blank member or a negative zero after a member that fails, an EXISTS
with a path into a key attribute ahead of a later refusal, an UPDATE
with such a path beside a SELECT without the full key, and a map member
in brackets in a projection list.
ExecuteStatement refused an UPDATE that sets a secondary index key to a
value of the wrong type for a path into a key attribute among the
WHERE's other conditions, such as AND NOT pk.c = 1, when the statement
had both. DynamoDB refuses it for the index key type first, as it does
for a SET or REMOVE of a key attribute, whether or not the item exists,
and BatchExecuteStatement answers the same way. run_update now calls
update_refusal in place of targets_a_key, and leaves the statement to
UpdateItem when update_refusal finds either refusal. An index key given
the value of another attribute is checked on the written item, so the
path refusal still comes first for it.

update_refusal and insert_refusal move from batch/translation.rs to
write/update.rs and write/insert.rs, with index_keys and from_wire, and
take the translated request instead of building it. The batch
translation calls them and answers as before.

In ExecuteTransaction an INSERT or UPDATE member that gives an index key
a value of the wrong type had no reason of its own. Its refusal came
from TransactWriteItems, which the request reaches only when no member
has a reason, and which names the first such member alone. DynamoDB
gives each such member the index key refusal as its reason, beside the
reasons of the other members, and an UPDATE gets it after a SET of a
key attribute and ahead of a path into a key attribute. member_reason
now gives it in that order, through update_index_key_refusal and
insert_refusal.

Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The
dual-target tests are in tests/test_partiql_key_conditions.py,
tests/test_partiql_transaction.py and tests/test_partiql_batch.py.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@hitesh96db, each of your 12 threads has a reply with the commit and the tests that answer it. The branch is at c745a3c: main merged at 38992eb, the twelve fixes, and one follow-up described below. Every fix has dual-target tests that pass against DynamoDB in us-east-1 and against ExtendDB.

The follow-up came from combining the fixes. ExecuteStatement now answers UPDATE ... SET g = 1 WHERE pk = 'k' AND sk = 'b' AND NOT pk.c = 1, where g is a string index key, with DynamoDB's index key type error in place of the key path refusal. The refusals now come in DynamoDB's order. A SET or REMOVE of a key attribute is first. An index key with a value of the wrong type is next, and a path into a key attribute among the other conditions is last. BatchExecuteStatement already gave this order. In ExecuteTransaction, an INSERT or UPDATE member that gives an index key a value of the wrong type now has that refusal as its cancellation reason. The other members keep their own reasons beside it.

All 35 tests/test_partiql_*.py files against a local SQLite server: 2390 passed, 3 skipped, 8 xfailed, with no change of outcome for an existing test. fmt, the CI clippy command and the engine library tests (447) pass. CI runs the other backends.

These answers still differ from DynamoDB. Each answered the same way at af0fac7, and each is a separate change:

  • An empty string key value (pk = ''). DynamoDB answers "One or more key attribute values are not valid: The AttributeValue for a key attribute cannot contain an empty string value." ExtendDB gives the classic parameter message, and its ExecuteStatement SELECT returns no items.
  • A key compared with a list, a map or a boolean, such as pk = ('a', 'b'). DynamoDB answers "Key value must be of type S, N, or B", and ExtendDB gives the schema type message.
  • The select list in parentheses (SELECT (a)), a comparison in parentheses as an operand (a = (b = 1)), and an Ion literal in brackets. DynamoDB runs these, and ExtendDB refuses them.
  • U+00A0 or U+0085 alone as a statement. DynamoDB answers "Statement wasn't well formed, can't be processed: " with nothing after the colon, and ExtendDB adds "Unexpected term".
  • In ExecuteTransaction, DynamoDB gives a key past its size limit and an INSERT's empty index key value as member reasons beside the others. ExtendDB leaves both to TransactWriteItems.
  • The index units of a transaction's ConsumedCapacity: TransactWriteItems leaves index write units out of ConsumedCapacity #866. The classic TransactWriteItems has the same gap, and two strict xfail tests hold DynamoDB's answer.

#867 is a related gap in the classic TransactGetItems, found while measuring the transaction projections.

let params = &req.parameters;
let refusal = match stmt {
Statement::Update(update) => {
let parts = write::build_update_parts(ctx, update, params).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Regression from 6c7e67c1: one write the batch can't build now fails the whole batch, so the good statements around it never run.

Setup: see the review summary (table, seed item, endpoints).

This line, and update_refusal/build_insert_parts/insert_refusal next to it, run for every write before any statement executes. Each returns its error with ?, which fails the whole request. Before 6c7e67c1, each write ran through write::run, and its error became that statement's own entry. DynamoDB keeps it per statement:

Experiment 1 [Difference]: SET a = size(m) between two good INSERTs

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
  },
  {
    "Statement": "UPDATE partiql_probe_sf SET a = size(m) WHERE pk = 'h' AND sk = 's'"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: size"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written.
ExtendDB request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: SET a = attribute_exists(m) between two good INSERTs

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
  },
  {
    "Statement": "UPDATE partiql_probe_sf SET a = attribute_exists(m) WHERE pk = 'h' AND sk = 's'"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: attribute_exists"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written.
ExtendDB request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: SET a = begins_with(m, 'x') between two good INSERTs

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
  },
  {
    "Statement": "UPDATE partiql_probe_sf SET a = begins_with(m, 'x') WHERE pk = 'h' AND sk = 's'"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: begins_with"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written.
ExtendDB request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1, :v". Both get-item calls return no item: neither INSERT was written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: SET a = upper(m) between two good INSERTs

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
  },
  {
    "Statement": "UPDATE partiql_probe_sf SET a = upper(m) WHERE pk = 'h' AND sk = 's'"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "Unrecognized function: upper at 1:33:5"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written.
ExtendDB request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 5 [Difference]: the same UPDATE through ExecuteStatement (the message behind it)

aws dynamodb execute-statement --region us-east-1 --statement "UPDATE partiql_probe_sf SET a = size(m) WHERE pk = 'h' AND sk = 's'"
Endpoint Result
DynamoDB ValidationException The function is not allowed in an update expression; function: size
ExtendDB ValidationException Syntax error; token: "(", near: "(#n1)"

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Turning the errors from build_*_parts / *_refusal into that statement's Untranslated/Refused entry would fix experiments 1–4. Experiment 5 is the message underneath, which also shows up in ExecuteStatement.

.or_else(|| condition_key_refusal(update.where_clause.as_ref(), key_info, params))
}
Statement::Insert(insert) => {
let parts = write::build_insert_parts(ctx, insert, params).await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Regression from 6c7e67c1: an empty set (<<>>) in a batch write is an InternalServerError, and nothing in the batch runs.

Setup: see the review summary (table, seed item, endpoints).

The item or value map built here goes through item_depth::from_wire (item_depth.rs:381), which turns a deserialize failure into InternalServerError. The ? then fails the whole request. The messages also differ outside the batch path (experiments 3 and 4).

Experiment 1 [Difference]: INSERT with an empty set, next to a good INSERT

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-b','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Empty bags are not supported"}}, {TableName: partiql_probe_sf}]. get-item b2-a: no item; get-item b2-b: item written.
ExtendDB InternalServerError: Internal server error (HTTP 500). Neither item written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: UPDATE SET to an empty set, next to a good INSERT

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "UPDATE partiql_probe_sf SET v = <<>> WHERE pk = 'h' AND sk = 's'"
  },
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-b','sk':'s'}"
  }
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
Endpoint Result
DynamoDB HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Empty bags are not supported"}}, {TableName: partiql_probe_sf}]. get-item b2-b: item written.
ExtendDB InternalServerError: Internal server error (HTTP 500). b2-b not written.

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: the same INSERT through ExecuteStatement

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb execute-statement --region us-east-1 --statement "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"
Endpoint Result
DynamoDB ValidationException Empty bags are not supported
ExtendDB ValidationException One or more parameter values were invalid: An string set may not be empty

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 4 [Difference]: the same INSERT through ExecuteTransaction

aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
  {
    "Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"
  }
]
EOF
aws dynamodb execute-transaction --region us-east-1 --transact-statements file://req.json
Endpoint Result
DynamoDB ValidationException Validation failed in TransactStatements[0]: Empty bags are not supported
ExtendDB ValidationException One or more parameter values were invalid: An string set may not be empty (no TransactStatements[0] prefix)

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

TokenKind::Ident(word) => match word_class(&word) {
WordClass::Name => {
if matches!(self.nth_kind(1), Some(TokenKind::Dot | TokenKind::LBracket)) {
self.path()?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] Regression from 94e69367: a path inside brackets recurses with no depth limit, so a single statement can overflow the stack.

Setup: see the review summary (table, seed item, endpoints).

bracket_step reads a path inside [...] by calling self.path() again (here, and self.path_steps at line 130). Nothing on that route goes through self.nested(..), so the caps in limits.rs don't apply, and the parse never moves to the large stack. On the release build the server survived the longest statement allowed (8,192 characters). A review run of the parser alone hit fatal runtime error: stack overflow, aborting at about 1,000 levels in a debug build, and at the maximum length on a 1.5 MB stack. That would take the whole server down. DynamoDB refuses at the first nested step, whatever the depth, so it never recurses:

Experiment 1 [Difference]: 10 levels (m[k[k[…k]…]], 88 characters)

aws dynamodb execute-statement --region us-east-1 --statement "SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[k[k[k[k[k[k[k[k[k[k[k]]]]]]]]]]] = 1"
Endpoint Result
DynamoDB ValidationException Unexpected path component at 1:53:1
ExtendDB ValidationException Unexpected path component at 1:71:1

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 2 [Difference]: 1,000 levels (3,058 characters)

STMT="SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[$(printf 'k[%.0s' $(seq 1000))k$(printf ']%.0s' $(seq 1000))] = 1"
echo "${#STMT} characters"
aws dynamodb execute-statement --region us-east-1 --statement "$STMT"
Endpoint Result
DynamoDB ValidationException Unexpected path component at 1:53:1
ExtendDB ValidationException Unexpected path component at 1:2051:1

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

Experiment 3 [Difference]: 2,700 levels (8,158 characters, just under the statement limit)

STMT="SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[$(printf 'k[%.0s' $(seq 2700))k$(printf ']%.0s' $(seq 2700))] = 1"
echo "${#STMT} characters"
aws dynamodb execute-statement --region us-east-1 --statement "$STMT"
Endpoint Result
DynamoDB ValidationException Unexpected path component at 1:53:1 (8 of 9 runs; one run answered InternalFailure with an empty message)
ExtendDB ValidationException Unexpected path component at 1:5451:1 (the release server survived)

Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.

DynamoDB's column 53 is the first k inside m[; ExtendDB's column is the innermost k, which shows it walked every level. Refusing a path inside brackets at its first step would match DynamoDB, and it removes the recursion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants