Repository navigation
feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction - #411
LeeroyHannigan wants to merge 22 commits into
Conversation
2c60924 to
afe7993
Compare
afe7993 to
2036786
Compare
Adds crates/core/src/partiql, compiled only with the new default-on `partiql` Cargo feature. It turns a statement string into an AST and then into a plan made of the operations the engine already has. Nothing else in core depends on it. The lexer and parser take SELECT, INSERT, UPDATE (also in its FROM-first form), DELETE and, inside ExecuteTransaction, EXISTS, with positional parameters. They refuse what the service refuses, with its messages: a statement longer than 8192 characters, a WHERE of more than 300 operators, an IN list of more than 100 members, a value nested inside 32 maps or lists, a number with more than 38 significant digits or outside the service's magnitude range, and reserved words used as names. A statement that nests deeper than 32 levels is parsed on a thread with a 64 MiB stack, and nesting past the service's limits (1000 groups, 1000 literal levels) is refused. The planner reads a WHERE the way the service does. OR and IN on key attributes expand into disjuncts, and each disjunct is a point read, a Query of one partition, or part of a scan. Overlapping key conditions, key values of the wrong type and paths into key attributes are refused in the order the service checks them. DuplicateItemException is a new error variant. Only a PartiQL INSERT of a key that already exists returns it. Each rule and message was measured against Amazon DynamoDB in us-east-1 between 2026-10-19 and 2026-11-03, and the code cites the measurement where it implements the rule.
…eTransaction Adds crates/engine/src/partiql behind the same `partiql` feature. It translates each statement onto the handlers the engine already has (get, query, scan, put, update and delete, and their batch and transaction forms) and changes no storage backend and no classic handler. The dispatch gains three arms and is_known_operation three entries, each behind the feature. With the feature off the server answers the three operations with UnknownOperationException as before, and removing PartiQL is deleting the two partiql modules, those arms and entries, the DuplicateItemException variant and the feature. SELECT reads by key when every disjunct of its WHERE is keyed, and scans the table or index otherwise. A read by key that splits into more than 50 disjuncts is refused with "Too many decomposed read operations for a given query." A page visits at most 10 disjuncts, the first page in key order (or the ORDER BY order) and later pages in the order the service uses, and stops at Limit items or at 1 MB of item bytes read. A page of a read by key has a LastEvaluatedKey only when it holds Limit items, and every page reports the capacity of its own reads. An index read takes one Query for each disjunct. A disjunct that fails a check made when it is read (a key over the size limit, a key attribute in its filter) is held back: it takes a visit and reads nothing, and the request whose page would visit only held-back disjuncts gets its refusal. The NextToken carries where the next page goes on from and digests of the statement and its parameters, and a token sent with another statement or other parameter values is refused. INSERT creates an item and answers DuplicateItemException for a key that exists. UPDATE needs the full key, does not create a missing item, and supports SET (with arithmetic, list_append, set_add and set_delete), REMOVE and the RETURNING modes. DELETE needs the full key, succeeds on a missing item and allows RETURNING ALL OLD *. Every other term of a write's WHERE becomes its condition expression. BatchExecuteStatement takes 1 to 25 statements, all reads or all writes, and answers each statement in its own response entry. ExecuteTransaction takes 1 to 100 statements, all reads or all writes and EXISTS checks, cancels as a whole with positional cancellation reasons, and honours ClientRequestToken. The requests a statement is translated into run with the classic expression size and token limits raised to 131,072 (16 for each of the 8192 characters a statement may hold), so the translation of a statement the service accepts is not refused for its length. docs/differences-from-dynamodb.md lists the three operations as supported and adds a PartiQL section. The difference it records: a PartiQL request is not throttled by table, because the throttle reads the request's top-level TableName and a PartiQL request names its table in the statement. Behaviour was measured against Amazon DynamoDB in us-east-1 between 2026-10-19 and 2026-11-03.
Adds 30 files, tests/test_partiql_*.py, with 1,883 test cases. Each case runs against an ExtendDB server when EXTENDDB_TEST_ENDPOINT is set and against Amazon DynamoDB otherwise, with the same assertions. Every expected value was measured on the service in us-east-1, and on 2026-11-03 the whole set ran against the service: 1,880 passed and 3 were skipped. Where the service gives more than one answer to the same request, the tests accept each answer it gave: whether a LastEvaluatedKey or an empty page follows the last item of a partition, which depends on where the partition sits in storage, and the two forms in which a validation message echoes a statement list (its fields, or the name of the service's request object). A test that reads an index first waits until the index shows every item it wrote. Expected failures on ExtendDB: - The order in which parameter map members are checked: request order when the build includes the MongoDB crate, name order otherwise. Not strict. - Two divergences in code PartiQL shares with the classic operations, to be fixed there: set_add and set_delete on a list element (classic ADD and DELETE), and the 32-level nesting check on parameter values a statement does not store. Strict.
Until this commit the server authorized the three PartiQL operations
once, before dispatch, with the operation's action against table/*. A
Deny on one table did not apply, a dynamodb:LeadingKeys condition had no
values to check, and a policy that allows dynamodb:PartiQLSelect and the
other PartiQL actions allowed nothing. The service authorizes each
statement by itself, with dynamodb:PartiQLSelect, PartiQLInsert,
PartiQLUpdate or PartiQLDelete against the table it reads or writes, or
against the index alone for a read of an index. It needs no action for
the operation itself, and the classic actions grant nothing.
The engine gains an ItemAuthorizer trait, carried in OperationContext.
The server's implementation evaluates the caller's policies with
check_authorization against this deployment's table or index ARN, with
the request-level condition keys every other operation carries and the
item-level keys of the statement. In developer mode it allows every
access, as authorize_request does for the other operations.
authorize_request no longer authorizes the three operations, and the
engine refuses every statement when its context has no authorizer. The
trait is not behind the partiql feature, and only PartiQL uses it.
Each executor makes its checks in the order a permitted caller gets
them and authorizes the statement before it reads or writes an item.
The checks of the request and the refusals made at parse time come
first, as before. Among the checks of a statement, these also come
before authorization: a parameter shortfall, SELECT COUNT(*),
ConsistentRead on a global secondary index, a path into a key attribute,
overlapping disjuncts, more than 50 disjuncts, a NextToken that does not
decode or that another statement gave, a DELETE RETURNING clause other
than ALL OLD *, a BatchExecuteStatement read without the full primary
key, and an EXISTS member of the wrong form or on an index. Every other
check comes after it, so a caller with no permission gets
AccessDeniedException for a missing table, a missing index, a key of the
wrong type or a duplicate INSERT. A check made before authorization is
answered first only when no check ahead of it fails. Otherwise the
statement is authorized first.
Three refusals are made before authorization by analogy with the
measured refusals beside them, and none is measured for a caller with no
permission: a read transaction member that reads an index, one that
does not name the full primary key, and two batch statements on one
item. The module documentation says so where each is made.
A read by key authorizes each disjunct with its own partition key value,
at most 50 values, the MAX_DECOMPOSED_READS limit. A refused disjunct is
held back like one that fails a check made when it is read: it takes a
visit and reads nothing, and the request whose page would visit only
held-back disjuncts gets AccessDeniedException. Across the disjuncts of
one read, an authorization refusal ranks below the refusals of the other
checks. A refused statement in BatchExecuteStatement is its own response
entry, {"Error": {"Code": "AccessDenied", ...}, "TableName": ...}. A
refused member of ExecuteTransaction cancels the transaction before any
condition is evaluated, with AccessDenied for each refused member and
None for the others.
The condition keys:
- dynamodb:LeadingKeys holds the partition key value of each disjunct or
written item. A number is compared in the form Java's new
BigDecimal(s).toString() prints for the text the statement or the
parameter gives it (NumberForm::Decimal): under ForAllValues ["10"],
pk = 10 and pk = 010 run and pk = 10.0 is refused.
- dynamodb:FullTableScan is "true" for a scan and "false" for a read by
key in ExecuteStatement, and absent in BatchExecuteStatement,
ExecuteTransaction and every write. RequestParams in the auth crate
gains full_table_scan to carry "false".
- dynamodb:EnclosingOperation is BatchExecuteStatement or
ExecuteTransaction inside those operations, and absent in
ExecuteStatement.
- dynamodb:Attributes holds the table's key attributes and every
top-level name the statement uses: projection, WHERE, SET values,
REMOVE targets and conditions.
- dynamodb:Select is ALL_ATTRIBUTES for * on a table,
ALL_PROJECTED_ATTRIBUTES for * on an index, and SPECIFIC_ATTRIBUTES
for a projection list.
- dynamodb:ReturnValues is NONE for a write without RETURNING and
follows RETURNING otherwise.
An EXISTS member is authorized as a read. An index read of a missing
table is authorized against the table ARN.
Two refinements outside authorization, measured with a permitted caller:
- A map literal that names a key twice is refused at parse time with
"Duplicate keys in tuple. Key: <name> under <path> at
<line>:<column>:<length>". At the top level of an INSERT VALUE the
position is the second occurrence of the name, with the length of the
quoted name. In any other map it is that map's opening brace, with
length 1. Before this commit ExtendDB accepted the literal and kept the
last value. common::insert_values is the one place that reads an
INSERT's values, for authorization, the write and the duplicate check
of a batch or transaction.
- A SELECT makes its key checks in the service's order: a path into a
key attribute, an ORDER BY term that is not a key attribute, the key
values, the hash-key ORDER BY rule, overlapping disjuncts, and then
the number of disjuncts. ExtendDB checked the key values and overlap
first, so 8 measured statements got another refusal, for example
(pk = 'a' OR pk = 'a') ORDER BY n.
Outside the two partiql modules this commit changes 10 files, with 346
lines added and 3 removed:
- crates/engine/src/item_authorizer.rs, new: the trait and ItemAccess,
with the field and re-export in crates/engine/src/lib.rs.
- crates/server/src/item_authz.rs, new: the server's authorizer, with
its module line in crates/server/src/lib.rs and its construction in
crates/server/src/handler.rs.
- crates/server/src/request_helpers.rs: authorize_request returns early
for the three operations.
- crates/auth/src/policy/context.rs: RequestParams.full_table_scan, with
a unit test.
- crates/server/src/leading_keys.rs: extract_typed_value is pub(crate).
- crates/engine/Cargo.toml and Cargo.lock: extenddb-storage-sqlite as a
dev-dependency, for the unit tests that build an OperationContext.
Removing PartiQL now also means deleting item_authz.rs and its lines in
handler.rs and lib.rs, the early return in authorize_request, and the
trait module, field and re-export in the engine. full_table_scan and the
visibility of extract_typed_value change no other operation and can
stay.
Each rule was measured against Amazon DynamoDB in us-east-1 on
2026-11-03.
Adds ExtendDB-only wire tests for the authorization of each statement, and dual-target tests for the two refinements of the previous commit. The authorization tests, 57 cases in three files, are ExtendDB-only. They create users, roles and policies through the management API, so they need the builtin auth provider and the admin login in EXTENDDB_ADMIN_USER and EXTENDDB_ADMIN_PASSWORD. Each test gives an IAM user the permissions the measured role session had, in its identity policy, and expects the answer the service gave that session on 2026-11-03. - test_partiql_authorization.py: the action of each statement type, table and index resources, the answers of the three operations, dynamodb:LeadingKeys for each disjunct and each write, dynamodb:FullTableScan, dynamodb:EnclosingOperation and session policies. Two tests send every statement path, and a NextToken a permitted caller got, from a caller without a PartiQL action. - test_partiql_authorization_keys.py: EXISTS members, dynamodb:ReturnValues, dynamodb:Attributes, dynamodb:Select, number partition keys, writes without dynamodb:FullTableScan, and an index read of attributes the index does not hold. - test_partiql_authorization_order.py: the checks on each side of authorization, and the pages of a read by key with refused disjuncts. - partiql_authz_helpers.py holds the principals, policies and refusal checks the three files share. One case is a strict expected failure: the refusal of a role session. The shared authorization message names the session with an arn:aws:iam:: ARN and says that no identity-based policy allows the action. The service names it with an arn:aws:sts:: ARN and names the session policy. Every operation uses that message, so the fix belongs there. The dual-target cases, 23 in two files, ran against Amazon DynamoDB in us-east-1 on 2026-11-03, and all 23 passed. - test_partiql_repeated_names.py: a map literal that names a key twice, at the top level of a VALUE, in a nested map, a list, a SET value and a WHERE literal. Also names that differ only in case, a syntax error after the value, and the refusal in BatchExecuteStatement and ExecuteTransaction. - test_partiql_order_by.py, 12 new cases: the ORDER BY refusals among the key checks, with ORDER BY on a key and on another attribute.
`cargo clippy -- -W clippy::pedantic` reported 144 warnings in the code this PR adds that main does not have. This commit clears the 24 that are not doc_markdown, with no change in behaviour: - single_match_else, manual_let_else, if_not_else, match_same_arms, needless_for_each and many_single_char_names take the form clippy suggests. - next_prefix in plan/ranges.rs returns Option<Vec<u8>> instead of Option<Option<Bound>>, and the BeginsWith arm maps the result back to the type of its bound. - shape_response and write::reshape return DispatchResult instead of a Result that was always Ok. The internal helpers of base_join.rs and returning.rs take the serde_json error by reference. - detect_duplicate_keys keeps a HashSet instead of a HashMap with unit values. - The server boxes the check_authorization future, as large_futures suggests. - split in plan/split.rs moves the cross product of an AND into its own function. lex, batch::run and select::run keep their length, and Parser keeps its six flags, each with an allow and its reason. The 120 doc_markdown warnings stay. They flag names such as PartiQL and NextToken in doc comments, which main also leaves unquoted: main has 542 such warnings.
Since #416, the SQLite and PostgreSQL storage layers size a standalone UpdateItem by its statement: a base of 3, 18 per SET clause, and the written paths with their values, checked against the limit beside the finished item. A PartiQL UPDATE runs as an UpdateItem, so it inherited that rule. At a three-byte key it refused a finished item of 409,600 bytes, which the service writes. The service sizes a PartiQL UPDATE by its finished item alone, as it sizes a transacted Update (measured in us-east-1 on 2026-11-04): - UPDATE ... SET p = ? writes a finished item of exactly 409,600 bytes at a key where UpdateItem with SET p = :p stops 17 bytes short. - One byte more is refused with "Item size to update has exceeded the maximum allowed size". - A second SET clause or an if_not_exists adds nothing to the size. - BatchExecuteStatement gives the refusal as the statement's ValidationError entry. ExecuteTransaction cancels with a ValidationError reason with the same message. - A WHERE term that fails, and the implicit check that the item exists, are answered first: an UPDATE over the limit that would not run gets ConditionalCheckFailedException. ExpressionMaps gains finished_item_sizing. When it is set, validate_update_item_size checks only the finished item, as validate_transact_update_item_size does. handle_update_item now calls handle_update_item_with(body, ctx, false), and the PartiQL UPDATE executor calls handle_update_item_with with true, in ExecuteStatement and BatchExecuteStatement. ExecuteTransaction already ran through the transacted Update and its sizing. Outside the two partiql modules this commit changes: - crates/core/src/expression/resolver.rs: the finished_item_sizing field. - crates/core/src/validation/mod.rs: the branch in validate_update_item_size, with a unit test. - crates/engine/src/update_item.rs: handle_update_item_with. - docs/differences-from-dynamodb.md: the Item Size section names the PartiQL UPDATE beside the transacted Update. Removing PartiQL now also means deleting the field, the branch and handle_update_item_with. tests/test_partiql_item_size.py adds 13 dual-target cases. All 13 passed against the service on 2026-11-04. The cases that need a refusal skip on MongoDB and Cassandra, whose update sizing differs from the service's, as docs/differences-from-dynamodb.md records.
2036786 to
af0fac7
Compare
hitesh96db
left a comment
There was a problem hiding this comment.
How these differences were measured
Every inline comment in this review reports results from one experiment, described here once.
When and where. On 2026-11-05 the same requests were sent to two endpoints:
- Amazon DynamoDB in
us-east-1 - this PR at
af0fac76, built with the default PostgreSQL backend and served athttps://127.0.0.1:18443
The requests were sent with boto3, with client retries off, so each case is exactly one request. The commands in the comments are the AWS CLI form of those requests. The same command was run against both endpoints; for ExtendDB it adds --endpoint-url https://127.0.0.1:18443.
Table. Both endpoints had the same table, created with CreateTable and seeded with PutItem:
- table
partiql_probe_t: hash keypk(S), range keysk(S), on-demand - GSI
gsi: hash keyg(S), range keygs(S), projection ALL - seed items
{pk, sk, a: 1, b: 2, x: 0, m: {k: 1, 'my key': 1}, g: <pk>, gs: '1'}for(pk, sk)=('a','b'),('a','c')(withgs: '2') and('b','b')
Write cases. Each write case first puts a fresh item of the same shape under its own pk with sk = 'b' (an INSERT case deletes it instead). After the request it reads that item back with a consistent GetItem, which is where "stored" and "unchanged" in the comments come from.
Names. The comments write the table as t. Positions inside error messages (at line:column:length) are shifted to match the shorter name. Every such position is on line 1, so the shift is exact.
How to read the comments. Each experiment is labelled:
- [Difference]: DynamoDB and ExtendDB answered differently. The suggestion is to change ExtendDB to match DynamoDB and add the experiment as a dual-target test.
- [Same]: both answered the same, usually a control next to a difference. No code change is needed, but the experiment is worth adding as a dual-target test so the behaviour stays pinned.
Every experiment ends with that suggestion. All of them could go into the existing tests/test_partiql_*.py files, which already run against both targets.
Batch and transaction requests. These are written as a req.json here-doc followed by aws dynamodb batch-execute-statement --statements file://req.json or aws dynamodb execute-transaction --transact-statements file://req.json.
| Some(KeyEquality::Partition(value)) => (&mut pk_value, value, pk), | ||
| Some(KeyEquality::Sort(value)) => (&mut sk_value, value, sk.unwrap_or_default()), | ||
| None => { | ||
| residual.push(term); |
There was a problem hiding this comment.
A path into a key attribute in a write's WHERE becomes part of the condition; DynamoDB refuses the statement.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: DELETE with a map lookup on the hash key
aws dynamodb execute-statement --statement "DELETE FROM t WHERE pk = 'k1' AND sk = 'b' AND pk.c = 1"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key pk; item unchanged |
| ExtendDB | ConditionalCheckFailedException The conditional request failed; item unchanged |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: UPDATE with a list index on the range key
aws dynamodb execute-statement --statement "UPDATE t SET x = 1 WHERE pk = 'k2' AND sk = 'b' AND sk[0] = 'z'"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key sk; item unchanged |
| ExtendDB | ConditionalCheckFailedException The conditional request failed; item unchanged |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
SELECT already gives the DynamoDB refusal. In the write path the term goes to residual here and becomes part of the condition expression.
There was a problem hiding this comment.
ExtendDB now refuses both statements before it checks the condition, with DynamoDB's message: the DELETE with pk.c = 1 gets "Key attributes must be scalars; list random access '[]' and map lookup '.' are not allowed: Key pk", the UPDATE with sk[0] = 'z' gets the same message naming sk, and the item is unchanged. A path into a key in a condition that does not compare it with values, such as NOT pk.c = 1, gets the message DynamoDB gives at its later check, with "Key:" before the name, after the key type checks and also for an item that does not exist. Fixed in 095190c. The dual-target tests are test_a_write_refuses_a_path_into_a_key_tested_against_values and test_a_write_refuses_a_path_into_a_key_in_its_condition in tests/test_partiql_key_conditions.py.
| pub fn check_number(text: &str) -> Result<(), NumberProblem> { | ||
| let decimal = read_decimal(text).ok_or(NumberProblem::Unconvertible)?; | ||
| let significant = decimal.digits.trim_start_matches('0'); | ||
| if significant.is_empty() { |
There was a problem hiding this comment.
Negative-zero decimals are stored as 0; DynamoDB refuses them, in values and in WHERE.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: INSERT a negative-zero decimal
aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z1','sk':'b','x':-0.0}"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Negative zero decimal is not supported under key root at 1:46:3 (the position is the 0.0 after the sign) |
| ExtendDB | OK; GetItem returns x = 0 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: UPDATE SET with an exponent form
aws dynamodb execute-statement --statement "UPDATE t SET x = -0e0 WHERE pk = 'z2' AND sk = 'b'"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Negative zero decimal is not supported under key root at 1:19:3 |
| ExtendDB | OK; GetItem returns x = 0 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: negative zero inside a list
aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z4','sk':'b','x':[-0.0]}"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Negative zero decimal is not supported under key root[0] at 1:47:3 |
| ExtendDB | OK; GetItem returns x = [0] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: negative zero in WHERE
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND x = -0.0"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Negative zero decimal is not supported under key root at 1:41:3 |
| ExtendDB | OK, 2 items |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Same]: an integer zero with a sign
aws dynamodb execute-statement --statement "INSERT INTO t VALUE {'pk':'z3','sk':'b','x':-0}"| Endpoint | Result |
|---|---|
| DynamoDB | OK; GetItem returns x = 0 |
| ExtendDB | OK; GetItem returns x = 0 |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
The location in DynamoDB's message names the value (root, root[0]), not the attribute.
There was a problem hiding this comment.
ExtendDB now refuses a negative zero decimal literal with DynamoDB's message and position: -0.0 in the INSERT, -0e0 in the SET and -0.0 in the WHERE get "Negative zero decimal is not supported under key root" at the number after the sign, and [-0.0] names root[0]. Two minus signs that cancel and a sign before parentheses, as in - -0.0 and -(0.0), are refused too, as DynamoDB refuses them. The integer -0 and a parameter whose N value is -0.0 are still stored as 0. Fixed in eb92f2f. The dual-target tests are test_a_negative_zero_decimal_in_an_insert_is_refused, test_a_negative_zero_decimal_in_a_set_or_a_condition_is_refused and test_an_integer_zero_and_a_negative_zero_parameter_are_zero in tests/test_partiql_grammar.py.
| } | ||
| } | ||
| TokenKind::QuotedIdent(_) => Ok(Operand::Path(self.path()?)), | ||
| _ => Err(self.term_error()), |
There was a problem hiding this comment.
One value or path in parentheses is refused here; DynamoDB treats the parentheses as grouping and runs the statement.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: a value in parentheses
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = ('a')"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Statement wasn't well formed, can't be processed: Unexpected term |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: a key path in parentheses
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE (pk) = 'a'"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Statement wasn't well formed, can't be processed: Expected a comparison operator |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: a parameter in parentheses
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = (?)" --parameters '[{"S":"a"}]'| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Statement wasn't well formed, can't be processed: Unexpected term |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: a non-key path in parentheses
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND (a) = 1"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Statement wasn't well formed, can't be processed: Expected a comparison operator |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Difference]: a signed number in parentheses in SET
aws dynamodb execute-statement --statement "UPDATE t SET x = -(5) WHERE pk = 'p4' AND sk = 'b'"| Endpoint | Result |
|---|---|
| DynamoDB | OK; GetItem returns x = -5 |
| ExtendDB | ValidationException Statement wasn't well formed, can't be processed: Unexpected term; x stays 0 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
There was a problem hiding this comment.
ExtendDB now reads one value, parameter or path in parentheses as grouping, as DynamoDB does: pk = ('a'), (pk) = 'a', pk = (?) and pk = 'a' AND (a) = 1 each return the 2 items, and SET x = -(5) stores -5. A comma inside the parentheses makes a list, and an operand alone in parentheses keeps the refusal it gets without them. Fixed in b156d29, and 90aea5f gives DynamoDB's syntax error when a token that does not continue the operand follows it, as in (a b), which gets "Expected RIGHT_PAREN" in place of "Expected a comparison operator". The dual-target tests are test_parentheses_around_one_value_or_path_group_it, test_a_parameter_in_parentheses_and_a_key_in_parentheses_read_by_key, test_a_set_value_in_parentheses and test_a_token_after_an_operand_ends_it in tests/test_partiql_grammar.py.
| // anything else that is not a name, after a dot. | ||
| _ => return Err(ParseError::well_formed("Invalid path dot component")), | ||
| } | ||
| } else if self.eat(&TokenKind::LBracket) { |
There was a problem hiding this comment.
A string or name subscript is refused here; DynamoDB reads m['k'] and m[k] as the map step m.k.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: a string subscript in WHERE
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m['k'] = 1"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items (same as m.k = 1) |
| ExtendDB | ValidationException Unexpected path component at 1:38:3 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: a string subscript that is not a valid bare name
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m['my key'] = 1"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Unexpected path component at 1:38:8 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: a bare name in brackets
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m[k] = 1"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | ValidationException Unexpected path component at 1:38:1 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: a string subscript as a SET target
aws dynamodb execute-statement --statement "UPDATE t SET m['k'] = 5 WHERE pk = 's6' AND sk = 'b'"| Endpoint | Result |
|---|---|
| DynamoDB | OK; GetItem returns m = {k: 5, 'my key': 1} |
| ExtendDB | ValidationException Unexpected path component at 1:16:3; item unchanged |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Difference]: an empty string subscript
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m[''] = 1"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Path component cannot be an empty string |
| ExtendDB | ValidationException Unexpected path component at 1:38:2 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 6 [Same]: the dotted form
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' AND m.k = 1"| Endpoint | Result |
|---|---|
| DynamoDB | OK, 2 items |
| ExtendDB | OK, 2 items |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
There was a problem hiding this comment.
ExtendDB now reads a string or a name in brackets as a map member, as DynamoDB does: m['k'] = 1, m['my key'] = 1 and m[k] = 1 return the same 2 items as m.k = 1, SET m['k'] = 5 sets the member k, and m[''] gets "Path component cannot be an empty string". Other contents of brackets keep the refusals DynamoDB gives them. Fixed in 94e6936. The dual-target tests are test_a_string_or_a_name_in_brackets_is_a_map_member, test_a_member_in_brackets_in_an_update and test_other_contents_of_brackets_are_refused in tests/test_partiql_grammar.py.
| table: bool, | ||
| value_of: &dyn Fn(&Operand) -> Option<AttributeValue>, | ||
| ) -> Result<(), DynamoDbError> { | ||
| let Some(order) = order.filter(|_| table) else { |
There was a problem hiding this comment.
ORDER BY on an index skips the ORDER BY checks, and a path term isn't checked on a table either. DynamoDB refuses all three.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: ORDER BY on an index with no WHERE
aws dynamodb execute-statement --statement 'SELECT * FROM "t"."gsi" ORDER BY g'| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Must have WHERE clause in the statement when using ORDER BY clause. |
| ExtendDB | OK, scans the index and returns its items |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: ORDER BY on an index without an equality on the index hash key
aws dynamodb execute-statement --statement "SELECT * FROM \"t\".\"gsi\" WHERE g > 'a' ORDER BY g"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Must have at least one non-optional hash key condition in WHERE clause when using ORDER BY clause. |
| ExtendDB | OK, returns items |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: ORDER BY a path on the table
aws dynamodb execute-statement --statement "SELECT * FROM t WHERE pk = 'a' ORDER BY m.x"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Argument to ORDER BY must be of type VariableReference. Type: PATH at 1:41:1 (the position of m) |
| ExtendDB | OK, 2 items |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Same]: ORDER BY on an index with the hash key
aws dynamodb execute-statement --statement "SELECT * FROM \"t\".\"gsi\" WHERE g = 'a' ORDER BY gs DESC"| Endpoint | Result |
|---|---|
| DynamoDB | OK |
| ExtendDB | OK |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
Experiment 5 [Same]: ORDER BY on the table with no WHERE
aws dynamodb execute-statement --statement "SELECT * FROM t ORDER BY pk"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Must have WHERE clause in the statement when using ORDER BY clause. |
| ExtendDB | ValidationException Must have WHERE clause in the statement when using ORDER BY clause. |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
The table path already gives the first message; filter(|_| table) here drops the checks for an index.
There was a problem hiding this comment.
ExtendDB now makes the ORDER BY checks on an index as on a table, with DynamoDB's messages: the index read with no WHERE gets "Must have WHERE clause in the statement when using ORDER BY clause.", the one with g > 'a' gets "Must have at least one non-optional hash key condition in WHERE clause when using ORDER BY clause.", and ORDER BY m.x gets "Argument to ORDER BY must be of type VariableReference. Type: PATH" at the position of m, on a table and on an index. Fixed in c8c51ed. The dual-target tests are test_an_order_by_on_an_index_needs_a_where_that_reads_it_by_key, test_a_path_in_the_order_by_list_is_refused_at_its_first_token and test_a_path_in_the_order_by_list_of_an_index_read in tests/test_partiql_order_by.py.
| } | ||
|
|
||
| // Two statements naming the same item fail the whole request. | ||
| detect_duplicate_keys(&parsed, &key_infos, is_write_batch)?; |
There was a problem hiding this comment.
The duplicate-key check counts a statement that failed validation, so the whole batch is refused. On DynamoDB the failed statement gets its own error and the other statement runs.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: an INSERT with an unbound ? and a DELETE of the same key (item cb3 exists beforehand; no Parameters)
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO t VALUE {'pk':'cb3','sk':'b','x':?}"
},
{
"Statement": "DELETE FROM t WHERE pk = 'cb3' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Number of parameters in request and statement don't match."}}, {TableName: t}]; GetItem afterwards: cb3 was deleted |
| ExtendDB | ValidationException Provided list of item keys contains duplicates; GetItem afterwards: cb3 still there |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Same]: two well-formed statements on one item
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'bwc' AND sk = 'b'"
},
{
"Statement": "DELETE FROM t WHERE pk = 'bwc' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Provided list of item keys contains duplicates |
| ExtendDB | ValidationException Provided list of item keys contains duplicates |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
There was a problem hiding this comment.
Fixed in 6c7e67c, the same change as the mix check. Only a statement that translates and passes the checks of its request counts toward the duplicate refusal now, so the INSERT with the unbound ? gets its own error and the DELETE runs. An UPDATE of a key attribute, or an index key of the wrong type, is left out too, and two writes that reach the item are still duplicates even when the first one's condition fails. The tests are test_a_write_refused_before_its_item_is_not_a_duplicate and test_two_writes_that_reach_one_item_are_duplicates in tests/test_partiql_batch.py.
| other => Self { | ||
| code: "ValidationError", | ||
| message: other.message().to_owned(), | ||
| include_table_name: false, |
There was a problem hiding this comment.
A per-statement ValidationError in a batch has no TableName; DynamoDB includes it when the error comes from running the statement.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: an UPDATE that sets a key attribute
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET pk = 'z' WHERE pk = 'cb4' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | Responses: [{TableName: t, Error: {Code: ValidationError, Message: "One or more parameter values were invalid: Cannot update attribute pk. This attribute is part of the key"}}] |
| ExtendDB | Responses: [{Error: {Code: ValidationError, Message: "One or more parameter values were invalid: Cannot update attribute pk. This attribute is part of the key"}}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Same]: an error raised while reading the statement (a ? with no Parameters)
cat > req.json <<'EOF'
[
{
"Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
},
{
"Statement": "SELECT * FROM t WHERE pk = ? AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | the second entry is {Error: {...}} with no TableName |
| ExtendDB | the second entry is {Error: {...}} with no TableName |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
There was a problem hiding this comment.
Fixed in 6c7e67c. An error the item operation raises now carries TableName: your key attribute update, a REMOVE of a key attribute, an index key of the wrong type, an operand of the wrong type and a path the item doesn't have. Errors from reading or translating the statement still carry none, the unbound ? included. The tests are test_an_error_of_the_item_operation_names_the_table and test_an_error_of_reading_the_statement_names_no_table in tests/test_partiql_batch.py.
| // A statement of another type is left to the executors. | ||
| Some(_) => return, | ||
| }; | ||
| let constraint = if statement.is_empty() { |
There was a problem hiding this comment.
A statement of only whitespace is read as a syntax error; DynamoDB refuses it as an empty statement.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: BatchExecuteStatement with a whitespace-only statement
aws dynamodb batch-execute-statement --statements '[{"Statement":" "}]'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "The statement was empty."}}] |
| ExtendDB | HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Statement wasn't well formed, can't be processed: Unexpected term"}}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: ExecuteTransaction with a whitespace-only statement
aws dynamodb execute-transaction --transact-statements '[{"Statement":" "}]'| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException ExecuteTransaction can not have empty statement. (no TransactStatements[0] prefix) |
| ExtendDB | ValidationException Validation failed in TransactStatements[0]: Statement wasn't well formed, can't be processed: Unexpected term |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
The check here only looks for an empty string.
There was a problem hiding this comment.
ExtendDB now refuses a statement of only whitespace as an empty statement, as DynamoDB does: in BatchExecuteStatement its response is the ValidationError "The statement was empty.", and ExecuteTransaction gets "ExecuteTransaction can not have empty statement." with no TransactStatements prefix. The whitespace characters are those DynamoDB treats as whitespace, and the tests cover tab, newline, U+001F, U+2028 and U+3000 among them. Fixed in a80f39e. The dual-target tests are test_a_blank_statement_in_a_batch_fails_that_statement and test_a_blank_statement_in_a_transaction_fails_the_request in tests/test_partiql_requests.py.
| } | ||
| } | ||
|
|
||
| let reads = parsed.iter().filter(|(_, s)| s.is_read()).count(); |
There was a problem hiding this comment.
When a transaction has several faults, ExtendDB reports a different one than DynamoDB. ExtendDB parses every member, then checks RETURNING on every member, then the read/write mix, then each member's form. DynamoDB checks each member fully, in order, and stops at the first member that fails, so a fault in an earlier member wins over any fault in a later one.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: read, write, then a syntax error
cat > req.json <<'EOF'
[
{
"Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
},
{
"Statement": "INSERT INTO t VALUE {'pk':'ct5','sk':'b','a':1}"
},
{
"Statement": "SELECT * FROM"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException ExecuteTransaction API does not support both read and write operations in the same request. |
| ExtendDB | ValidationException Validation failed in TransactStatements[2]: Statement wasn't well formed, can't be processed: Unexpected term |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: a RETURNING clause, then a syntax error
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET x = 1 WHERE pk = 'ct6' AND sk = 'b' RETURNING ALL NEW *"
},
{
"Statement": "SELECT * FROM"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: RETURNING clause is not supported in ExecuteTransaction. |
| ExtendDB | ValidationException Validation failed in TransactStatements[1]: Statement wasn't well formed, can't be processed: Unexpected term |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: an index read, then a write
cat > req.json <<'EOF'
[
{
"Statement": "SELECT * FROM \"t\".\"gsi\" WHERE g = 'a' AND gs = '1'"
},
{
"Statement": "INSERT INTO t VALUE {'pk':'ct7','sk':'b','a':1}"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: Reads on indices are not supported within transactions. |
| ExtendDB | ValidationException ExecuteTransaction API does not support both read and write operations in the same request. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: a read without the sort key, then a read of a missing table
cat > req.json <<'EOF'
[
{
"Statement": "SELECT * FROM t WHERE pk = 'a'"
},
{
"Statement": "SELECT * FROM nosuch_tbl WHERE pk = 'a' AND sk = 'b'"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: Select statements within ExecuteTransaction must specify the primary key in the where clause. |
| ExtendDB | TransactionCanceledException, CancellationReasons: [{Code: None}, {Code: ResourceNotFound, Message: "Requested resource not found"}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Difference]: an EXISTS with only the key, then a read
cat > req.json <<'EOF'
[
{
"Statement": "EXISTS(SELECT * FROM t WHERE pk = 'a' AND sk = 'b')"
},
{
"Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: EXISTS() must contain a single item read with additional condition |
| ExtendDB | ValidationException ExecuteTransaction API does not support both read and write operations in the same request. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 6 [Difference]: an EXISTS on an index of a missing table
cat > req.json <<'EOF'
[
{
"Statement": "EXISTS(SELECT * FROM \"nosuch_tbl\".\"idx\" WHERE pk = 'a' AND x = 1)"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: Operations on indices are not supported in transactions |
| ExtendDB | TransactionCanceledException, CancellationReasons: [{Code: ResourceNotFound, Message: "Requested resource not found"}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 7 [Difference]: a read, then an INSERT with an unbound ? (no Parameters)
cat > req.json <<'EOF'
[
{
"Statement": "SELECT * FROM t WHERE pk = 'a' AND sk = 'b'"
},
{
"Statement": "INSERT INTO t VALUE {'pk':'ct11','sk':?}"
}
]
EOF
aws dynamodb execute-transaction --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[1]: Number of parameters in request and statement don't match. |
| ExtendDB | ValidationException ExecuteTransaction API does not support both read and write operations in the same request. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Where one of these refusals was measured on its own (the read/write mix, RETURNING, a partial key, a key-only EXISTS, a syntax error), the message is the same on both sides.
There was a problem hiding this comment.
Fixed in 26695e3. The executor checks each member in order and stops at the first that fails: its statement and parameters, then its form against its table, then the read and write mix, with a missing table left to cancel the request at the end. The seven cases match DynamoDB now, along with 19 more orderings measured on 5 Nov, and a read transaction with a missing table and a wrong key type cancels with both reasons. The tests are test_the_first_member_that_fails_is_reported and test_a_read_cancellation_gives_every_member_its_reason in tests/test_partiql_transaction.py.
| obj.insert("TableName".to_owned(), Value::String(table.clone())); | ||
| obj.insert("CapacityUnits".to_owned(), json!(units)); | ||
| if let Some(tu) = table_units { | ||
| obj.insert("Table".to_owned(), json!({ "CapacityUnits": tu })); |
There was a problem hiding this comment.
ConsumedCapacity leaves out GSI writes, and a batch leaves out the units of a failed write.
Setup: see the experiment described in the review summary.
Experiment 1 [Difference]: BatchExecuteStatement, INDEXES, one UPDATE that changes the GSI key
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET g = 'x' WHERE pk = 'cb17' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity INDEXES --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | [{TableName: t, CapacityUnits: 3.0, Table: {CapacityUnits: 1.0}, GlobalSecondaryIndexes: {gsi: {CapacityUnits: 2.0}}}] |
| ExtendDB | [{TableName: t, CapacityUnits: 3.0, Table: {CapacityUnits: 1.0}}] (no GlobalSecondaryIndexes) |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: ExecuteTransaction, INDEXES, one UPDATE that changes the GSI key
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET g = 'x' WHERE pk = 'ct17' AND sk = 'b'"
}
]
EOF
aws dynamodb execute-transaction --return-consumed-capacity INDEXES --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | [{TableName: t, CapacityUnits: 4.0, WriteCapacityUnits: 4.0, Table: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}, GlobalSecondaryIndexes: {gsi: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}}}] |
| ExtendDB | [{TableName: t, CapacityUnits: 2.0, WriteCapacityUnits: 2.0, Table: {CapacityUnits: 2.0, WriteCapacityUnits: 2.0}}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: ExecuteTransaction, TOTAL, two UPDATEs of a (projected into the GSI)
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'twr' AND sk = 'b'"
},
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'tws' AND sk = 'b'"
}
]
EOF
aws dynamodb execute-transaction --return-consumed-capacity TOTAL --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | [{TableName: t, CapacityUnits: 6.0, WriteCapacityUnits: 6.0}] |
| ExtendDB | [{TableName: t, CapacityUnits: 4.0, WriteCapacityUnits: 4.0}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: BatchExecuteStatement, TOTAL, one UPDATE that succeeds and one on an absent item (fails its condition)
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'cb15' AND sk = 'b'"
},
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'cb16' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity TOTAL --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | [{TableName: t, CapacityUnits: 3.0}] |
| ExtendDB | [{TableName: t, CapacityUnits: 2.0}] |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Same]: BatchExecuteStatement, TOTAL, two UPDATEs that both succeed
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'bwh' AND sk = 'b'"
},
{
"Statement": "UPDATE t SET a = 9 WHERE pk = 'bwi' AND sk = 'b'"
}
]
EOF
aws dynamodb batch-execute-statement --return-consumed-capacity TOTAL --statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | [{TableName: t, CapacityUnits: 4.0}] |
| ExtendDB | [{TableName: t, CapacityUnits: 4.0}] |
Suggested: no code change; add this experiment as a dual-target test so the behaviour stays pinned.
The batch merge here keeps Table but drops GlobalSecondaryIndexes (experiment 1), and only sums statements that succeeded (experiment 4). The transaction numbers (experiments 2 and 3) presumably come from the shared TransactWriteItems path; that wasn't measured separately here.
There was a problem hiding this comment.
The batch part is fixed in 251490d. The batch merge keeps GlobalSecondaryIndexes and LocalSecondaryIndexes, and a write that fails at its item adds the stored item's units, 1 when it's missing, to its table's CapacityUnits, so experiments 1, 4 and 5 match; those units only count when another statement on the same table succeeded, as on DynamoDB. The transaction numbers come from TransactWriteItems, which reports no index units for the classic API either, because the engine has no old or new item for an Update or Delete member. Matching that needs the storage layer to return those items on all four backends, so it's in #866, and experiments 2 and 3 are held as strict expected failures in test_a_transaction_reports_its_index_units and test_a_transaction_total_counts_its_index_units.
crates/engine/Cargo.toml: main added futures to the engine's dependencies. This branch added a dev-dependency on the SQLite storage crate and the default-on partiql feature. The merged file keeps all three. Cargo.lock: the merged file is main's, with extenddb-storage-sqlite added to the dependencies of extenddb-engine for this branch's dev-dependency. crates/engine/src/partiql/snapshot.rs and crates/engine/src/partiql/test_context.rs: main added the backup runtime to OperationContext. The context a batch or a transaction runs in now carries the request's runtime, and the unit test context holds a runtime of one slot with no configured store.
The service translates each statement of a BatchExecuteStatement into the read or the write it runs as, then sorts the statements that translate into reads and writes. A statement whose translation fails gets its own error entry and counts neither as a read nor as a write, nor in the check for two statements on one item: a missing table or index, a key value of the wrong type, an INSERT without a key attribute, a write whose WHERE does not give the full key, a ? with no parameter, and a DELETE returning clause other than RETURNING ALL OLD *. A statement that translates and then fails the check of its request counts as a read or a write, but not in the duplicate check: a read that is not of one item by its full primary key, an UPDATE that sets or removes a key attribute, and a write that gives a secondary index key a value of the wrong type. The batch counted every statement that parsed, so one statement that failed refused the whole request with the mixed read and write refusal or the duplicate key refusal. The errors of the item operation a statement runs as now carry the statement's TableName: an UPDATE of a key attribute, a secondary index key of the wrong type, an update the item cannot take (an operand of the wrong type, a document path the item does not have), a failed condition and a duplicate key. The errors of reading and translating a statement carry none. Before, a key value past its size limit, a failed condition and a duplicate key were the only errors that named the table. Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests are the new cases in tests/test_partiql_batch.py, on a table with a global secondary index from tests/partiql_gsi_table.py.
A BatchExecuteStatement sums the ConsumedCapacity of its statements per table. At INDEXES the sum now keeps the units of each global and local secondary index, as it kept the Table units. The batch's per-table sum dropped GlobalSecondaryIndexes and LocalSecondaryIndexes, so an UPDATE that moved an item in its index reported a total of 3 with a Table arm of 1 and no index arm. A write that fails at its item is charged the write units of the item stored under its key, or one unit when there is none: a failed condition, a duplicate INSERT, and an update the item cannot take (an operand of the wrong type, a document path the item does not have). Those units count in the CapacityUnits of the write's table alone, and only when another statement on that table succeeded; a table on which every write failed reports nothing. An UPDATE of a key attribute and a secondary index key of the wrong type are refused before the item is read and are not charged. The batch counted the statements that succeeded alone, so an UPDATE of a missing item beside one that succeeded reported 2 where the service reports 3. ExecuteTransaction still reports no index units. ExtendDB's TransactWriteItems has no old or new item of an Update or Delete member to rate an index write from, for the classic operation as for PartiQL. Two tests in tests/test_partiql_transaction.py hold the service's answer as expected failures against ExtendDB. Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests are the ConsumedCapacity cases in tests/test_partiql_batch.py and the unit tests of the capacity list.
The service checks the members of an ExecuteTransaction in order and stops at the first one that fails, so a refusal of a member wins over a refusal of any later member. For each member it reads and parses the statement, checks its parameters, then checks the member against its table, and then refuses a read beside a write. The executor parsed every member, then checked every RETURNING clause, then the mix of reads and writes, then the form of each member, so a syntax error in a later member, or the mix, could win over a refusal of an earlier member. The checks of a member against its table, each with the member's position: a read of an index, existing or not, then a read whose WHERE does not give the full primary key; an EXISTS through an index, ahead of a missing table, then an EXISTS that reads one item with no further condition; a RETURNING clause. A member on a missing table gets none of them and cancels the request once every member is checked. A `?` with no parameter is checked with the statement, ahead of the mix. In a read transaction a missing table and a key value of the wrong type now cancel the request together, each member with its own reason, in either order. The missing table was reported alone, with None for the other members. Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests are test_the_first_member_that_fails_is_reported and test_a_read_cancellation_gives_every_member_its_reason in tests/test_partiql_transaction.py.
A read member of an ExecuteTransaction returned the whole item whatever
its projection list named: the TransactGetItems member it was
translated into carried only the table and the key. The executor now
reads each item whole and gives it the member's projection, keeping the
shape of each path as a ProjectionExpression gives it, so SELECT a, m.k
returns {a, m: {k}}. An item that has none of the paths is returned
empty, and a member whose item is missing returns no item. ExtendDB's
TransactGetItems leaves out an item that its projection leaves empty,
where the service returns it empty in a PartiQL transaction, so the
projection is applied here.
A projection list that names an output identifier twice, or whose paths
overlap, is refused on all three operations, the identifier refusal
first. A transaction refuses it with the member's position, with the
checks of the statement, ahead of a missing table, a partial key and an
index read. A batch statement gets it as its own error and counts as
neither a read nor a write. ExecuteStatement refused a repeated
identifier and now also refuses two paths that overlap, which it
returned items for. The overlap message names the later path first, as
an UPDATE's does, and the two share the check.
Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The tests
are the projection cases in tests/test_partiql_transaction.py and
tests/test_partiql_batch.py, and the unit tests of check_projection.
A BatchExecuteStatement member whose statement holds only whitespace now gets the ValidationError "The statement was empty." in its entry, and an ExecuteTransaction with such a member fails with "ExecuteTransaction can not have empty statement." and no member position. The check comes before the member's parameters are converted and before its statement is parsed, so a transaction reports a syntax error of an earlier member first. ExecuteStatement keeps its syntax error for such a statement. The whitespace set follows the service: tab to carriage return, the information separators, space, the line and paragraph separators and the Unicode spaces other than the no-break ones. A comment, U+0085 and U+00A0 are not whitespace here. The dual-target tests in tests/test_partiql_requests.py pin each case.
An ORDER BY on an index skipped the checks of the WHERE it reads, so a read of an index with no WHERE, or with no equality on the index partition key, ran as a scan. The index now gets the refusals a table gets: no WHERE, a WHERE that does not read by key, and more than one partition key value without the partition key in the list. A path with steps in the list, such as m.x, m[0] or pk.x, was not checked and the statement ran. It is now refused with "Argument to ORDER BY must be of type VariableReference. Type: PATH at" the position of the path's first token. The parser keeps that position on each term. The terms are checked in list order together with the rule that a name must be a key attribute. A statement with no WHERE is refused for it ahead of the list, so ORDER BY n with no WHERE gets the missing WHERE refusal on a table and on an index. The dual-target tests in tests/test_partiql_order_by.py pin each case.
A path into a key attribute in the WHERE of an UPDATE or a DELETE, such as pk.c = 1 or sk[0] = 'z', went into the condition the write carries, so the statement failed its condition or ran. The service refuses the statement in two places, and ExtendDB now does the same. A condition that tests the path against values alone (a comparison with a value on either side, BETWEEN, IN with values, begins_with, contains, attribute_type, an IS test of a type, or one of these compared with a boolean) is refused while the WHERE is read for the key, with "Key attributes must be scalars; ...: Key pk". The refusal and the one of two values for one key attribute come in statement order, ahead of the full key and the key types. A transaction member reports the key schema mismatch reason for it, and an EXISTS check fails the request naming its position. Any other condition with a path into a key attribute (OR, NOT, a function of the path, IS MISSING, IS NOT, the path compared with a path) is refused after the key types and after the refusal of a SET of a key attribute, also for an item that does not exist. Its message has "Key:" before the name, which is the sort key when a path into it is among the conditions. A batch entry for it names the table, and a transaction member has it as its cancellation reason. The dual-target tests in tests/test_partiql_key_conditions.py pin each form.
A path step in brackets took only a number, so m['k'] and m[k] were refused with "Unexpected path component". A string, a quoted name or a name in brackets is now a map member, as after a dot: m['k'], m["k"] and m[k] are m.k in a condition, a SET target, a REMOVE, a SET value and the select list, and m['my key'] names a member a dot cannot. An empty string gets "Path component cannot be an empty string". Other contents of brackets parse and are refused once the statement parses, at their first token: a parameter, a number that is not a whole number, a literal word and a path. A minus before a whole number is refused as an index out of range. A reserved word in brackets is a syntax error, and so is a bracket left open, which now reads "Expected RIGHT_BRACKET" with the syntax error prefix. The steps after a path root are read by a function of their own, which the next change uses for a path in parentheses. The dual-target tests in tests/test_partiql_grammar.py pin each case.
Parentheses around one operand were refused: pk = ('a') and pk = (?)
got "Unexpected term", and (pk) = 'a' got "Expected a comparison
operator". They now only group what they hold, as the service reads
them.
In an operand position, (x) is x: a value, a parameter, a path, a call
or an arithmetic operand. A path in parentheses takes the steps after
them, so (m).k is m.k, and a comma of their own makes a list, so SET x =
('a', 'b') stores a list. Signs before a number fold through
parentheses: SET x = -(5) stores -5 and a = -(-1) compares with 1. A
sign before anything else in parentheses keeps its refusal. INSERT
values and literal elements take parentheses the same way.
Where a condition starts, a group that holds one operand and nothing
else is that operand, and the predicate goes on after the group:
(pk) = 'a' reads by key, and (a) BETWEEN 0 AND 2, (a) IN [1] and NOT (a)
= 2 run. An operand alone in parentheses is refused as the operand alone
is. That refusal now also applies inside a group and after a NOT, as the
service gives it: (a AND b = 1) is refused for the arguments of AND, and
NOT a gets "The argument to NOT must be a Type or Expression". Before
anything else such an operand keeps its syntax error.
The dual-target tests in tests/test_partiql_grammar.py pin each case.
A zero written with a decimal point or an exponent and a minus sign
before it, such as -0.0, -0e0 or -.0, was stored or compared as 0. The
service refuses such a literal with "Negative zero decimal is not
supported under key root at 1:46:3", which names where the number sits
in its value (root, root[0], root.y) and the span of the number after
its signs. ExtendDB now gives the same answer.
The parser reads the signs before a number literal, also through
parentheses, so -(0.0), (-0.0) and +-0.0 are refused, and two minus
signs that cancel (- -0.0) keep the refusal. A zero written as an
integer (-0), +0.0 and a parameter whose N value is -0.0 are still
stored as 0, as the service stores them.
The refusal comes when the statement is read, before the table is
resolved and before the key type checks. In a condition, the walk meets
it where it meets a number that breaks an item number rule. In an
INSERT value or a SET clause it comes in the order the statement is
written with those numbers and with a parameter inside a literal: for
{'x':1e126,'y':-0.0} the overflow is reported, and for
{'x':{'b':-0.0,'a':1e126}} the zero. It comes ahead of a sign before a
SET value that is not a number, and after what the condition meets.
The dual-target tests in tests/test_partiql_grammar.py pin each case.
The rustfmt check of toolchain 1.97.0 failed on eight files that the six commits before this one change. This commit applies rustfmt to them, with no change in behaviour.
An operand in a condition followed by a token other than a comparison, AND, OR, the end of the statement or a ) that closes a group got "Expected a comparison operator", which the service does not answer. The service ends the operand there and reads the token in the part of the statement around the operand, and ExtendDB now does the same. WHERE a b and WHERE pk = 'a' AND a b get "Unexpected token after expression", in a SELECT and in a DELETE. (a b) and (pk = 'a' AND a LIMIT 1) get "Expected RIGHT_PAREN". WHERE a LIMIT 1 is refused for the LIMIT clause at its value, and WHERE a ORDER BY pk is refused for the operand with "WHERE clause must contain an expression that resolves to a boolean". A syntax error after the operand still comes ahead of the operand's own refusal. A size call with no comparison after it keeps "Expected a comparison operator", which is not measured. The dual-target tests in tests/test_partiql_grammar.py pin each form.
crates/engine/src/partiql/common.rs: feat/partiql renamed StatementError::from_key_error to from_item_error, which names the table for every error of the item operation a statement runs as, and removed is_key_size_error. The parser fixes added the refusal of a path into a key attribute among a write's other conditions to from_key_error. The result keeps from_item_error and drops that addition. The merged batch answers that refusal once the statement is authorized, through from_item_error, so the entry names the table, and test_a_batch_names_the_table_only_for_a_path_in_the_condition passes. crates/engine/src/partiql/common/tests.rs: feat/partiql renamed the key size test to an_error_of_the_item_operation_names_the_batch_table. The parser fixes added three tests beside it. The result keeps the renamed test and the tests of where_key and condition_key_refusal, and drops the test of from_key_error together with that function. crates/engine/src/partiql/transaction.rs: feat/partiql moved the read and the parse of each member into parse_member, so each member is checked in full before the next one is parsed. The parser fixes refused a statement of only whitespace in the old parse loop. The result makes that refusal the first check in parse_member, ahead of the member's parameters and its parse. DynamoDB reports an earlier member that fails its checks against its table, and a read and a write in one request, ahead of the blank member, and the result does the same. crates/engine/src/partiql/batch/translation.rs: an UPDATE or a DELETE with a path into a key attribute among its other conditions gets that refusal from its translation, after the refusals of the UpdateItem request. Such a statement counts as a write and stays out of the check for two statements on one item, which is how DynamoDB counts it. Without this change the merged code refused such a statement and a second write of the same item as duplicates. Tests: dual-target cases in tests/test_partiql_batch.py, tests/test_partiql_key_conditions.py and tests/test_partiql_transaction.py check these answers, each measured against DynamoDB: the two batch classes of a path into a key attribute, a blank member or a negative zero after a member that fails, an EXISTS with a path into a key attribute ahead of a later refusal, an UPDATE with such a path beside a SELECT without the full key, and a map member in brackets in a projection list.
ExecuteStatement refused an UPDATE that sets a secondary index key to a value of the wrong type for a path into a key attribute among the WHERE's other conditions, such as AND NOT pk.c = 1, when the statement had both. DynamoDB refuses it for the index key type first, as it does for a SET or REMOVE of a key attribute, whether or not the item exists, and BatchExecuteStatement answers the same way. run_update now calls update_refusal in place of targets_a_key, and leaves the statement to UpdateItem when update_refusal finds either refusal. An index key given the value of another attribute is checked on the written item, so the path refusal still comes first for it. update_refusal and insert_refusal move from batch/translation.rs to write/update.rs and write/insert.rs, with index_keys and from_wire, and take the translated request instead of building it. The batch translation calls them and answers as before. In ExecuteTransaction an INSERT or UPDATE member that gives an index key a value of the wrong type had no reason of its own. Its refusal came from TransactWriteItems, which the request reaches only when no member has a reason, and which names the first such member alone. DynamoDB gives each such member the index key refusal as its reason, beside the reasons of the other members, and an UPDATE gets it after a SET of a key attribute and ahead of a path into a key attribute. member_reason now gives it in that order, through update_index_key_refusal and insert_refusal. Measured against Amazon DynamoDB in us-east-1 on 2026-11-05. The dual-target tests are in tests/test_partiql_key_conditions.py, tests/test_partiql_transaction.py and tests/test_partiql_batch.py.
|
@hitesh96db, each of your 12 threads has a reply with the commit and the tests that answer it. The branch is at c745a3c: main merged at 38992eb, the twelve fixes, and one follow-up described below. Every fix has dual-target tests that pass against DynamoDB in us-east-1 and against ExtendDB. The follow-up came from combining the fixes. ExecuteStatement now answers All 35 These answers still differ from DynamoDB. Each answered the same way at af0fac7, and each is a separate change:
#867 is a related gap in the classic TransactGetItems, found while measuring the transaction projections. |
| let params = &req.parameters; | ||
| let refusal = match stmt { | ||
| Statement::Update(update) => { | ||
| let parts = write::build_update_parts(ctx, update, params).await?; |
There was a problem hiding this comment.
[Blocker] Regression from 6c7e67c1: one write the batch can't build now fails the whole batch, so the good statements around it never run.
Setup: see the review summary (table, seed item, endpoints).
This line, and update_refusal/build_insert_parts/insert_refusal next to it, run for every write before any statement executes. Each returns its error with ?, which fails the whole request. Before 6c7e67c1, each write ran through write::run, and its error became that statement's own entry. DynamoDB keeps it per statement:
Experiment 1 [Difference]: SET a = size(m) between two good INSERTs
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
},
{
"Statement": "UPDATE partiql_probe_sf SET a = size(m) WHERE pk = 'h' AND sk = 's'"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: size"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written. |
| ExtendDB | request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: SET a = attribute_exists(m) between two good INSERTs
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
},
{
"Statement": "UPDATE partiql_probe_sf SET a = attribute_exists(m) WHERE pk = 'h' AND sk = 's'"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: attribute_exists"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written. |
| ExtendDB | request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: SET a = begins_with(m, 'x') between two good INSERTs
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
},
{
"Statement": "UPDATE partiql_probe_sf SET a = begins_with(m, 'x') WHERE pk = 'h' AND sk = 's'"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "The function is not allowed in an update expression; function: begins_with"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written. |
| ExtendDB | request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1, :v". Both get-item calls return no item: neither INSERT was written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: SET a = upper(m) between two good INSERTs
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-before','sk':'s'}"
},
{
"Statement": "UPDATE partiql_probe_sf SET a = upper(m) WHERE pk = 'h' AND sk = 's'"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b1-after','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-before"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b1-after"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{TableName: partiql_probe_sf}, {Error: {Code: ValidationError, Message: "Unrecognized function: upper at 1:33:5"}}, {TableName: partiql_probe_sf}]. Both get-item calls return the item: both INSERTs were written. |
| ExtendDB | request-level ValidationException Invalid UpdateExpression: Syntax error; token: "(", near: "(#n1)". Both get-item calls return no item: neither INSERT was written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 5 [Difference]: the same UPDATE through ExecuteStatement (the message behind it)
aws dynamodb execute-statement --region us-east-1 --statement "UPDATE partiql_probe_sf SET a = size(m) WHERE pk = 'h' AND sk = 's'"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException The function is not allowed in an update expression; function: size |
| ExtendDB | ValidationException Syntax error; token: "(", near: "(#n1)" |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Turning the errors from build_*_parts / *_refusal into that statement's Untranslated/Refused entry would fix experiments 1–4. Experiment 5 is the message underneath, which also shows up in ExecuteStatement.
| .or_else(|| condition_key_refusal(update.where_clause.as_ref(), key_info, params)) | ||
| } | ||
| Statement::Insert(insert) => { | ||
| let parts = write::build_insert_parts(ctx, insert, params).await?; |
There was a problem hiding this comment.
[Blocker] Regression from 6c7e67c1: an empty set (<<>>) in a batch write is an InternalServerError, and nothing in the batch runs.
Setup: see the review summary (table, seed item, endpoints).
The item or value map built here goes through item_depth::from_wire (item_depth.rs:381), which turns a deserialize failure into InternalServerError. The ? then fails the whole request. The messages also differ outside the batch path (experiments 3 and 4).
Experiment 1 [Difference]: INSERT with an empty set, next to a good INSERT
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-b','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Empty bags are not supported"}}, {TableName: partiql_probe_sf}]. get-item b2-a: no item; get-item b2-b: item written. |
| ExtendDB | InternalServerError: Internal server error (HTTP 500). Neither item written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: UPDATE SET to an empty set, next to a good INSERT
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "UPDATE partiql_probe_sf SET v = <<>> WHERE pk = 'h' AND sk = 's'"
},
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-b','sk':'s'}"
}
]
EOF
aws dynamodb batch-execute-statement --region us-east-1 --statements file://req.json
aws dynamodb get-item --region us-east-1 --table-name partiql_probe_sf --consistent-read --key '{"pk": {"S": "b2-b"}, "sk": {"S": "s"}}'| Endpoint | Result |
|---|---|
| DynamoDB | HTTP 200, Responses: [{Error: {Code: ValidationError, Message: "Empty bags are not supported"}}, {TableName: partiql_probe_sf}]. get-item b2-b: item written. |
| ExtendDB | InternalServerError: Internal server error (HTTP 500). b2-b not written. |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: the same INSERT through ExecuteStatement
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
aws dynamodb execute-statement --region us-east-1 --statement "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Empty bags are not supported |
| ExtendDB | ValidationException One or more parameter values were invalid: An string set may not be empty |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 4 [Difference]: the same INSERT through ExecuteTransaction
aws dynamodb delete-item --region us-east-1 --table-name partiql_probe_sf --key '{"pk": {"S": "b2-a"}, "sk": {"S": "s"}}'
cat > req.json <<'EOF'
[
{
"Statement": "INSERT INTO partiql_probe_sf VALUE {'pk':'b2-a','sk':'s','v':<<>>}"
}
]
EOF
aws dynamodb execute-transaction --region us-east-1 --transact-statements file://req.json| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Validation failed in TransactStatements[0]: Empty bags are not supported |
| ExtendDB | ValidationException One or more parameter values were invalid: An string set may not be empty (no TransactStatements[0] prefix) |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
| TokenKind::Ident(word) => match word_class(&word) { | ||
| WordClass::Name => { | ||
| if matches!(self.nth_kind(1), Some(TokenKind::Dot | TokenKind::LBracket)) { | ||
| self.path()?; |
There was a problem hiding this comment.
[Blocker] Regression from 94e69367: a path inside brackets recurses with no depth limit, so a single statement can overflow the stack.
Setup: see the review summary (table, seed item, endpoints).
bracket_step reads a path inside [...] by calling self.path() again (here, and self.path_steps at line 130). Nothing on that route goes through self.nested(..), so the caps in limits.rs don't apply, and the parse never moves to the large stack. On the release build the server survived the longest statement allowed (8,192 characters). A review run of the parser alone hit fatal runtime error: stack overflow, aborting at about 1,000 levels in a debug build, and at the maximum length on a 1.5 MB stack. That would take the whole server down. DynamoDB refuses at the first nested step, whatever the depth, so it never recurses:
Experiment 1 [Difference]: 10 levels (m[k[k[…k]…]], 88 characters)
aws dynamodb execute-statement --region us-east-1 --statement "SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[k[k[k[k[k[k[k[k[k[k[k]]]]]]]]]]] = 1"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Unexpected path component at 1:53:1 |
| ExtendDB | ValidationException Unexpected path component at 1:71:1 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 2 [Difference]: 1,000 levels (3,058 characters)
STMT="SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[$(printf 'k[%.0s' $(seq 1000))k$(printf ']%.0s' $(seq 1000))] = 1"
echo "${#STMT} characters"
aws dynamodb execute-statement --region us-east-1 --statement "$STMT"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Unexpected path component at 1:53:1 |
| ExtendDB | ValidationException Unexpected path component at 1:2051:1 |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
Experiment 3 [Difference]: 2,700 levels (8,158 characters, just under the statement limit)
STMT="SELECT * FROM partiql_probe_sf WHERE pk = 'h' AND m[$(printf 'k[%.0s' $(seq 2700))k$(printf ']%.0s' $(seq 2700))] = 1"
echo "${#STMT} characters"
aws dynamodb execute-statement --region us-east-1 --statement "$STMT"| Endpoint | Result |
|---|---|
| DynamoDB | ValidationException Unexpected path component at 1:53:1 (8 of 9 runs; one run answered InternalFailure with an empty message) |
| ExtendDB | ValidationException Unexpected path component at 1:5451:1 (the release server survived) |
Suggested: change ExtendDB to match DynamoDB, and add this experiment as a dual-target test.
DynamoDB's column 53 is the first k inside m[; ExtendDB's column is the innermost k, which shows it walked every level. Refusing a path inside brackets at its first step would match DynamoDB, and it removes the recursion.
What
This PR adds the three PartiQL operations:
ExecuteStatement,BatchExecuteStatementandExecuteTransaction. They run SELECT, INSERT, UPDATE and DELETE statements, and EXISTS checks inside a transaction. The server authorizes each statement by itself, as Amazon DynamoDB does.The engine translates each statement into operations that it already has. A statement runs as the point reads, Queries, Scans and item writes that the service makes for it, through the existing handlers. The PR changes no storage backend and no handler of a classic operation.
crates/core/src/partiql/holds the lexer, the parser, and the planner that maps a WHERE to key reads.crates/engine/src/partiql/holds the executors and the code that authorizes each statement.partiqlCargo feature. With the feature off, the three operations answerUnknownOperationException, as on main.Reads
Writes, batch and transaction
DuplicateItemExceptionfor a key that exists. UPDATE and DELETE need the full key. The other terms of the WHERE become the condition expression.UpdateItemby its statement. At a three-byte key an UPDATE writes an item of exactly 409,600 bytes, whileUpdateItemstops 17 bytes short. The UPDATE executor asks the storage layer for the finished item rule.BatchExecuteStatementtakes 1 to 25 statements, all reads or all writes, with one response entry for each statement.ExecuteTransactiontakes 1 to 100 statements and runs all of them or none. It gives positional cancellation reasons and honoursClientRequestToken.Authorization
dynamodb:PartiQLSelect,PartiQLInsert,PartiQLUpdateorPartiQLDelete. The resource is the table, or the index alone for a read of an index. The operation itself needs no action.ItemAuthorizerin itsOperationContext. The server's authorizer evaluates the caller's policies withcheck_authorization, and allows every access in developer mode. Without an authorizer, the engine refuses every statement.dynamodb:LeadingKeysvalue. A refused statement in a batch gets its ownAccessDeniedentry. A refused transaction member cancels the transaction.LeadingKeys,FullTableScan,EnclosingOperation,Attributes,SelectandReturnValueshold the values that the service gives them.AccessDeniedExceptionfor a missing table, a missing index or a key of the wrong type.Differences from DynamoDB
docs/differences-from-dynamodb.mdrecords one difference: ExtendDB does not throttle a PartiQL request by table. The throttle reads the top-levelTableNameof a request, and a PartiQL request names its table in the statement.Known gaps
The measurements do not cover these orderings for a caller that has no permission. In each case the code follows the measured refusals beside it, and the module documentation names the choice.
The measurements do not cover these orderings for a caller with permission either:
The refusal message for a role session differs from the service's message. ExtendDB names the session with an
arn:aws:iam::ARN and says that no identity-based policy allows the action. The service uses anarn:aws:sts::ARN and names the session policy. Every operation uses this message, so one test stays a strict expected failure until the shared code changes.The NextToken is not signed. The server authorizes each request that presents a NextToken again, from the statement in that request.
Commits
Each commit message lists the rules that its part implements.
Why
On main, the server answers the three operations with
UnknownOperationException, so a client that uses PartiQL cannot run against ExtendDB. The tests expect the values and messages that Amazon DynamoDB gave in us-east-1, measured between 2026-10-19 and 2026-11-03.Testing done
tests/test_partiql_*.py, with 1,976 cases.tests/test_partiql_item_size.pygave 13 passed. On 2026-11-03 the other 31 files gave 1,902 passed, 3 skipped and 1 failed. The failed case makes an eventually consistent read right after aPutItem, and the service returned no item. The case passed on a rerun, and so did the rest of its file.set_addon a list element (classic ADD), and the 32-level nesting check on 3 parameter values that a statement does not store.tests/test_partiql_item_size.pythat need a refusal skip, because those backends size updates differently (MongoDB applies no item size limit to updates, and Cassandra sizes updates on the finished item against a fixed 400 KiB with the PutItem wording #818).test_fgac_leading_keys.py,test_fgac_attributes.py,test_auth_permissions.pyandtest_authz_resource_arn.py): 143 passed and 1 skipped.test_item_size_at_limit.pyandtest_index_key_validation_on_update.py): 122 passed.cargo test --workspaceandcargo +1.88.0 check --workspace --all-targets.cargo fmt --all -- --check,cargo +1.97.0 clippy --all-targets -- -D warningsandcargo check --no-default-featuresof the core and engine crates are clean.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. The PR adds the three operations as the service runs them, and authorizes each statement as the service does.
docs/differences-from-dynamodb.mdrecords the one difference.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.