[LeeroyHannigan]

What

A character outside the expression grammar, such as é, now gets the syntax error DynamoDB gives, with the same token and window. The token is the character as sent. The window runs from the start of the previous token to the end of the next one. Both tokenizers follow this rule:

• The tokenizer behind condition and filter expressions serves ConditionExpression on PutItem, UpdateItem, DeleteItem and TransactWriteItems, FilterExpression on Scan and Query, and KeyConditionExpression on Query. The key condition goes through the server's dynamodb:LeadingKeys parse, which runs before the authorization decision for every caller.
• The tokenizer behind UpdateExpression, ProjectionExpression and the engine's key condition parse.

Not changed here

These differences from the service are on main and stay as they are:

• When a parser fault comes before the character (#v = = é), the service reports that fault. ExtendDB reports the character, because it tokenizes the whole expression before parsing.
• # or : followed by a character that cannot start a name keeps main's texts (empty name reference '#', empty value placeholder ':'). The service answers token: "#", near: "#é" by the same window rule.
• The service strips one leading byte-order mark and accepts the expression. ExtendDB refuses it.
• #v = 1: the service refuses a bare number as an operand. ExtendDB reads it as an attribute name.
• The expression size error: the service no longer appends ; expression size: <n> on filter and condition expressions. ExtendDB does.
• The 1 validation error detected: prefix the service adds on write paths.

Not measured: the service's answer to a caller fenced by dynamodb:LeadingKeys for such a key condition. ExtendDB answers ValidationException under ForAllValues and AccessDeniedException under ForAnyValue, with no items either way.

Files

tokenizer.rs passed 600 lines with the fix, so the second commit moves its tests into tokenizer_tests.rs with no other change, using the #[path] pattern of evaluator_tests.rs. tokenizer.rs is now 449 lines.

Other PRs

git merge-tree --write-tree is clean against #558 and #479. #511 is already on main. #558 builds a window for parser faults from the same rule. Its window also starts at the first token when nothing comes before the fault. The service starts at the start of the expression. #558 fixes that separately.

Why

This PR began as a fix for a panic. The tokenizer behind condition and filter expressions sliced the input five bytes either side of the offending byte. When that range ended inside a multi-byte character (#v = ééé, #v = 日本語), the slice panicked, the request task ended and the client saw the connection close. #791 has since fixed that panic on main. It moves both ends of the five-byte window out to character boundaries and reports the whole character as the token.

The windows on main still differ from the service's. The condition and filter tokenizer answers five bytes either side of the character. The tokenizer behind UpdateExpression, ProjectionExpression and the engine's key condition parse answers the character with one byte after it as the window (near: "é", near: "$x").

What the service answers

Measured on 2026-10-25 at 16:09, 16:34, 16:38, 16:43 and 16:54 UTC in us-east-1, on all five expression kinds. Each request went out twice and got the same answer both times. When the character is the first fault in the expression:

• The token is the character as sent, a whole code point: é, 日, 😀, or $, @ and ! for ASCII characters outside the grammar.
• The window runs from the start of the previous token to the end of the next one, keeping the whitespace between them. Each character outside the grammar counts as one token, and a name stops at the first such character:
  • #v = ééé answers token: "é", near: "= éé".
  • abéc = :v answers near: "abéc".
  • #v = é abc answers near: "= é abc".
  • #v é<> :v answers near: "#v é<>".
  • contains(#v, é) answers near: ", é)".
• When no token comes before the character, the window starts at the start of the expression: é = :v answers near: " é =".
• The service removes U+0000 from the token and the window: #v = \^@x answers token: "", near: "= x".

The merge with main

The merge of main (807937e) keeps this rule and drops main's five-byte window. #791 added three tests:

• tokenize_multibyte_utf8_near_an_invalid_byte_does_not_panic stays as fix(core): keep the tokenizer's error context on character boundaries #791 wrote it. Its one window assertion, $ 日 for the input $ 日, matches the measured rule.
• tokenize_reports_the_whole_invalid_character keeps its token assertion. Its window for a = é changes from a = é to = é, the window that the measured row for #v = é gives.
• The merge drops tokenize_ascii_error_context_is_five_bytes_either_side. It expects the window s(a) $ b for attribute_exists(a) $ b, and the measured rule answers ) $ b.

Testing done

• crates/core/src/expression/tokenizer_tests.rs: one table test with 44 measured rows, run through both tokenizers. The test fails on each of these broken versions of the rule:
  • the window starts at the character
  • the window does not reach the next token
  • the token is read as a byte
  • no two-character operators
  • no name runs
  • no digit runs
  • U+0000 stays in
• tests/test_expression_unknown_character.py: 50 dual-target wire tests on Scan and Query filters, PutItem and UpdateItem conditions, Query key conditions, UpdateItem update expressions and GetItem projections. The Query key condition rows include the shape that closed the connection before fix(core): keep the tokenizer's error context on character boundaries #791.

On the merge commit 807937e, on 2026-11-05, against a SQLite debug build:

• cargo +1.97.0 fmt --all -- --check: no changes.
• cargo +1.97.0 clippy --locked -p extenddb-core --all-targets -- -D warnings: no warnings.
• cargo test --locked -p extenddb-core --lib: 765 passed, 1 ignored.
• tests/test_expression_unknown_character.py: 50 passed, and the server log has no panic.

On 2026-10-25, before #791 merged, against a SQLite release build:

• the new file: 50 passed at the head, 50 failed on main.
• the 11 wire files that assert expression syntax errors, plus the new file: 431 passed at the head, and the server log has no panic.
• cargo test -p extenddb-core on stable (521 passed) and on 1.88.0 (520 passed).

The full SQLite wire suite at the third commit had 2 failures, in tests/test_gsi_async.py. That file starts a server from ./target/release/extenddb, a path that did not exist in that run. main failed the same 2. CI runs the new file in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.