[LeeroyHannigan]

What

tokenize_with_limit builds the near text of a syntax error from the five bytes either side of the byte it cannot tokenize. When that window ended inside a multi-byte UTF-8 character, slicing the expression string panicked, and the request failed with a panic in its handler instead of a ValidationException.

• Both ends of the window now move out to the nearest character boundary.
• The token in the message is the whole character at that position, as tokenize_for already reports it. Before, a byte that starts a multi-byte character was shown as a single Latin-1 character.
• Messages for ASCII input are unchanged.

tokenize_with_limit tokenizes ConditionExpression and FilterExpression in the engine, and the key condition the dynamodb:LeadingKeys check reads.

Testing

• New unit tests. Five inputs whose window ends inside a two-, three- or four-byte character now return a ValidationException; on main each of them panics at the slice. A multi-byte invalid character is reported whole. An ASCII error keeps its five-byte window.
• cargo test -p extenddb-core -p extenddb-engine (951 tests), cargo +1.97.0 clippy --all-targets -- -D warnings, cargo fmt --check, and cargo +1.88.0 check --all-targets -p extenddb-core.

ADR / RFC: n/a

[eliseong]

This change fixes the request failure and the token it now reports matches the live service. No blocking issues.

These hold on this branch and match the live service:

• An error-context window can fall inside a multi-byte character. On main, these inputs fail the request. On this branch, they return a ValidationException. Two examples: a FilterExpression of three two-byte characters, and $ before a three-byte character.
• The reported token is the whole invalid character. The live service does the same. a = é returns token: "é" and a = 😀 returns token: "😀", on the service and on this branch. The previous code reported the first byte as a Latin-1 character.
• The forward end of the window moves to a character boundary, so a trailing multi-byte character stays whole. $ before a run of three-byte characters returns near: "$日日". $xyzw before a three-byte character returns near: "$xyzw".
• This holds for ConditionExpression, FilterExpression, KeyConditionExpression, UpdateExpression, and ProjectionExpression, at the expression size limit, and under deep nesting.

One point is pre-existing and out of scope here. The near window is a fixed five bytes on each side of the token, and the live service builds it from the token stream, one token on each side. attribute_exists(a) $ b returns near: ") $ b" on the service and near: "s(a) $ b" here. #567 already replaces the byte window with a token-based one in both tokenizer entry points and matches the service, so it does not need a separate change.