Skip to content

Ignore raw grammar metadata when assigning rule IDs - #306

Open
RKS (rksharma-owg) wants to merge 1 commit into
microsoft:mainfrom
rksharma-owg:fix/raw-grammar-rule-ids
Open

RKS (rksharma-owg) wants to merge 1 commit into
microsoft:mainfrom
rksharma-owg:fix/raw-grammar-rule-ids

Conversation

@rksharma-owg

Copy link
Copy Markdown

Raw grammar rules can contain metadata named id, which the rule compiler currently trusts as an internal compiled rule identifier. A string or out-of-range numeric value crashes tokenization; a colliding numeric value can silently select the wrong rule.

Clear incoming rule IDs on the existing cloned grammar before compilation. Traverse pattern, repository and capture rule containers plus injections, preserving repository entries named id, location metadata and caller-owned input. External grammars use the same initialization path.

Add 18 regressions using real Oniguruma tokenization. They cover string/numeric IDs in patterns, repository entries, captures, injections and external grammars; begin/end/while captures; exact token scopes/ranges; unchanged caller input and independent registries.

Validation:

  • All 18 regressions fail with the original initialization code.
  • Node 22 compilation and full suite: 285 passed, 1 existing skipped test, 0 failed.
  • Fork cloud validation verifies the exact patch and baseline SHAs on Ubuntu Node 22.23.3: compilation and all tests pass (patch 285 passed / 1 existing skip; baseline 267 passed / 1 existing skip).
  • npm run bundle fails identically on baseline and patch because webpack expects out/main.js while TypeScript emits out/src/main.js. API extraction is not reached. This existing packaging failure is outside this tokenizer change.

Fixes #229.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON key "id" breaks the tokenizer

1 participant