feat(extensions): register custom syntax grammars

[benvinegar]

Problem

Extensions can associate files with languages that Hunk already knows, but cannot provide a grammar for a new DSL or proprietary language. Mapping .arch to archlang, for example, currently leaves the highlighter with an unknown language.

Approach

• add API v17 hunk.registerSyntaxGrammar(...) for bounded, data-only TextMate grammars
• keep filename associations separate through registerFileLanguage
• validate, deep-copy, and freeze grammar data; reject external includes, injections, unsupported keys, oversized structures, bundled language collisions, and duplicate ownership
• synchronize grammar generations into the highlight worker before jobs run
• include grammar identity in caches and reset worker state after grammar changes or removal
• preserve plaintext fallback when grammar configuration or highlighting fails
• restore grammar registrations atomically when session reload fails
• dogfood the API with a self-contained Archlang syntax extension

Important boundaries

• grammars are presentation data, not executable extension loaders
• custom grammars cannot replace Hunk/Pierre bundled languages
• v1 grammars are self-contained; embedded languages and external scope includes are intentionally unsupported
• pathological regex protection is bounded by worker isolation and a 15-second worker restart timeout

Why TextMate grammar data

Pierre uses Shiki for lexical syntax highlighting, and Shiki consumes TextMate grammars evaluated with Oniguruma-compatible regex semantics. This API exposes a bounded, data-only subset of that existing format rather than introducing a Hunk-specific grammar system. Token scopes remain compatible with existing Shiki themes through hierarchical names such as keyword.control.archlang.

Benefits

• interoperates with the established VS Code, Shiki, and Sublime TextMate grammar ecosystem
• keeps grammar data serializable so it can cross into Hunk's highlight worker without retaining extension code or loader closures
• preserves Hunk's existing theme model and does not require language-specific theme entries
• isolates expensive or pathological grammar regexes from the terminal event loop
• supports complete replacement and cache invalidation when extensions reload or disappear

Costs and limitations

• TextMate grammars provide lexical highlighting, not a structural syntax tree; parser-backed semantics would require a different system such as Tree-sitter
• regex size limits cannot prove regex complexity, so the worker watchdog may terminate a pathological grammar after 15 seconds and fall back to plaintext
• the first worker highlight now performs one grammar-generation configuration handshake, even when the custom grammar set is empty
• rendered cache keys include one precomputed grammar digest lookup; grammar changes intentionally recreate worker/highlight state and invalidate caches
• expanded context, syntax-scope override themes, and environments without worker offload currently render custom languages as plaintext
• the checked-in Archlang example is a useful dogfood fixture but a made-up language; an established third-party grammar should also be exercised before merge to prove compatibility with real ecosystem grammar shapes

For existing bundled languages and themes, tokenization and theme resolution are unchanged. The expected steady-state overhead is limited to the precomputed digest in cache identity; no benchmark has yet quantified the one-time first-worker handshake against main.

Validation

• bun run typecheck
• bun run test — 2,134 passed, 3 skipped
• bun run test:integration — 139 passed, 1 skipped
• bun run test:tty-smoke — 9 passed
• bun run lint
• bun run deps:check
• bun run check:docs
• bun run website:check
• bun run build:npm
• bun run check:pack
• bun run changeset:status
• git diff --check origin/main...HEAD

Review

An independent code review found and drove fixes for worker incompatibility, current API drift, retired-loader authority, reload rollback, generator idempotence, and prototype-inherited local includes. Final review accepted the updated implementation with no blockers.

Platforms directly tested: Linux. Windows compiled fallback behavior is covered by predicates/tests but was not exercised on a real Windows standalone binary in this pass.

This PR description was generated by Pi using gpt-5.6-sol

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
hunk-web	Ignored	Preview	Sep 4, 2026 9:15pm UTC

[greptile-apps]

Greptile Summary

The PR adds extension API v3 support for lazily registered Shiki/TextMate syntax grammars and reconciles extension-provided file-language mappings across reloads.

• Adds syntax-language registration, validation, ownership, conflict handling, and attributed failure reporting.
• Serializes shared-highlighter lifecycle work and restores plaintext highlighting after grammar failures.
• Includes detected language in highlight cache identity.
• Updates extension declarations, package validation, tests, documentation, and release metadata.

Confidence Score: 5/5

The PR appears safe to merge; no concrete, changed-code-triggered defect remains.

The registration, application, fallback, cache invalidation, packaging, and public API paths are coordinated and covered by focused tests, with no established blocking or non-blocking failure.

Important Files Changed

Filename	Overview
src/extensions/apply.ts	Applies lazy syntax-language registrations with ownership checks, validation, idempotent reload behavior, and attributed errors.
src/core/fileLanguage.ts	Integrates Pierre’s custom grammar and replaceable extension-mapping registries while preserving Hunk’s built-in mappings.
src/extensions/runExtension.ts	Adds validated syntax registration to the extension API and includes it in factory rollback and sealing behavior.
src/ui/diff/pierre.ts	Serializes complete highlighter lifecycles and resets the shared highlighter before falling back to plaintext after failures.
src/ui/diff/useHighlightedDiff.ts	Adds the detected file language to highlight cache keys so language changes produce fresh results.
src/extension-api/types.ts	Advances the public contract to API v3 and exposes import-free grammar and loader types.
scripts/check-pack.ts	Exercises the new public API in package checks and guards declarations against both static and dynamic imports.

Sequence Diagram

sequenceDiagram
  participant E as Extension
  participant R as Extension Registry
  participant A as Apply Boundary
  participant P as Pierre
  participant U as Diff UI
  E->>R: registerSyntaxLanguage(id, loader)
  E->>R: registerFileLanguage(extension, id)
  A->>P: register lazy grammar loader
  A->>P: replace file-language mappings
  U->>P: prepare shared highlighter
  P->>E: invoke loader on first use
  alt grammar succeeds
    P-->>U: highlighted diff
  else grammar fails
    U->>P: dispose shared highlighter
    U->>P: prepare plaintext highlighter
    P-->>U: plaintext diff
  end

Loading

_{Reviews (1): Last reviewed commit: "feat(extensions): support custom syntax ..." | Re-trigger Greptile}

[benvinegar]

Archlang is a made-up language that usefully dogfoods the API, but it only proves that a small grammar we authored alongside the feature works. Before merge, we should also exercise an established third-party TextMate grammar against a representative source file. That would test compatibility with real grammar shapes, repositories, captures, and regex patterns rather than only our controlled fixture.

I suggest keeping the self-contained Archlang example for documentation, while adding a test fixture based on a real non-bundled language grammar with its license/provenance recorded. The test should verify registration, .ext association, worker transfer, visible token scopes, reload/removal, and plaintext fallback after rejection.

TextMate grammars being regex-driven is normal for Shiki: they conventionally use match, begin/end, captures, and repository includes interpreted with Oniguruma semantics. This API therefore follows the grammar format already consumed by Hunk's highlighter. It is lexical highlighting rather than structural parsing; Tree-sitter would be the alternative if Hunk wanted parser-backed grammars later.

This comment was generated by Pi using gpt-5.6-sol