Skip to content

Auto-translate: send the advanced engines' batch JSON as raw UTF-8 instead of \uXXXX escapes - #16830

Merged
niksedk merged 1 commit into
SubtitleEdit:mainfrom
Makar8000:claude/advanced-engine-raw-utf8-batch-json
Nov 11, 2026
Merged

niksedk merged 1 commit into
SubtitleEdit:mainfrom
Makar8000:claude/advanced-engine-raw-utf8-batch-json

Conversation

@Makar8000

@Makar8000 Makar8000 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

The advanced engines (GUI llama.cpp / Ollama / LM Studio advanced, seconv *-advanced) put each batch into the user message as a JSON object, {"history":[...],"lines":[{"n":1,"text":"..."}]}. That object is built with a Utf8JsonWriter on the default encoder, which writes every character above U+007F (and <, >, &, ', +) as a \uXXXX escape. The server decodes only the request's outer JSON layer, so those escapes reach the model as literal text: where the subtitle said 你昨晚在哪里? the model read the 42 ASCII characters \u4F60\u6628\u665A\u5728\u54EA\u91CC\uFF1F.

Effect

  • Every source with non-ASCII text is hit, in the GUI and in seconv alike: CJK scripts entirely, Cyrillic, Greek, Arabic, Hebrew, Thai, and Latin languages wherever they use accents or typographic punctuation. The default encoder also escapes ASCII quotes, apostrophes, angle brackets, ampersands and plus signs, so even English text with a contraction or an italics tag reached the model with escapes in it. History pairs are affected on both sides, so a Latin source with a non-Latin target loses the consistency the history is there for.
  • Small and translation-tuned models cannot read the escapes. Hy-MT2 7B answered a Chinese batch with fluent English unrelated to the source ("The battle began in the dense forest." for "Where were you last night?"), and because the reply is schema-forced to the batch's line numbers, every guard in AdvancedTranslatorBase passed and the file was reported as translated. Large chat models partly decode the escapes themselves, which is why English-source tests and big models never showed it.
  • The system prompt (synopsis, glossary) is written into the outer body and was decoded by the server, so it reached the model intact. That is how glossary terms turned up inside invented sentences.

Changes

  • LlamaCppAdvancedProtocol gets a shared JsonWriterOptions with JavaScriptEncoder.UnsafeRelaxedJsonEscaping, used by BuildUserContent (the fix), BuildResponseFormatJson (digits only, changed for consistency) and the client's BuildRequestJson (the outer body, so captures and logs are readable and the user content is escaped once). Quotes, backslashes and control characters are still escaped; the JSON stays valid and ParseTranslations and the server-side parsers are unaffected. "Unsafe" only concerns JSON embedded in HTML, which this never is. JavaScriptEncoder.Create(UnicodeRanges.All) was not enough: it still escapes quotes, apostrophes, angle brackets, ampersands and plus signs. The relaxed encoder is what seconv's JSON output and the language files already use. Characters outside the Basic Multilingual Plane (emoji), U+2028, U+2029 and DEL stay escaped with every JavaScriptEncoder; the reply is parsed as JSON, so those come back intact in the subtitle.
  • AiReviewProtocol.BuildUserContent and AiReviewClient embed the chunk JSON the same way and get the same options.
  • Tests: the user content, the request body (decoded one layer, as the server does) and the AI Review chunk keep Chinese, Japanese and accented text readable with no \u escape, quotes still escaped, and the response schema still parses.

Verification

  • LibUiLogicTests 1641 passed (3 new), the UI's AI Review tests 90 passed (1 new), SeConvTests 711 passed.
  • seconv llamacpp-advanced, --translate-from:zh-CN --translate-to:en, llama-server with Hy-MT2 7B, an 8-line Chinese sample, requests captured through a logging proxy, same server and settings before and after:
Before After, 7B After, 1.8B
\uXXXX sequences in the user message the model reads 67 0 0
Raw CJK characters the model reads 0 57 57
Requests 1 1 1
Lines translated correctly, of 8 0 8 8

Before: 你昨晚在哪里? became "The battle began in the dense forest." and the other seven lines continued the invented battle, reported as a successful conversion. After: "Where were you last night?", "I stayed at the office until late.", "Can anyone prove it?", "The security guard saw me leave." and so on, with the 1.8B model as good as the 7B on this sample.

  • Drafted with AI assistance (Claude Code); the tests and runs above were performed as listed.

🤖 Generated with Claude Code

…stead of \uXXXX escapes

The advanced engines put each batch into the user message as a JSON object
({"history":[...],"lines":[{"n":1,"text":"..."}]}) built with a
Utf8JsonWriter on the default encoder, which writes every non-ASCII
character as a \uXXXX escape. The server decodes only the request's outer
JSON layer, so those escapes reach the model as literal text: a Chinese
line arrived as "\u4F60\u6628\u665A...". Hy-MT2 7B answered every such
batch with fluent English unrelated to the source, and because the reply
is schema-forced to the batch's line numbers every guard passed and the
file was reported as translated. Any source with non-ASCII text is hit,
CJK scripts entirely and Latin languages wherever they use accents or
typographic punctuation, in the GUI and in seconv alike; history pairs are
affected on both sides.

The model-facing writers now use JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
which keeps quotes, backslashes and control characters escaped and the JSON
valid, and leaves text readable. The outer request body gets the same
options so captures and logs are readable and the user content is escaped
once. The AI Review protocol embeds its chunk JSON the same way and gets the
same change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Makar8000
Makar8000 force-pushed the claude/advanced-engine-raw-utf8-batch-json branch from 854b11c to 5aba8ff Compare November 11, 2026 08:26
@niksedk
niksedk requested a balanced review from Copilot November 11, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified in the JSON-encoding changes for preserving readable multilingual subtitle text.

No open issues 🎉

What changed in this PR

This PR keeps non-ASCII subtitle text readable in the nested JSON messages sent by advanced translation engines and AI Review, rather than exposing models to literal Unicode escape sequences.

Changes:

  • Applies relaxed JSON escaping to translation batches, review chunks, and their outer request bodies.
  • Reuses the translation writer options for response-schema generation.
  • Adds regression tests for multilingual text, quote escaping, decoded request content, and JSON validity.
File Description
tests/​UI/​Features/​Tools/​AiReview/​AiReviewProtocolTests.cs Tests readable multilingual review text and actor names.
tests/​libuilogic/​Translate/​LlamaCppAdvancedProtocolEncodingTests.cs Tests batch encoding, request content, and response-schema JSON.
src/​ui/​Features/​Tools/​AiReview/​AiReviewProtocol.cs Defines relaxed writer options and applies them to review chunks.
src/​ui/​Features/​Tools/​AiReview/​AiReviewClient.cs Applies review writer options to request bodies.
src/​libuilogic/​Translate/​LlamaCppAdvanced/​LlamaCppAdvancedProtocol.cs Applies shared relaxed writer options to batches and response schemas.
src/​libuilogic/​Translate/​LlamaCppAdvanced/​LlamaCppAdvancedClient.cs Applies translation writer options to request bodies.

Review effort: Balanced

@niksedk
niksedk merged commit f87830e into SubtitleEdit:main Nov 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants