Repository navigation
Auto-translate: send the advanced engines' batch JSON as raw UTF-8 instead of \uXXXX escapes - #16830
Merged
niksedk merged 1 commit intoNov 11, 2026
Conversation
…stead of \uXXXX escapes
The advanced engines put each batch into the user message as a JSON object
({"history":[...],"lines":[{"n":1,"text":"..."}]}) built with a
Utf8JsonWriter on the default encoder, which writes every non-ASCII
character as a \uXXXX escape. The server decodes only the request's outer
JSON layer, so those escapes reach the model as literal text: a Chinese
line arrived as "\u4F60\u6628\u665A...". Hy-MT2 7B answered every such
batch with fluent English unrelated to the source, and because the reply
is schema-forced to the batch's line numbers every guard passed and the
file was reported as translated. Any source with non-ASCII text is hit,
CJK scripts entirely and Latin languages wherever they use accents or
typographic punctuation, in the GUI and in seconv alike; history pairs are
affected on both sides.
The model-facing writers now use JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
which keeps quotes, backslashes and control characters escaped and the JSON
valid, and leaves text readable. The outer request body gets the same
options so captures and logs are readable and the user content is escaped
once. The AI Review protocol embeds its chunk JSON the same way and gets the
same change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Makar8000
force-pushed
the
claude/advanced-engine-raw-utf8-batch-json
branch
from
November 11, 2026 08:26
854b11c to
5aba8ff
Compare
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were identified in the JSON-encoding changes for preserving readable multilingual subtitle text.
No open issues 🎉
What changed in this PR
This PR keeps non-ASCII subtitle text readable in the nested JSON messages sent by advanced translation engines and AI Review, rather than exposing models to literal Unicode escape sequences.
Changes:
- Applies relaxed JSON escaping to translation batches, review chunks, and their outer request bodies.
- Reuses the translation writer options for response-schema generation.
- Adds regression tests for multilingual text, quote escaping, decoded request content, and JSON validity.
| File | Description |
|---|---|
tests/UI/Features/Tools/AiReview/AiReviewProtocolTests.cs |
Tests readable multilingual review text and actor names. |
tests/libuilogic/Translate/LlamaCppAdvancedProtocolEncodingTests.cs |
Tests batch encoding, request content, and response-schema JSON. |
src/ui/Features/Tools/AiReview/AiReviewProtocol.cs |
Defines relaxed writer options and applies them to review chunks. |
src/ui/Features/Tools/AiReview/AiReviewClient.cs |
Applies review writer options to request bodies. |
src/libuilogic/Translate/LlamaCppAdvanced/LlamaCppAdvancedProtocol.cs |
Applies shared relaxed writer options to batches and response schemas. |
src/libuilogic/Translate/LlamaCppAdvanced/LlamaCppAdvancedClient.cs |
Applies translation writer options to request bodies. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The advanced engines (GUI llama.cpp / Ollama / LM Studio advanced, seconv
*-advanced) put each batch into the user message as a JSON object,{"history":[...],"lines":[{"n":1,"text":"..."}]}. That object is built with aUtf8JsonWriteron the default encoder, which writes every character above U+007F (and<,>,&,',+) as a\uXXXXescape. The server decodes only the request's outer JSON layer, so those escapes reach the model as literal text: where the subtitle said 你昨晚在哪里? the model read the 42 ASCII characters\u4F60\u6628\u665A\u5728\u54EA\u91CC\uFF1F.Effect
AdvancedTranslatorBasepassed and the file was reported as translated. Large chat models partly decode the escapes themselves, which is why English-source tests and big models never showed it.Changes
LlamaCppAdvancedProtocolgets a sharedJsonWriterOptionswithJavaScriptEncoder.UnsafeRelaxedJsonEscaping, used byBuildUserContent(the fix),BuildResponseFormatJson(digits only, changed for consistency) and the client'sBuildRequestJson(the outer body, so captures and logs are readable and the user content is escaped once). Quotes, backslashes and control characters are still escaped; the JSON stays valid andParseTranslationsand the server-side parsers are unaffected. "Unsafe" only concerns JSON embedded in HTML, which this never is.JavaScriptEncoder.Create(UnicodeRanges.All)was not enough: it still escapes quotes, apostrophes, angle brackets, ampersands and plus signs. The relaxed encoder is what seconv's JSON output and the language files already use. Characters outside the Basic Multilingual Plane (emoji), U+2028, U+2029 and DEL stay escaped with everyJavaScriptEncoder; the reply is parsed as JSON, so those come back intact in the subtitle.AiReviewProtocol.BuildUserContentandAiReviewClientembed the chunk JSON the same way and get the same options.\uescape, quotes still escaped, and the response schema still parses.Verification
llamacpp-advanced,--translate-from:zh-CN --translate-to:en, llama-server with Hy-MT2 7B, an 8-line Chinese sample, requests captured through a logging proxy, same server and settings before and after:\uXXXXsequences in the user message the model readsBefore: 你昨晚在哪里? became "The battle began in the dense forest." and the other seven lines continued the invented battle, reported as a successful conversion. After: "Where were you last night?", "I stayed at the office until late.", "Can anyone prove it?", "The security guard saw me leave." and so on, with the 1.8B model as good as the 7B on this sample.
🤖 Generated with Claude Code