Skip to content

Auto-translate: accept Chinese/Japanese batch replies that drop the "." separators (#16386) - #16824

Merged
niksedk merged 1 commit into
SubtitleEdit:mainfrom
Makar8000:claude/cjk-batch-line-count-fallback-16386
Nov 11, 2026
Merged

niksedk merged 1 commit into
SubtitleEdit:mainfrom
Makar8000:claude/cjk-batch-line-count-fallback-16386

Conversation

@Makar8000

Copy link
Copy Markdown
Contributor

Fixes #16386. Sources without sentence endings (Chinese, Japanese) are sent as rows joined by lone . separator lines, and models often drop some or all of those separators and answer with one line per row. The line-count fallback added in 5cac6fb was meant to accept such a reply but never did, so every affected batch was retried and then translated row by row.

Cause

  • ProcessNoSentenceEndingRow appended the next row (behind a separator) to the item it was closing, so every item but the last carried two rows. TrySplitByLineCount then dealt two reply lines to the first item, ran out of lines for the last ones, and HasSameEmptyLines rejected the reply. The same inflated item text skewed HasPlausibleProportions for these sources.
  • The fallback's Chinese/Japanese branch only opened when the reply contained no . line at all, so a reply that kept some separators was rejected before any mapping was tried.

Changes

  • Each item now holds its own row only; the request text still carries the separators (one line removed).
  • TrySplitByLineCount drops the . lines from the reply and compares content lines with the request's, for replies that kept none or only some separators. A reply with the wrong number of content lines, merged or split rows, or a row with an internal line break answered on one line, is still rejected.
  • Tests: a reply with no separators and one with a separator missing are accepted for zh and ja; merged and split replies and the line-break case are rejected; the merge step's items are pinned to their own rows. The one-separator-missing reply that MergeAndTranslateIfPossible_InvalidPeriodMarkerSplitDoesNotUpdateRows asserted as rejected is exactly the shape this issue asks to accept, so that case moved to the new accepted theory; the extra-row and doubled-separator cases stay rejected.

Verification

  • LibUiLogicTests 1638 passed, 94 of them in MergeAndSplitHelperTests (8 new cases, 2 moved); SeConvTests unchanged and passing.
  • Real runs of the regular llamacpp engine with Hy-MT2 7B and its official prompt on two Chinese WebVTT scripts (38 and 146 cues), requests counted through a logging proxy, same model, prompt and files before and after:
38 cues, before 38 cues, after 146 cues, before 146 cues, after
Requests 5 4 68 9
Batch replies that dropped separators 1 of 5 1 of 3 14 of 20 6 of 9
Single-row requests 0 1 48 0
Cues left untranslated 0 0 1 0
Time 12 s 10 s 44 s 26 s

Every cue kept its timing in all four outputs. Before the fix, each batch reply with a dropped separator was rejected and its rows translated one request each; after it, those replies are mapped by line count and the row-by-row retries disappear.

  • Drafted with AI assistance (Claude Code); the tests and runs above were performed as listed.

🤖 Generated with Claude Code

…." separators (SubtitleEdit#16386)

Sources without sentence endings are sent as rows joined by lone "." lines,
and models often drop some or all of those separators and answer with one
line per row. The line-count fallback meant to accept such a reply never
did, for two reasons:

- ProcessNoSentenceEndingRow appended the next row (behind a separator) to
  the item it was closing, so every item but the last carried two rows. The
  fallback then dealt two reply lines to the first item, ran out of lines
  for the last ones, and the empty-line check rejected the reply. Each item
  now holds its own row only; the request text still carries the separators.

- TrySplitByLineCount only opened when the reply had no "." line at all. It
  now drops the separator lines from the reply and compares content lines
  with the request's, so a reply that kept some separators is mapped too.

A reply with the wrong number of content lines (merged or split rows, or a
row with an internal line break answered on one line) is still rejected.
The one-separator-missing reply the previous tests asserted as rejected is
exactly the shape this issue asks to accept, so that case moves to the new
accepted theory.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@niksedk
niksedk merged commit 1c8aa15 into SubtitleEdit:main Nov 11, 2026
4 checks passed
@Makar8000
Makar8000 deleted the claude/cjk-batch-line-count-fallback-16386 branch November 11, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants