Skip to content

fix: localize repeated LLM spans distinctly - #87

Closed
kocaemre wants to merge 1 commit into
KRLabsOrg:mainfrom
kocaemre:fix/repeated-llm-spans
Closed

fix: localize repeated LLM spans distinctly#87
kocaemre wants to merge 1 commit into
KRLabsOrg:mainfrom
kocaemre:fix/repeated-llm-spans

Conversation

@kocaemre

@kocaemre kocaemre commented Jul 26, 2026

Copy link
Copy Markdown

Summary

  • Map repeated LLM-returned span strings to distinct unused answer occurrences in left-to-right order.
  • Reserve occurrences before applying is_hallucination and confidence filters so later accepted items keep stable offsets.
  • Update generic LLM response instructions to ask for answer-order, one item per distinct occurrence.
  • Add network-free regression tests for repeated strings, metadata preservation, filtering, duplicate overflow, first-match fallback, and token projection.

Why

The generic LLM judge path used a fresh first-match search for every returned substring. When the model returned the same text more than once, all items collapsed onto the first occurrence, so later repeated hallucinations were never represented in spans or token output.

Fixes #85.

Test plan

  • python3 -m pytest tests/test_llm_detector_pytest.py::TestRepeatedSpanLocalization -q

  • python3 tests/run_pytest.py -q — 149 passed

  • ruff check lettucedetect/detectors/llm.py tests/test_llm_detector_pytest.py

  • git diff --check

  • I certify that I have the right to submit this code.

Signed-off-by: Emre K <110906681+kocaemre@users.noreply.github.com>
@kocaemre
kocaemre force-pushed the fix/repeated-llm-spans branch from b10c7ec to 2245530 Compare July 31, 2026 16:10
@adaamko

adaamko commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Thanks — this is careful work, and the test coverage here is broader than what just merged. It crossed with #86 though, which was opened four days earlier for the same issue (#85) with the same reservation design, so first-come wins and #86 is now in main. Two parts of yours are genuinely additive and didn't land with #86: the response-format notes in _RESPONSE_FORMAT/_response_format_block (86 only updated the detection prompt), and several of your edge-case tests (metadata pairing, min_confidence reservation, the documented first-match fallback). A rebased follow-up PR with those would be very welcome — just include the rights-attestation line from the PR template, which this PR is also missing. Closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Map repeated LLM span strings to distinct answer occurrences

2 participants