report detector input provenance in translated hitlogs - #2002
report detector input provenance in translated hitlogs#2002SarveshaKumarKS wants to merge 1 commit into
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Sarvesha Kumar Kombaiah Seetha <202573753+SarveshaKumarKS@users.noreply.github.com>
jmartin-tech
left a comment
jmartin-tech
left a comment
There was a problem hiding this comment.
This is definitely a gap that needs some attention, the changes here impact areas that I think may be a bit too coupled to a specific view.
The idea that an Attempt needs to hold onto details of the detector actions is a bit more than just knowing if translation was involved and will likely be best to address in a more complex feature.
I have offered a more generic serialization idea that aligns hitlog entry changes with how existing Attempt data is currently being stored.
There was a problem hiding this comment.
This value is not needed, the attempt does not need to store this information.
Translation either exists on the attempt or does not, the simple path forward here is to just serialize a original response and the translation for each hit.
There is another core issue related to this, at some point detector results should come with evidence from detection. That support is a more complex enhancement that needs standardization.
There was a problem hiding this comment.
This value is not needed, the attempt does not need to store this information.
There was a problem hiding this comment.
This value is not needed, the attempt does not need to store this information.
| detector_input_record = attempt._detector_inputs.get(detector_name) | ||
| if ( | ||
| detector_input_record is not None | ||
| and detector_input_record["source"] | ||
| == "reverse_translation_outputs" | ||
| and idx < len(detector_input_record["outputs"]) | ||
| ): | ||
| detector_input = detector_input_record["outputs"][idx] | ||
| hitlog_record["detector_input"] = ( | ||
| asdict(detector_input) if detector_input else None | ||
| ) | ||
| hitlog_record["detector_input_source"] = detector_input_record[ | ||
| "source" | ||
| ] |
There was a problem hiding this comment.
Translation either exists on the attempt or does not, the simple path forward here is to just serialize a original response and the translation for each hit.
reverse_translation_output = None
if (
len(attempt.reverse_translation_outputs) >= idx and attempt.reverse_translation_outputs[idx]
):
reverse_translation_output = attempt.reverse_translation_outputs[idx]
hitlog_record["reverse_translation_output"] = reverse_translation_output
jmartin-tech
added
reporting
2 participants
Summary
detector_inputanddetector_input_sourcefields to translated-run hitlog recordsoutputfield as the raw model generationTreeSearchProbedetector pathWhy
Language-aware detectors can score
reverse_translation_outputswhile the evaluator only recordsattempt.outputsin the standalone hitlog. The score remains positionally aligned, but the evidence that produced a translated-run hit is missing from that record.This carries detector-input provenance through the attempt so the evaluator does not need to infer a detector's language scope after the fact. Raw-run hitlog schema remains unchanged; the new fields are emitted only when reverse-translated input was selected.
Closes #2001.
Duplicate check
Before implementation I checked issue #2001 and searched open pull requests for both
2001 in:bodyandhitlog detector input translation; no competing PR or claim was present. #1959 covers positional alignment, while #1174, #1201, and #1826 concern broader multilingual or serialization behavior rather than detector-scoped hitlog evidence.Tests
python -m pytest -q tests/evaluators/test_evaluators.py— 49 passedpython -m pytest -q tests/harnesses/test_harnesses.py— 5 passedpython -m pytest -q tests/test_attempt.py— 23 passedpython -m black --check -W 1 ...for all six changed files — passedtests/probes/test_probes.py— 1,347 passed; 9 unrelated metadata cases require unavailable optional packages or network-fetched fixturesAI assistance
OpenAI Codex assisted with investigation, implementation, tests, and drafting this pull request. The human submitter will review every changed line before marking the PR ready and remains responsible for the contribution.