Skip to content

DRU-1092 -- Strip OpenAI citation markers from Codex output - #1155

Merged
czpython merged 1 commit into
czpython:mainfrom
p-r-a-v-i-n:dru-1092-strip-codex-citations
Nov 10, 2026
Merged

czpython merged 1 commit into
czpython:mainfrom
p-r-a-v-i-n:dru-1092-strip-codex-citations

Conversation

@p-r-a-v-i-n

@p-r-a-v-i-n p-r-a-v-i-n commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

CodexHarness.parse now removes OpenAI citation markers from a Codex result. Before, a Codex run that searched the web could leave markers in its result: U+E200 cite, a U+E202 reference ID, and U+E201. They showed as stray text in saved results and in the prompts of later runs that read those results.

  • parse serializes the payload with ensure_ascii=False, removes each marker and the spaces before it with _CITATION_MARKER, and loads the JSON again. One pass covers every nested string.
  • The pattern stops at a quote, so a match stays inside one JSON string. A marker with no closing U+E201 is left in place, and the JSON stays valid.
  • output.json is written from the cleaned payload.

@p-r-a-v-i-n
p-r-a-v-i-n marked this pull request as draft November 10, 2026 05:20
@p-r-a-v-i-n
p-r-a-v-i-n marked this pull request as ready for review November 10, 2026 05:40
@p-r-a-v-i-n
p-r-a-v-i-n marked this pull request as draft November 10, 2026 05:41
@p-r-a-v-i-n
p-r-a-v-i-n marked this pull request as ready for review November 10, 2026 05:50

@czpython czpython left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The round trip in parse is correct. Checked with quotes, backslashes, newlines, nested lists, and a marker written as a \ue200 escape. Each one comes back clean and valid.

The pattern needs two changes. See the inline comments. The test needs the same escapes.

Comment thread backend/druks/harnesses/codex.py Resolved
Comment thread backend/tests/test_cost_capture.py Resolved
@p-r-a-v-i-n
p-r-a-v-i-n force-pushed the dru-1092-strip-codex-citations branch from dc0bb87 to 95b542e Compare November 10, 2026 13:37
@p-r-a-v-i-n
p-r-a-v-i-n requested a review from czpython November 10, 2026 13:56
@czpython
czpython merged commit 52f80a1 into czpython:main Nov 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants