Repository navigation
DRU-613 -- Skip unparseable lines in the pi stream - #1154
p-r-a-v-i-n wants to merge 1 commit into
Conversation
czpython
left a comment
There was a problem hiding this comment.
The skip is right, and so is ValueError: a cut inside a UTF-8 character raises UnicodeDecodeError, which is not a JSONDecodeError.
One change: drop stray text as a cause. In print and json mode, pi 1.5.0 points process.stdout.write at stderr (takeOverStdout in dist/main.js). Only the session header and the events go to the real stdout. A run on the sandbox image with this harness's flags and a test extension shows it: console.log, console.info and process.stdout.write went to stderr, and process.emitWarning printed nothing. So the only bad line is the first line after the buffer trim. The two inline suggestions fix the comment and the test. Please also drop the warning as a cause in the description.
Not blocking: when stdout passes STDOUT_BUFFER_LIMIT, the call now passes, but the cost counts only the message_end events in the kept tail. Long commands make this common. pi's bash tool sends 10 updates per second, each up to 103 KiB, so steady output passes 50 MiB in less than a minute. With the real _tee_stream and this parse, a 76 MiB stream records a cost of 8.0 out of 13.0. On main the same stream fails and records nothing, so this PR is still better. The full stream is in <artifact_dir>/<run_id>/stdout.jsonl, and parsing that file gives 13.0.
7fcd5d8 to
425aab2
Compare
PiHarness.parsenow skips a stdout line that is not JSON. Before, it decoded every line in one list comprehension, so a single bad line raisedJSONDecodeError. Theexcept ValueErroraround the parse turned that intoHarnessInvalidOutputError("pi wrote no usable stream."), and the call failed even when pi exited 0 and wroteoutput.json. That error is retryable, so the agent call ran again.The bad line is the first line after the sandbox trims the front of its stdout buffer at
STDOUT_BUFFER_LIMIT. The loop now skips a line that does not decode, ascollapse_claude_streamdoes for Claude. It suppressesValueError, not onlyJSONDecodeError, because a cut inside a multibyte character makesjson.loadsraiseUnicodeDecodeError. The cost still comes from themessage_endevents that decode, and the result still comes fromoutput.json.A stdout with no events at all no longer fails on the stream. After a zero exit, the call fails only when
output.jsonis missing. The test that expected a stream with no JSON to fail on the stream is removed for that reason. A new test feeds a line cut inside a multibyte character before onemessage_endevent, and checks that the result and the cost come back. It fails if the skip catches onlyJSONDecodeError.