Skip to content

fix(toolcall-schema-fuzz): apply timeoutMs to reading the response body - #144

Merged
abhinav-pola merged 1 commit into
mainfrom
devin/1791576453-toolcall-fuzz-body-timeout
Oct 9, 2026
Merged

abhinav-pola merged 1 commit into
mainfrom
devin/1791576453-toolcall-fuzz-body-timeout

Conversation

@abhinav-pola

@abhinav-pola abhinav-pola commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR

timeoutMs in the toolcall_schema_fuzz solver now covers reading the response body, not just receiving headers. A server that sends headers and then stalls produces a 408 ModelError instead of hanging the sample.

What changed?

  • requestCompletion now executes the request, checks the status, and parses the JSON body inside one withTimeout boundary, and it returns the parsed body. The 408 ModelError conversion and the 429/5xx retry behaviour are unchanged.
  • solver.test.ts: the timeout test now runs against two fake servers, one that never responds and one that sends 200 headers and then stalls the body.

Why?

Found by Devin Review on openrouter-web#50308, which vendors #143. Without the stalled-body case passing, timeoutMs stops applying as soon as headers arrive. I confirmed the new stalled-body test fails without this fix.

Reviewer focus

  • Retries now re-run the full request and body read, since the body read is inside retrySalted.

Checklist

  • Tests cover changed behavior
  • Public API or configuration changes are backward compatible, or the break is documented
  • Benchmark changes document dataset provenance and licensing
  • No credentials, private results, or restricted dataset contents are included
  • Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/72a3f13c847645c5b06306d88cec089d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/72a3f13c847645c5b06306d88cec089d?variant=devin
Requested by: @abhinav-pola


Devin Review

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-benchmarks) ===
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
=== END THREAD HISTORY ===
Channel ID: C0BAKP8P5C3
Thread URL: https://openrouter.slack.com/archives/C0BAKP8P5C3/p1791572889759909?thread_ts=1791572889.759909&amp;cid=C0BAKP8P5C3

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@abhinav-pola
abhinav-pola merged commit 5f32560 into main Oct 9, 2026
5 checks passed
@abhinav-pola
abhinav-pola deleted the devin/1791576453-toolcall-fuzz-body-timeout branch October 9, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant