fix(toolcall-schema-fuzz): score 400/422 rejections per sample and correct README

[abhinav-pola]

TL;DR

A stack that rejects a schema with HTTP 400/422 now fails just that sample as request_rejected, so the run keeps going. Before, the CLI aborted on the first rejection. The README now matches what the code does and includes a verified run command.

What changed?

• solver.ts: 400 and 422 responses become a completed state with rawResponse.rejected = { status, message }. 429/5xx still retry. Other 4xx (401, 402, 403, 404) still abort the run.
• scorer.ts: new request_rejected category, scored Incorrect. cli/report.ts picks it up through the explanation prefix.
• README.md:
  • Removes the claim that every case names a control (no realistic case does).
  • Notes that regressions.json is empty, so regressions errors and sentinel is just the coverage subset.
  • Documents request_rejected.
  • Adds a Running section with the CLI and report commands.

Why?

We're sharing this with external serving teams (NVIDIA Dynamo). Rejecting an unsupported schema keyword is a finding, so it shouldn't kill the run. For example, OpenAI rejects format: uri and $ref with sibling keywords.

How to test

OPENROUTER_API_KEY=... bun run bench --benchmark toolcall_schema_fuzz \
  --model openai/gpt-4.1 --solver-config '{"providerOnly":["openai"]}' --concurrency 32
bun src/benchmarks/toolcall-schema-fuzz/cli/report.ts bench-results/<run>.parquet

Ran this end to end: the run completed all 2668 samples with exit 0, 2598 correct and 70 request_rejected. On main it aborts at the first 'uri' is not a valid format 400.

Benchmark impact

Runs that used to abort now complete. Rejected samples score Incorrect. Scores for accepted samples don't change.

Reviewer focus

• Whether to treat any other 4xx besides 400/422 as a per-sample rejection.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/69d66439a0a140fbb61647ab67ead2ec
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/69d66439a0a140fbb61647ab67ead2ec?variant=devin
Requested by: @abhinav-pola

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1791592499.792659]: @Devin is <https://github.com/OpenRouterTeam/benchmark-harness/tree/main/src/benchmarks/toolcall-schema-fuzz|github.com/OpenRouterTeam/benchmark-harness/…/toolcall-schema-fuzz> ready to share with people externally? nvidia dynamo team wants a dataset and i dont want to give them slop lol
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-benchmarks) ===
Abhinav Pola (U090K0G7JF3) [ts=1791592499.792659]: @Devin is <https://github.com/OpenRouterTeam/benchmark-harness/tree/main/src/benchmarks/toolcall-schema-fuzz|github.com/OpenRouterTeam/benchmark-harness/…/toolcall-schema-fuzz> ready to share with people externally? nvidia dynamo team wants a dataset and i dont want to give them slop lol
=== END THREAD HISTORY ===
Channel ID: C0BAKP8P5C3
Thread URL: https://openrouter.slack.com/archives/C0BAKP8P5C3/p1791592499792659?thread_ts=1791592499.792659&amp;cid=C0BAKP8P5C3

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

[devin-ai-integration]

Devin Review found 1 potential issue.

[devin-ai-integration]

🔴 Invalid run configuration corrupts benchmark scores

When a run-wide option triggers HTTP 400, requestCompletion records every response as a sample rejection. The run completes with incorrect scores instead of exposing the invalid configuration.

Learn more

An HTTP 400 can reject the entire request configuration, not just a particular tool schema. This branch converts every 400 into a successful solver result. evaluateOne therefore scores it as a sample, and the run continues without detecting the shared error. A single invalid request option can make every case appear unsupported by the serving stack.

Example: If an invalid provider override makes the endpoint return HTTP 400 for every case, all 2,668 default cases receive request_rejected rather than stopping the run with an invalid-configuration error.

Recommended fix: Distinguish schema-specific validation failures from run-wide request/configuration failures using the response's validated error details. Only convert confirmed case-specific rejections to rejected; propagate other 400/422 responses as solver errors, and test both classes.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Leaving this as is. The run isn't silent: each sample keeps the full upstream error text in its explanation, and cli/report.ts groups failures by category with an example. A misconfigured run therefore shows up as 2668/2668 request_rejected, all carrying the same message. Separating "schema rejected" from "config rejected" by parsing the error body won't work across stacks, because OpenAI, Anthropic, vLLM and Dynamo all word their 400s differently. That parsing is exactly what an external team pointing this at their own server would need, and it isn't reliable.

Written by Devin