feat(toolcall-schema-fuzz): move toolcall_schema_fuzz benchmark into the harness

[abhinav-pola]

TL;DR

Moves the whole toolcall_schema_fuzz benchmark (dataset, cases, solver, scorer, generators, CLIs) from openrouter-web internal-benchmarks into the harness and registers it as a catalog benchmark.

What changed?

• src/benchmarks/toolcall-schema-fuzz/: grid + fast-check arbs (grid.ts, arb/*, shape.ts, constructs.ts), cases.ts/sentinel.ts/regressions.ts (+ regressions.json), realistic rewording (realistic/*, realistic/wordings.json), json-schema.ts validator, solver.ts, scorer.ts, dataset.ts, benchmark.ts, and the cli/discover.ts / cli/reword.ts / cli/report.ts tools, with colocated tests.
• Registered in benchmark-config.ts (ToolCallSchemaFuzzBenchmarkConfigSchema, generator: realistic | sentinel | regressions, default realistic), benchmark-meta.ts (temperature 0, 1 epoch), registry.ts, and the CLI.
• Internal monorepo deps swapped for harness equivalents (internal/guards, internal/zod, internal/either, provider/retry helpers); fast-check added as a dependency.
• The earlier standalone src/datasets/toolcall-schema-fuzz export is dropped; the JSON now lives with the benchmark.

Why?

The benchmark only ran from the private monorepo. Moving it public lets Kepler run it like every other harness benchmark; openrouter-web will pick it up via scripts/subtree-pull-bench-harness.sh and delete its internal copy.

How to test

bun run bench --benchmark toolcall_schema_fuzz --model google/gemini-2.5-flash --limit 20 --concurrency 4
bun src/benchmarks/toolcall-schema-fuzz/cli/report.ts bench-results/toolcall_schema_fuzz-*.parquet

Local smoke: 18/20 passing on google/gemini-2.5-flash (first 20 realistic samples, single scenario).

Benchmark impact

New benchmark id in the public catalog. Sample ids and generator contents match the internal version; scoring classes are unchanged.

Reviewer focus

• solver.ts: talks to /chat/completions directly (tool history replay, parallel_tool_calls, endpoint/provider routing) rather than through ModelService.
• Copied comments were removed to satisfy openrouter(no-comments); context moved to README.md.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/72a3f13c847645c5b06306d88cec089d
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/72a3f13c847645c5b06306d88cec089d?variant=devin
Requested by: @abhinav-pola

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring

---

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
</latest_message>

=== BEGIN THREAD HISTORY (in #agents-benchmarks) ===
Abhinav Pola (U090K0G7JF3) [ts=1791572889.759909]: @Devin move the tool call fuzz dataset to the public repo and sync it back to the subtree
=== END THREAD HISTORY ===
Channel ID: C0BAKP8P5C3
Thread URL: https://openrouter.slack.com/archives/C0BAKP8P5C3/p1791572889759909?thread_ts=1791572889.759909&amp;cid=C0BAKP8P5C3

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)