Repository navigation
test: run builder's evals on Codex, Cursor and opencode, and fill the support table #17
Description
What
builder 1.0 ships adapters for Codex CLI, Cursor CLI and opencode (#16), but no eval has run on them yet. So the README support table lists no harness. This issue runs the remaining evals and fills the table.
The runs
- Pilot: one fixture, with builder only, on each of Codex, Cursor and opencode. Check:
- Codex: does the plugin
SessionStarthook fire incodex exec, before or after hook trust? Dump the hook env and add a Codex marker toharness.detect()(it falls back to the Claude shape today). - Codex: is the real
auth.jsonstill valid after the run? The runner links it, and a token refresh may replace the link. - Cursor: does
agent -pread the prompt on stdin? Does it accept--sandbox enabledwith--force? Does it honour the.cursor/cli.jsondeny rules? - opencode: does
opencode runread the prompt on stdin? Is thepermissionpattern syntax right? Does the plugin inject the soul in a live session (only fake-client tests cover it today)? - After each run:
git statusof the builder checkout stays clean.
- Codex: does the plugin
- Full: the other 7 fixtures, with builder only, on each harness (
bash tests/evals/run.sh --harness <h> <fixture>). - Parity:
bash tests/evals/parity.sh(it needs Claude Code signed in inside a separate config folder).
Done when
- Each run is graded bullet by bullet and recorded with
python3 scripts/support-table.py record. - The README block is rendered from
tests/evals/results/(support-table.py render), andvalidate.shpasses. - A harness shows
supportedonly when every fixture meets every bullet. Otherwise it showsexperimentalwith the reason.
Cost
The Codex and Cursor runs use plan quota. The opencode runs use whatever provider opencode is set up with (pay per token on an API key).
Activity
Lutherwaves commentedon Oct 22, 2026
Claude Code results (2026-10-22, before merge of #16)
All 8 fixtures ran with
claude -p, once without builder and once with it. With builder, the runs met 32 of 41 bullets; without it, 7. The README records Claude Code asexperimentaluntil every bullet is met.Bullets not met:
expect.mdnow uses neutral wording.)-ptext transcript holds only the final message, so the spec's anti-goals, the cost of each recommendation, the "Left to the builder" section and the ruling dates do not show. Check first whether the fixture can be graded from a text transcript at all.Nobody knows yet whether these misses were there before #16. No earlier runs exist for comparison. Run the same fixtures on
v0.1.4for a baseline.Also, the
reachfixture's allow ruleBash(python3 *lib/reach.py check *)makes Claude Code warn that the leading wildcard approves inserted options. Narrow it.docs: the README shows how builder did on Claude Code's evals, from t…
Lutherwaves commentedon Oct 22, 2026
Also check in the Cursor pilot: does Cursor set and expand
${CURSOR_PLUGIN_ROOT}for plugin hooks?hooks/hooks-cursor.jsonnow runs"${CURSOR_PLUGIN_ROOT}/hooks/session-start" --harness cursor(09fc85f). Without the variable, the hook fails closed and the soul is not loaded.docs: the README shows how builder did on Claude Code's evals, from t…
feat!: builder 2 — the builder soul, agent-led setup and peer messagi…