+ Our aim was a like-for-like comparison with the way Ruby on Rails' own + ai-evals measures agents: the same harness, the + same agent, a comparable sample app, and the same style of task with hidden acceptance checks. Where + we couldn't match Rails exactly, we say so. +
+ +1. What we measured, and why
+-
+
-
+ Same harness and agent as Rails' ai-evals. We used Rails'
+
lemansharness and itsminiswenagent unchanged, so a Wheels task is + run and scored the same way a Rails task is. +
+ - + A comparable sample app. Tasks run against a Writebook-style content app built + in Wheels — the same kind of app Rails' Stage 1 uses — so the asks line up. + +
- + Frozen tasks with hidden checks. Each task is a plain-English feature request. + The agent gets the prompt and nothing else; a hidden spec decides pass/fail. A task scores 1 only + if the app's own test suite is green and every hidden check passes. Tasks and + their checks were frozen and independently reviewed before any model ran. The only changes after + any model ran were the two over-strict check corrections disclosed in section 5, each reported + with both the before and after figures. + +
-
+ Two arms. Each model ran twice where possible: bare (a stock
+ Wheels app) and shipped docs (the same app with the AI context Wheels now ships
+ —
.ai/docs and a prebuilt.mcp.json). This isolates whether the + shipped context helps. +
+ - + Agent-native rows reported separately. Alongside the like-for-like rows we ran + Claude Code against the same frozen tasks, + via the "Harbor" setup. Those rows are a different agent and are always reported and labelled + separately — never pooled with the like-for-like numbers. + +
+ Ten tasks (plus a hello-world control) × three attempts per cell. A "cell" is one model in one + arm. Twenty-four cells in all: nine models in both arms, four more in the shipped arm only, + and two Claude Code cells. +
+++ ++ Integrity first. The case we wanted to test honestly is simply whether Wheels + deserves a seat at the table. So: no benchmark-specific caching, no pre-generated answers, and + no app state an agent wouldn't have in normal use. Every improvement we made to help agents shipped + to every Wheels user — in
+wheels new, the docs, and the MCP + server — not just to the test. Every run was recorded, including failures and infrastructure + errors, and those are reported below. +
2. Latest results (October 2026)
++ Overall, across the ten tasks, the agents passed 700 of 790 scored trials (88.6%) — raw, before two hidden-check corrections (section 5), 660 of 790 (83.5%). +
++ Steps are the agent's tool-use turns (a guard stopped a trial at 150). Spend is the settled + provider cost for that cell. The "docs" arm is the shipped AI context; "bare" is a stock app. +
+| Model | Provider | Agent | Arm | +Pass (raw) | Spend | Steps (mean) | +
|---|---|---|---|---|---|---|
| GPT-5.6 Luna | OpenAI | lemans | bare | 19/33 (18) | $0.70 | 15 |
| GPT-5.6 Luna | OpenAI | lemans | docs | 21/33 (20) | $0.71 | 16 |
| DeepSeek V4.1 Flash | DeepSeek | lemans | bare | 32/33 (31) | $2.02 | 81 |
| DeepSeek V4.1 Flash | DeepSeek | lemans | docs | 33/33 | $1.89 | 73 |
| GPT-5.6 Sol | OpenAI | lemans | bare | 32/33 (28) | $10.26 | 18 |
| GPT-5.6 Sol | OpenAI | lemans | docs | 32/33 (28) | $11.44 | 18 |
| Sonnet 5.5 | Anthropic | lemans | bare | 33/33 (29) | $10.61 | 18 |
| Sonnet 5.5 | Anthropic | lemans | docs | 33/33 (31) | $10.72 | 18 |
| Haiku 4.5 | Anthropic | lemans | bare | 12/33 (10) | $14.99 | 82 |
| Haiku 4.5 | Anthropic | lemans | docs | 9/33 | $13.15 | 74 |
| Gemini 3.8 Flash | lemans | bare | 31/33 (30) | $33.62 | 98 | |
| Gemini 3.8 Flash | lemans | docs | 32/33 (31) | $34.38 | 97 | |
| Qwen3.8-27B | Alibaba | lemans | bare | 31/32 (29) | $27.44 | 89 |
| Qwen3.8-27B | Alibaba | lemans | docs | 30/32 | $28.41 | 88 |
| Opus 5.5 | Anthropic | lemans | bare | 33/33 (31) | $21.39 | 24 |
| Opus 5.5 | Anthropic | lemans | docs | 32/33 (31) | $23.56 | 24 |
| GPT-6 Astra | OpenAI | lemans | bare | 33/33 (30) | $33.59 | 15 |
| GPT-6 Astra | OpenAI | lemans | docs | 33/33 (31) | $35.47 | 16 |
| Kimi K3 | Moonshot | lemans | docs | 33/33 (30) | $60.91 | 64 |
| GLM 5.3 | Zhipu | lemans | docs | 32/33 (29) | $77.11 | 95 |
| Grok 4.7 | xAI | lemans | docs | 33/33 (32) | $83.61 | 50 |
| Fable 5.1 | Anthropic | lemans | docs | 33/33 (31) | $87.74 | 42 |
| Opus 5.5 | Anthropic | Claude Code | docs | 29/33 | $30.39 | — |
| Fable 5.1 | Anthropic | Claude Code | docs | 29/33 | $90.36 | — |
Notes:
+-
+
- + Pass counts scored trials, with the two check corrections of section 5 + applied; the raw figure is in parentheses where it differs. + +
-
+ Qwen shows
/32: one trial in each arm was refused by the + provider's content moderation and is out of the denominator (section 5). +
+ - + Claude Code rows can't report a step count the same way (a different agent). + Both lost three trials of one task to a scoring artifact that couldn't be re-scored for that agent + (section 5); on the like-for-like pattern each cell would be 32/33. + +
- + Spend was well within budget on every cell; no single trial reached the per-trial + cost limit (the most expensive single trial was $12.27, under the $20 cap). + +
3. Did the shipped AI context help?
+For the nine models that ran in both arms, with both check corrections applied:
+| Bare (raw) | Shipped docs (raw) | |
|---|---|---|
| Total, 9 models | 256/296 (236) | 255/296 (244) |
+ Honestly: the effect is within noise. After correcting the two over-strict hidden + checks (section 5), the shipped docs did not produce a measurable across-the-board lift — the pooled + difference is a single trial, in the noise at three attempts per cell. Most of the apparent raw + gain (+8 trials) came from one of the corrected checks, which the docs arm happened to pass more + often by chance. +
++ Where the docs did help was narrower and more plausible: a model occasionally used a 4.2 API + it would otherwise have missed, and a couple of models took fewer steps with the docs present. + Cost and step counts barely moved with docs either way (within about ±10% on spend, under one + step for most models). +
++ We read this as: the strong models already know how to build a Wheels app from the stock + conventions, and the shipped context mostly helps at the margins and on newer APIs. We'd + rather report that plainly than overstate it. +
+++ ++ The bare/docs totals here were recomputed from the rescored data plus the erase-account + per-trial flips, and confirmed by re-running the tally: bare 256/296, docs 255/296 (raw 236 + vs 244). +
+
4. The ten tasks
++ Seven tasks mirror a Rails ai-evals task; three are Wheels-native (no Rails counterpart). Pass + rate is across all 24 cells, rescored. +
+| Task | Rails counterpart | Difficulty | Pass |
|---|---|---|---|
| ar-bulk-access-grants — grant a roster of readers in one insert, notify each once, + idempotent | yes | medium | 71/72 (99%) |
| ar-release-recap — keep a cached "recap" matching exactly what a publish committed | yes | medium | 71/72 (99%) |
| ar-compact-positions — compact hand-ordered items to a dense 1..n, trashed ones + included | yes | medium | 66/72 (92%) |
| as-purge-embedded-images — free image files on delete, with reference counting and + rollback safety | yes | hard | 63/72 (88%) |
| sup-legacy-conversions — find and fix every removed/changed 4.x API in an admin + report | yes | medium | 63/72 (88%) |
| wn-query-not-array — a "recently edited" list, and the same list as a JSON array | no (Wheels-native) | easy | 63/72 (88%) |
| ar-erase-account — erase an account and everything only it owns, atomically | yes | hard | 60/72 (83%) |
| wn-literal-route — add print + archive pages without breaking legacy redirect links | no (Wheels-native) | medium | 60/72 (83%) |
| wn-reversible-trash — soft-delete books to a trash with restore and a 30-day purge | no (Wheels-native) | medium | 57/72 (79%) |
| hw-scoped-broadcast — live updates that reach exactly the people who can currently + read a book | yes | hard | 54/70 (77%) |
(hello-world control: 72/72.)
What failed, and why:
+-
+
-
+ ar-erase-account asks for an atomic, hard delete of an account and everything
+ only it owns, through a soft-delete model where the default delete hides rows rather than removing
+ them. Most of the genuine misses were real model mistakes: a soft-delete-only erase that leaves
+ rows behind, calling object methods on raw query rows, a missing datasource, and one route-constraint
+ escaping bug (below). Two more were a documentation gap, not a model error — a raw SQL subquery
+ with a literal inside
where=threw an unhelpful error, since addressed (#4531). Two weaker models (GPT-5.6 Luna, Haiku 4.5) account for most of them. +
+ - + hw-scoped-broadcast asks the agent to add per-listener authorization to a live-update + stream that originally had none. Excluding the two weakest models, it passes 93%; the rest is + almost entirely those two. + +
- + wn-reversible-trash and wn-literal-route are Wheels-native tasks + about soft deletes and route declaration order; the misses are again concentrated in the two weakest + models. + +
- + Haiku 4.5 is the clear floor (12/33 bare, 9/33 docs) — its failures are spread + across most tasks, not one weakness. + +
+ A recurring, framework-agnostic lesson showed up too: CFML strings have no backslash escapes,
+ so a digit route constraint must be written "\d+", not "\\d+" — one agent
+ got this wrong and broke the app's reading URLs. (We've since made that explicit in the routing
+ guide.)
+
5. Corrections and deviations, in full
+We'd rather show the rough edges than hide them.
+-
+
-
+ Two hidden checks were over-strict and were corrected after the runs. Both figures
+ are kept; the corrected ("rescored") one is primary.
+
-
+
- + wn-literal-route demanded every section's title in the print view, + when the task asks for each section's content (and the app's own view renders a section + by its text). Re-scored with that one assertion relaxed and everything else unchanged, the + task went from 27/72 to 60/72. Of the twelve trials that don't reach that figure, six are + genuine failures (an HTTP 301 on the print route) and six are Claude Code trials that couldn't + be re-scored (see below). + +
- + ar-erase-account treated a book as "edited by nobody else," but a fixture + user was an administrator and the app lets administrators edit every book — so "nobody else + can edit" was ambiguous against the app's own rule. Re-scored with that fixture user demoted + (every other check unchanged), the task went from 53/72 to 60/72; seven trials that kept the + book because an administrator exists now pass, and no passing trial became a failure. An eighth + trial of the same kind — a Claude Code trial — couldn't be re-scored because its diff wasn't + captured, so it stays as originally scored. + +
+ -
+ Provider refusals. A provider's content moderation refused two Qwen3.8-27B trials
+ of one task (and its retry); those two trials are out of the denominator, reported as
/32. +
+ - + Infrastructure, not the model. A duplicate launch was discarded and re-run; a + pricing-metadata bug cost two trials at step one (fixed, re-run); a couple of transient provider + 503s were retried. Eight re-runs in all ($4.74). None changed a scored result; the originals are + kept as evidence. + +
- + One agent-native web fetch. In one Claude Code trial, the agent fetched a public + Wheels upgrade guide page twice and solved the task. The Claude Code rows allow web access by + design (it's an agent-native setup), so this is within that row's definition — but it's the first + time it actually happened, so we flag it: that one task's Claude Code result isn't strictly comparable + to the like-for-like rows. + +
- + Three Claude Code trials per cell couldn't be re-scored for the corrected wn-literal-route + check, because that agent's diff wasn't captured. They stay as originally scored. + +
6. What changed in Wheels because of this
++ Per our integrity rule, everything the rounds surfaced was fixed for all + Wheels users, not just the test. Running real agents at real tasks turned out to be an excellent + way to find rough edges. A selection of the user-facing changes the rounds drove, each a public + PR: +
+-
+
-
+
savedChanges()and saved-change state after a save + (#4349) — so an +afterCommitcan tell what the commit actually wrote, instead of apps hand-keeping + flags and getting the lifecycle wrong. +
+ -
+
transaction="savepoint"for nested transaction units (#4040). +
+ -
+ An application data cache —
appCacheFetch()and friends (#4388). +
+ -
+ Query assertions in specs —
assertQueries()and friends (#4357). +
+ -
+ A clock seam for tests —
travelTo()+ (#4387) — and a test client + that can set the request's client address (#4343). +
+ - + Forms inside cached fragments work correctly (#4032); production error pages return the right HTTP status (#4238); date comparisons in the query builder match correctly on SQLite (#4281); a failed background-job enqueue now raises instead of being silently dropped (#4266), and a job enqueued inside a datasource transaction is deferred to its commit so it runs + predictably (#4282). + +
-
+ Dozens of smaller docs, CLI, and upgrade-path fixes — a fresh-VM install-and-upgrade
+ walkthrough, driven by watching agents (and new users) hit the same cliffs, produced a long run
+ of corrections to
wheels new, the upgrade commands, and the guides. +
+
++ ++ Several of these — the application cache, query assertions, the clock seam — are new in 4.2 + precisely because an agent needed them and we agreed every Wheels developer would too. +
+
7. How we kept it honest
+-
+
- + Frozen before any model ran. Tasks and their hidden checks were fixed and independently + reviewed before the first run. The only post-run changes were the two over-strict check corrections + above, which relaxed over-strict checks and are reported with both figures. + +
- + Every run recorded. Passes, failures, refusals, and infrastructure errors are + all kept. Infrastructure failures are reported separately from wrong answers, and re-runs are + accounted for. + +
- + No benchmark-specific caching or pre-seeded answers. The app is a normal Wheels + app; the shipped context is the context every Wheels user gets. + +
- + Same harness, agent, and task style as Rails' ai-evals for the like-for-like + rows; the agent-native rows are labelled and reported on their own. + +
Credits
+
+ Models and providers tested: OpenAI (GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Astra), Anthropic (Opus
+ 5.5, Sonnet 5.5, Haiku 4.5, Fable 5.1), Google (Gemini 3.8 Flash), DeepSeek (V4.1 Flash),
+ Alibaba (Qwen3.8-27B), Moonshot (Kimi K3), Zhipu (GLM 5.3), xAI (Grok 4.7). Harness and agent:
+ Rails' lemans / miniswen (ai-evals) and Anthropic's Claude Code.
+