From 4a395f4e94e865a3731b8cb64eaad41237add18f Mon Sep 17 00:00:00 2001 From: Peter Amiri Date: Sun, 11 Oct 2026 17:14:31 -0700 Subject: [PATCH 1/2] docs(web): add AI coding-agent results page at /ai Add web/sites/landing/src/pages/ai.astro (route wheels.dev/ai), a results page reporting the Round 4 coding-agent evaluation against real feature work in a Wheels app: methodology, the full per-cell results table, the bare vs shipped-docs comparison, the ten tasks, the two post-run check corrections in full, and the user-facing framework changes the rounds drove. Built with the shared BaseLayout and @wheels-dev/ui tokens, matching index.astro. Primary figures are the rescored ones; raw figures are shown in parentheses. Section headings carry stable slug ids so deep links resolve. Data tables use a horizontal-scroll wrapper to avoid page-level scroll on small screens. astro build + astro check pass; Prettier clean. Co-Authored-By: Claude Opus 4.8 Signed-off-by: Peter Amiri --- web/sites/landing/src/pages/ai.astro | 640 +++++++++++++++++++++++++++ 1 file changed, 640 insertions(+) create mode 100644 web/sites/landing/src/pages/ai.astro diff --git a/web/sites/landing/src/pages/ai.astro b/web/sites/landing/src/pages/ai.astro new file mode 100644 index 0000000000..2858d88590 --- /dev/null +++ b/web/sites/landing/src/pages/ai.astro @@ -0,0 +1,640 @@ +--- +import BaseLayout from '../layouts/BaseLayout.astro'; +// Results page for the AI coding-agent evaluation. Route: wheels.dev/ai +// Primary figures are the rescored ones (two over-strict hidden checks were +// corrected after the runs); raw figures are shown in parentheses throughout. +// Open items for a later pass: cover/hero art, a results date, and a +// "methods + raw data" link if the data files are published. +--- + + +
+
+

AI coding agents

+

Building Wheels apps with AI coding agents

+

+ We ran a set of coding agents at real feature work in a Wheels application and measured how + often they finished the job. This page reports what we measured, every number, and what we + changed in Wheels as a result. +

+
+
700 / 790
+
+ scored trials passed — 88.6% + 660 / 790 (83.5%) raw, before two hidden-check corrections +
+
+
+
+ +
+

+ Our aim was a like-for-like comparison with the way Ruby on Rails' own + ai-evals measures agents: the same harness, the + same agent, a comparable sample app, and the same style of task with hidden acceptance checks. Where + we couldn't match Rails exactly, we say so. +

+ +

1. What we measured, and why

+
    +
  • + Same harness and agent as Rails' ai-evals. We used Rails' + lemans harness and its miniswen agent unchanged, so a Wheels task is + run and scored the same way a Rails task is. +
    • +
  • + A comparable sample app. Tasks run against a Writebook-style content app built + in Wheels — the same kind of app Rails' Stage 1 uses — so the asks line up. +
    • +
  • + Frozen tasks with hidden checks. Each task is a plain-English feature request. + The agent gets the prompt and nothing else; a hidden spec decides pass/fail. A task scores 1 only + if the app's own test suite is green and every hidden check passes. Tasks and + their checks were frozen and reviewed before any model ran, and never changed after we saw results. +
    • +
  • + Two arms. Each model ran twice where possible: bare (a stock + Wheels app) and shipped docs (the same app with the AI context Wheels now ships + — .ai/ docs and a prebuilt .mcp.json). This isolates whether the + shipped context helps. +
    • +
  • + Agent-native rows reported separately. Alongside the like-for-like rows we ran + Claude Code against the same frozen tasks, + via the "Harbor" setup. Those rows are a different agent and are always reported and labelled + separately — never pooled with the like-for-like numbers. +
    • +
+

+ Ten tasks (plus a hello-world control) × three attempts per cell. A "cell" is one model in one + arm. Twenty-four cells in all: nine models in both arms, four more in the shipped arm only, + and two Claude Code cells. +

+
+

+ Integrity first. The case we wanted to test honestly is simply whether Wheels + deserves a seat at the table. So: no benchmark-specific caching, no pre-generated answers, and + no app state an agent wouldn't have in normal use. Every improvement we made to help agents shipped + to every Wheels user — in wheels new, the docs, and the MCP + server — not just to the test. Every run was recorded, including failures and infrastructure + errors, and those are reported below. +

+
+ +

2. Round 4 results

+

+ Overall, across the ten tasks, the agents passed 700 of 790 scored trials (88.6%) — raw, before two hidden-check corrections (section 5), 660 of 790 (83.5%). +

+

+ Steps are the agent's tool-use turns (a guard stopped a trial at 150). Spend is the settled + provider cost for that cell. The "docs" arm is the shipped AI context; "bare" is a stock app. +

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelProviderAgentArmPass (raw)SpendSteps (mean)
GPT-5.6 LunaOpenAIlemansbare19/33 (18)$0.7015
GPT-5.6 LunaOpenAIlemansdocs21/33 (20)$0.7116
DeepSeek V4.1 FlashDeepSeeklemansbare32/33 (31)$2.0281
DeepSeek V4.1 FlashDeepSeeklemansdocs33/33$1.8973
GPT-5.6 SolOpenAIlemansbare32/33 (28)$10.2618
GPT-5.6 SolOpenAIlemansdocs32/33 (28)$11.4418
Sonnet 5.5Anthropiclemansbare33/33 (29)$10.6118
Sonnet 5.5Anthropiclemansdocs33/33 (31)$10.7218
Haiku 4.5Anthropiclemansbare12/33 (10)$14.9982
Haiku 4.5Anthropiclemansdocs9/33$13.1574
Gemini 3.8 FlashGooglelemansbare31/33 (30)$33.6298
Gemini 3.8 FlashGooglelemansdocs32/33 (31)$34.3897
Qwen3.8-27BAlibabalemansbare31/32 (29)$27.4489
Qwen3.8-27BAlibabalemansdocs30/32 (30)$28.4188
Opus 5.5Anthropiclemansbare33/33 (31)$21.3924
Opus 5.5Anthropiclemansdocs32/33 (31)$23.5624
GPT-6 AstraOpenAIlemansbare33/33 (30)$33.5915
GPT-6 AstraOpenAIlemansdocs33/33 (31)$35.4716
Kimi K3Moonshotlemansdocs33/33 (30)$60.9164
GLM 5.3Zhipulemansdocs32/33 (29)$77.1195
Grok 4.7xAIlemansdocs33/33 (32)$83.6150
Fable 5.1Anthropiclemansdocs33/33 (31)$87.7442
Opus 5.5AnthropicClaude Codedocs29/33$30.39—
Fable 5.1AnthropicClaude Codedocs29/33$90.36—
+
+

Notes:

+
    +
  • + Pass counts scored trials, with the two check corrections of section 5 + applied; the raw figure is in parentheses where it differs. +
    • +
  • + Qwen shows /32: one trial in each arm was refused by the + provider's content moderation and is out of the denominator (section 5). +
    • +
  • + Claude Code rows can't report a step count the same way (a different agent). + Both lost three trials of one task to a scoring artifact that couldn't be re-scored for that agent + (section 5); on the like-for-like pattern each cell would be 32/33. +
    • +
  • + Spend was well within budget on every cell; the per-trial cost guard never bound + (the most expensive single trial was $12.27 against a $20 guard). +
    • +
+ +

3. Did the shipped AI context help?

+

For the nine models that ran in both arms, with both check corrections applied:

+
+ + + + + + + +
Bare (raw)Shipped docs (raw)
Total, 9 models256/296 (236)255/296 (244)
+
+

+ Honestly: the effect is within noise. After correcting the two over-strict hidden + checks (section 5), the shipped docs did not produce a measurable across-the-board lift — the pooled + difference is a single trial, in the noise at three attempts per cell. Most of the apparent raw + gain (+8 trials) came from one of the corrected checks, which the docs arm happened to pass more + often by chance. +

+

+ Where the docs did help was narrower and more plausible: a model occasionally used a 4.2 API + it would otherwise have missed, and a couple of models took fewer steps with the docs present. + Cost and step counts barely moved with docs either way (within about ±10% on spend, under one + step for most models). +

+

+ We read this as: the strong models already know how to build a Wheels app from the stock + conventions, and the shipped context mostly helps at the margins and on newer APIs. We'd + rather report that plainly than overstate it. +

+
+

+ The bare/docs totals here were recomputed from the rescored data plus the erase-account + per-trial flips, and confirmed by re-running the tally: bare 256/296, docs 255/296 (raw 236 + vs 244). +

+
+ +

4. The ten tasks

+

+ Seven tasks mirror a Rails ai-evals task; three are Wheels-native (no Rails counterpart). Pass + rate is across all 24 cells, rescored. +

+
+ + + + + + + + + + + + + + + + +
TaskRails counterpartDifficultyPass
ar-bulk-access-grants — grant a roster of readers in one insert, notify each once, + idempotentyesmedium71/72 (99%)
ar-release-recap — keep a cached "recap" matching exactly what a publish committedyesmedium71/72 (99%)
ar-compact-positions — compact hand-ordered items to a dense 1..n, trashed ones + includedyesmedium66/72 (92%)
as-purge-embedded-images — free image files on delete, with reference counting and + rollback safetyyeshard63/72 (88%)
sup-legacy-conversions — find and fix every removed/changed 4.x API in an admin + reportyesmedium63/72 (88%)
wn-query-not-array — a "recently edited" list, and the same list as a JSON arrayno (Wheels-native)easy63/72 (88%)
ar-erase-account — erase an account and everything only it owns, atomicallyyeshard60/72 (83%)
wn-literal-route — add print + archive pages without breaking legacy redirect linksno (Wheels-native)medium60/72 (83%)
wn-reversible-trash — soft-delete books to a trash with restore and a 30-day purgeno (Wheels-native)medium57/72 (79%)
hw-scoped-broadcast — live updates that reach exactly the people who can currently + read a bookyeshard54/70 (77%)
+
+

(hello-world control: 72/72.)

+

What failed, and why:

+
    +
  • + ar-erase-account (the hardest Rails-counterpart task) asks for an atomic, hard + delete of an account and everything only it owns, through a soft-delete model where the default + delete hides rows rather than removing them. The genuine misses were real model mistakes: a soft-delete-only + erase that leaves rows behind, calling object methods on raw query rows, a missing datasource, + and one route-constraint escaping bug (below). Two weaker models (GPT-5.6 Luna, Haiku 4.5) account + for most of them. +
    • +
  • + hw-scoped-broadcast asks the agent to add per-listener authorization to a live-update + stream that originally had none. Excluding the two weakest models, it passes 93%; the rest is + almost entirely those two. +
    • +
  • + wn-reversible-trash and wn-literal-route are Wheels-native tasks + about soft deletes and route declaration order; the misses are again concentrated in the two weakest + models. +
    • +
  • + Haiku 4.5 is the clear floor (12/33 bare, 9/33 docs) — its failures are spread + across most tasks, not one weakness. +
    • +
+

+ A recurring, framework-agnostic lesson showed up too: CFML strings have no backslash escapes, + so a digit route constraint must be written "\d+", not "\\d+" — one agent + got this wrong and broke the app's reading URLs. (We've since made that explicit in the routing + guide.) +

+ +

5. Corrections and deviations, in full

+

We'd rather show the rough edges than hide them.

+
    +
  • + Two hidden checks were over-strict and were corrected after the runs. Both figures + are kept; the corrected ("rescored") one is primary. +
      +
    1. + wn-literal-route demanded every section's title in the print view, + when the task asks for each section's content (and the app's own view renders a section + by its text). Re-scored with that one assertion relaxed and everything else unchanged, the + task went from 27/72 to 60/72. Six genuine failures remain. +
      2. +
    2. + ar-erase-account treated a book as "edited by nobody else," but a fixture + user was an administrator and the app lets administrators edit every book — so "nobody else + can edit" was ambiguous against the app's own rule. Re-scored with that fixture user demoted + (every other check unchanged), the task went from 53/72 to 60/72; eight trials that kept the + book because an admin exists now pass, and no passing trial became a failure. +
      3. +
    + Each correction changed exactly one thing; every other check stayed as frozen. +
    • +
  • + Provider refusals. A provider's content moderation refused two Qwen3.8-27B trials + of one task (and its retry); those two trials are out of the denominator, reported as /32. +
    • +
  • + Infrastructure, not the model. A duplicate launch was discarded and re-run; a + pricing-metadata bug cost two trials at step one (fixed, re-run); a couple of transient provider + 503s were retried. Eight re-runs in all ($4.74). None changed a scored result; the originals are + kept as evidence. +
    • +
  • + One agent-native web fetch. In one Claude Code trial, the agent fetched a public + Wheels upgrade guide page twice and solved the task. The Claude Code rows allow web access by + design (it's an agent-native setup), so this is within that row's definition — but it's the first + time it actually happened, so we flag it: that one task's Claude Code result isn't strictly comparable + to the like-for-like rows. +
    • +
  • + Three Claude Code trials per cell couldn't be re-scored for the corrected wn-literal-route + check, because that agent's diff wasn't captured. They stay as originally scored. +
    • +
+ +

6. What changed in Wheels because of this

+

+ Per our integrity rule, everything the rounds surfaced was fixed for all + Wheels users, not just the test. Running real agents at real tasks turned out to be an excellent + way to find rough edges. A selection of the user-facing changes the rounds drove, each a public + PR: +

+
    +
  • + savedChanges() and saved-change state after a save + (#4349) — so an + afterCommit can tell what the commit actually wrote, instead of apps hand-keeping + flags and getting the lifecycle wrong. +
    • +
  • + transaction="savepoint" for nested transaction units (#4040). +
    • +
  • + An application data cache — appCacheFetch() and friends (#4388). +
    • +
  • + Query assertions in specs — assertQueries() and friends (#4357). +
    • +
  • + A clock seam for tests — travelTo() + (#4387) — and a test client + that can set the request's client address (#4343). +
    • +
  • + Forms inside cached fragments work correctly (#4032); production error pages return the right HTTP status (#4238); date comparisons in the query builder match correctly on SQLite (#4281); background jobs enqueued across datasources behave predictably (#4266, + #4282). +
    • +
  • + Dozens of smaller docs, CLI, and upgrade-path fixes — a fresh-VM install-and-upgrade + walkthrough, driven by watching agents (and new users) hit the same cliffs, produced a long run + of corrections to wheels new, the upgrade commands, and the guides. +
    • +
+
+

+ Several of these — the application cache, query assertions, the clock seam — are new in 4.2 + precisely because an agent needed them and we agreed every Wheels developer would too. +

+
+ +

7. How we kept it honest

+
    +
  • + Frozen before any model ran. Tasks and their hidden checks were fixed and independently + reviewed before the first run. Nothing changed after we saw results; the two corrections above + relaxed over-strict checks and are reported with both figures. +
    • +
  • + Every run recorded. Passes, failures, refusals, and infrastructure errors are + all kept. Infrastructure failures are reported separately from wrong answers, and re-runs are + accounted for. +
    • +
  • + No benchmark-specific caching or pre-seeded answers. The app is a normal Wheels + app; the shipped context is the context every Wheels user gets. +
    • +
  • + Same harness, agent, and task style as Rails' ai-evals for the like-for-like + rows; the agent-native rows are labelled and reported on their own. +
    • +
+ +

Credits

+

+ Models and providers tested: OpenAI (GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Astra), Anthropic (Opus + 5.5, Sonnet 5.5, Haiku 4.5, Fable 5.1), Google (Gemini 3.8 Flash), DeepSeek (V4.1 Flash), + Alibaba (Qwen3.8-27B), Moonshot (Kimi K3), Zhipu (GLM 5.3), xAI (Grok 4.7). Harness and agent: + Rails' lemans / miniswen (ai-evals) and Anthropic's Claude Code. +

+
+ + + From b5dabf87c7a85cc1a729ee8c7077769d2a42c5ed Mon Sep 17 00:00:00 2001 From: Peter Amiri Date: Sun, 11 Oct 2026 17:27:08 -0700 Subject: [PATCH 2/2] docs(web): correct AI results page per review - erase-account rescore flipped seven trials, not eight; note the eighth (a Claude Code trial) as unrescorable - reword the frozen-before-runs claim in sections 1 and 7 so it no longer reads as contradicting the two disclosed check corrections in section 5 - drop the "hardest task" label on ar-erase-account (hw-scoped-broadcast is lower) - note two erase misses were a where= subquery documentation gap since addressed (#4531), not model errors - literal-route: reconcile "six remain" with the twelve not passing (six genuine HTTP 301, six unrescorable Claude Code trials) - drop the redundant raw parens on the Qwen docs cell (raw equals rescored) - correct #4266's description (a failed enqueue raises) vs #4282 (defer an enqueue inside a datasource transaction) - state the per-trial cost guard plainly - rename the section 2 heading to "Latest results (October 2026)" Co-Authored-By: Claude Opus 4.8 Signed-off-by: Peter Amiri --- web/sites/landing/src/pages/ai.astro | 46 ++++++++++++++++------------ 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/web/sites/landing/src/pages/ai.astro b/web/sites/landing/src/pages/ai.astro index 2858d88590..ef11f1fd05 100644 --- a/web/sites/landing/src/pages/ai.astro +++ b/web/sites/landing/src/pages/ai.astro @@ -54,7 +54,9 @@ import BaseLayout from '../layouts/BaseLayout.astro'; Frozen tasks with hidden checks. Each task is a plain-English feature request. The agent gets the prompt and nothing else; a hidden spec decides pass/fail. A task scores 1 only if the app's own test suite is green and every hidden check passes. Tasks and - their checks were frozen and reviewed before any model ran, and never changed after we saw results. + their checks were frozen and independently reviewed before any model ran. The only changes after + any model ran were the two over-strict check corrections disclosed in section 5, each reported + with both the before and after figures.
  • Two arms. Each model ran twice where possible: bare (a stock @@ -85,7 +87,7 @@ import BaseLayout from '../layouts/BaseLayout.astro';

    -

    2. Round 4 results

    +

    2. Latest results (October 2026)

    Overall, across the ten tasks, the agents passed 700 of 790 scored trials (88.6%)89 Qwen3.8-27BAlibabalemansdocs30/32 (30)Qwen3.8-27BAlibabalemansdocs30/32$28.4188 @@ -240,8 +242,8 @@ import BaseLayout from '../layouts/BaseLayout.astro'; (section 5); on the like-for-like pattern each cell would be 32/33.

  • - Spend was well within budget on every cell; the per-trial cost guard never bound - (the most expensive single trial was $12.27 against a $20 guard). + Spend was well within budget on every cell; no single trial reached the per-trial + cost limit (the most expensive single trial was $12.27, under the $20 cap).
    • @@ -356,12 +358,14 @@ import BaseLayout from '../layouts/BaseLayout.astro';

    What failed, and why:

    • - ar-erase-account (the hardest Rails-counterpart task) asks for an atomic, hard - delete of an account and everything only it owns, through a soft-delete model where the default - delete hides rows rather than removing them. The genuine misses were real model mistakes: a soft-delete-only - erase that leaves rows behind, calling object methods on raw query rows, a missing datasource, - and one route-constraint escaping bug (below). Two weaker models (GPT-5.6 Luna, Haiku 4.5) account - for most of them. + ar-erase-account asks for an atomic, hard delete of an account and everything + only it owns, through a soft-delete model where the default delete hides rows rather than removing + them. Most of the genuine misses were real model mistakes: a soft-delete-only erase that leaves + rows behind, calling object methods on raw query rows, a missing datasource, and one route-constraint + escaping bug (below). Two more were a documentation gap, not a model error — a raw SQL subquery + with a literal inside where= threw an unhelpful error, since addressed (#4531). Two weaker models (GPT-5.6 Luna, Haiku 4.5) account for most of them.
    • hw-scoped-broadcast asks the agent to add per-listener authorization to a live-update @@ -396,14 +400,18 @@ import BaseLayout from '../layouts/BaseLayout.astro'; wn-literal-route demanded every section's title in the print view, when the task asks for each section's content (and the app's own view renders a section by its text). Re-scored with that one assertion relaxed and everything else unchanged, the - task went from 27/72 to 60/72. Six genuine failures remain. + task went from 27/72 to 60/72. Of the twelve trials that don't reach that figure, six are + genuine failures (an HTTP 301 on the print route) and six are Claude Code trials that couldn't + be re-scored (see below).
    • ar-erase-account treated a book as "edited by nobody else," but a fixture user was an administrator and the app lets administrators edit every book — so "nobody else can edit" was ambiguous against the app's own rule. Re-scored with that fixture user demoted - (every other check unchanged), the task went from 53/72 to 60/72; eight trials that kept the - book because an admin exists now pass, and no passing trial became a failure. + (every other check unchanged), the task went from 53/72 to 60/72; seven trials that kept the + book because an administrator exists now pass, and no passing trial became a failure. An eighth + trial of the same kind — a Claude Code trial — couldn't be re-scored because its diff wasn't + captured, so it stays as originally scored.
      • Each correction changed exactly one thing; every other check stayed as frozen. @@ -476,10 +484,10 @@ import BaseLayout from '../layouts/BaseLayout.astro'; href="https://github.com/wheels-dev/wheels/pull/4238">#4238); date comparisons in the query builder match correctly on SQLite (#4281); background jobs enqueued across datasources behave predictably (); a failed background-job enqueue now raises instead of being silently dropped (#4266, - #4282). + >), and a job enqueued inside a datasource transaction is deferred to its commit so it runs + predictably (#4282).
    • Dozens of smaller docs, CLI, and upgrade-path fixes — a fresh-VM install-and-upgrade @@ -498,8 +506,8 @@ import BaseLayout from '../layouts/BaseLayout.astro';
      • Frozen before any model ran. Tasks and their hidden checks were fixed and independently - reviewed before the first run. Nothing changed after we saw results; the two corrections above - relaxed over-strict checks and are reported with both figures. + reviewed before the first run. The only post-run changes were the two over-strict check corrections + above, which relaxed over-strict checks and are reported with both figures.
      • Every run recorded. Passes, failures, refusals, and infrastructure errors are