[bpamiri]

Summary

Adds a results page to the landing site (wheels.dev) reporting our Round 4
evaluation of AI coding agents building real features in a Wheels application:
methodology, the full per-cell results table, the bare vs shipped-docs
comparison, the ten tasks, the two post-run check corrections in full, the
user-facing framework changes the rounds drove, and credits.

• File / route: web/sites/landing/src/pages/ai.astro → wheels.dev/ai
  (file-based routing; picks up BaseLayout + @wheels-dev/ui tokens, like index.astro).
• Figures: the rescored numbers are primary; raw figures are shown in
  parentheses throughout (two over-strict hidden checks were corrected after the
  runs, both reported in §5).
• Anchors: section headings carry stable slug ids (e.g.
  #5-corrections-and-deviations-in-full) so deep links resolve.
• Tables: wrapped in a horizontal-scroll container so they never force
  page-level scroll on small screens.

Verification

• astro build — passes; /ai/index.html emitted.
• astro check — 0 errors, 0 warnings, 0 hints.
• Prettier — clean.
• Every results-table cell and task-table row diffed against the source copy; all match.

Review notes / open items

• Final URL is /ai (the route the page ships at); /ai-results and /agents
  are also free if preferred.
• Cover/hero art, a published results date, and an optional "methods + raw data"
  link are left as follow-ups.

🤖 Generated with Claude Code

[bpamiri]

Review: head 4a395f4. REQUEST CHANGES: four accuracy fixes before this goes public. The figures themselves are right.

Verified independently against the source data

• Headline. Rescored per-cell data (wn-literal-route corrected) gives 693/790. Adding the seven ar-erase-account trials the second correction flips gives 700/790 (88.6%). Raw is 660/790 (83.5%).
• Every one of the 24 rows (pass, raw-in-parentheses, spend, mean steps) matches, including the seven erase-account flips (DeepSeek bare +1, Sol bare +1, Sol docs +2, Sonnet bare +1, Qwen bare +2).
• Docs effect: bare 256/296 (raw 236) vs docs 255/296 (raw 244).
• Task table: all ten task rows match, and they sum to 700 with the hello-world control (72/72). The "93% excluding the two weakest models" for hw-scoped-broadcast is right (54/58).
• The web-fetch footnote matches the record: one Claude Code (Fable) trial on sup-legacy-conversions fetched the public 4.1 → 4.2 upgrade guide twice and solved the task.
• PR links. All 11 linked PRs are public and merged. Descriptions match, except fix(job): throw Wheels.Job.EnqueueFailed when a job cannot be written #4266 (below).
• Wording. No internal names or paths, and no security-fix detail. "Wheels" throughout, no superlatives.
• Build. Deploy landing passes. Tables sit in an overflow-x: auto wrapper (min-width: 520px), so they scroll inside the page on phones. I didn't render a phone screenshot.

Accuracy fixes (MEDIUM)

1. "eight trials that kept the book because an admin exists now pass" (§5, ar-erase-account) is wrong. Eight trials had that reading, but only seven flipped (53 → 60). The eighth was a Claude Code trial whose diff wasn't captured, so it couldn't be re-scored and stays a miss. Say seven, and add that trial to the "couldn't be re-scored" bullet next to the three wn-literal-route ones.
2. §1 says the checks "never changed after we saw results", and §7 says "Nothing changed after we saw results". Two checks were corrected after the runs (§5). Suggest: "Tasks and checks were frozen before any model ran and never changed during the runs; after the runs we corrected two over-strict checks, each by one assertion, and report both figures." As written, §1 contradicts §5.
3. "ar-erase-account (the hardest Rails-counterpart task)" isn't true after the correction. It's 60/72 (83%), and hw-scoped-broadcast (also a Rails counterpart) is lower at 54/70 (77%), as the page's own table shows.
4. "The genuine misses were real model mistakes" for ar-erase-account leaves out a class. The per-trial diagnosis puts 2 of the remaining misses down to a docs gap: a raw SQL subquery inside where= throws, and nothing in the docs said it can't hold one with a literal. The rest are model mistakes, plus the one unrescorable Claude Code trial. Name the docs gap, and say whether the docs have since been updated (they should be, per the page's own integrity rule).

LOW

5. wn-literal-route: "Six genuine failures remain" reads as if only six misses remain. 60/72 leaves 12: the six genuine ones plus the six Claude Code trials that couldn't be re-scored. Say both in one sentence.
6. Qwen docs row shows 30/32 (30). The raw figure is the same, and the notes say raw is shown only "where it differs"; drop the (30).
7. fix(job): throw Wheels.Job.EnqueueFailed when a job cannot be written #4266 is described as part of "background jobs enqueued across datasources behave predictably". fix(job): throw Wheels.Job.EnqueueFailed when a job cannot be written #4266 is "throw Wheels.Job.EnqueueFailed when a job cannot be written"; only fix(job): defer an enqueue inside a tenant-datasource transaction to its commit #4282 is the cross-datasource fix. Give fix(job): throw Wheels.Job.EnqueueFailed when a job cannot be written #4266 its own clause (a failed enqueue now raises a clear error).
8. Budget detail. "well within budget on every cell; the per-trial cost guard never bound … against a $20 guard" discloses budget/guard settings. The spend figures are enough; I'd drop the budget/guard sentence (a judgement call for the maintainers).

INFO

• The heading "2. Round 4 results" and "the rounds" imply earlier rounds a reader can't find. Either link them or title it "Results"; a judgement call for the maintainers.
• Still pending in PR CI: Visual regression, Deploy api, Lucee 7 + SQLite (LuCLI). The rest pass.

[bpamiri]

Re-review at b5dabf8: APPROVE (ready to publish). Every item from the previous review is fixed. I re-checked each one against the source data, not the draft:

• Erase-account correction: "seven trials … now pass" (53 → 60), and the eighth, a Claude Code trial whose diff wasn't captured, is now named as unrescorable and stays as scored.
• "Frozen" wording. §1 and §7 now say the only post-run changes were the two disclosed check corrections, reported with both figures. That's consistent with §5.
• "Hardest" claim: removed. hw-scoped-broadcast is lower, as the table shows.
• Docs-gap misses. The two where= subquery misses are now described as a documentation gap, not a model error, with fix(model): name a where= value that has no model column to bind against #4531 (merged: "name a where= value that has no model column to bind against") as the follow-up.
• wn-literal-route. The 12 misses are reconciled as six genuine failures plus six unrescorable Claude Code trials. The six genuine ones are all an HTTP 301 on /books/:id/print, per the per-trial diagnosis.
• Small fixes:
  • the Qwen docs cell is 30/32;
  • fix(job): throw Wheels.Job.EnqueueFailed when a job cannot be written #4266 is now "a failed enqueue raises instead of being silently dropped" (matching its PR: the job was previously lost behind persisted: false);
  • fix(job): defer an enqueue inside a tenant-datasource transaction to its commit #4282 is "deferred to its commit";
  • the heading is "Latest results (October 2026)".
• Cost-cap sentence: kept and restated plainly, a maintainers' call.

All the figures verified earlier are unchanged: 700/790 rescored (88.6%), 660/790 raw (83.5%), all 24 cells, bare 256/296 vs docs 255/296, and the task table.

PR CI at b5dabf8: 19 pass, 0 fail, 3 skipped, including Deploy landing and Visual regression.