Skip to content

init: lessons from a blind run on metabase.com - #130

Merged
tmashara merged 1 commit into
mainfrom
claude/init-eval-fixes
Nov 2, 2026
Merged

tmashara merged 1 commit into
mainfrom
claude/init-eval-fixes

Conversation

@tmashara

@tmashara tmashara commented Nov 2, 2026

Copy link
Copy Markdown
Owner

A blind /renderdiff init run on metabase.com's marketing app (174 routes, 870 shots) produced a working skill; these are the places init.md cost it time.

  • Sample every page that loads a third-party script, iframe or form, not only one page per template: one-off pages carried 10 of the noisy embeds.
  • Narrowest fix first; maskMedia hides first-party iframes too. Blocking or hiding drops content from the check: keep what you can, list the rest under Ignore.
  • Hooks run at domcontentloaded: wait for a component's ready marker.
  • Noise a rerun clears and nothing reproduces goes under Ignore, not a mask; then the full set twice (/jobs came out clean once by chance).
  • Curl local vs the live site; sample.txt in the tree; --out per run instead of a fixed outDir.

🤖 Generated with Claude Code

Sample every page with a third-party embed, narrowest fix first, wait for
a component's ready marker in hooks, full set twice, --out per run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tmashara
tmashara merged commit e9e1ada into main Nov 2, 2026
3 checks passed
@tmashara
tmashara deleted the claude/init-eval-fixes branch November 2, 2026 18:28
@tmashara tmashara mentioned this pull request Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant