Add Web Skill Factory: Evolving Reusable, Verified, Code-Native Skills for Web Agents

[DEM1TASSE]

What

Adds Web Skill Factory (webwright.skill_factory) — a self-evolving skill library on top of Webwright's code-as-action solves. It turns the final_script.py every solve already produces into reusable, parameterized skills that are plain Python + Playwright: code you can run without a model and compose into the next task instead of re-exploring a site.

solve → gate → group by template → distill → replay-verify → library → next solve reuses

At solve time the agent queries the library once and gets one of three recommendations — use, adapt, or skip — plus the skill's source to reuse or modify. Reuse never blocks the agent from continuing on its own. Both integration points are additive; the agent loop and default config are unchanged.

Why code, not notes:

• Runs standalone, no model. A learned skill re-executes in ~40 s with zero tokens — cron-able, instead of a model re-reading a note and redoing the work every time.
• Real software-engineering surface. Because skills are code, they inherit code's properties for free: tests, versioning, history, encapsulation — executable and verifiable, not prose the model must interpret.
• Verified twice before it lands. An input gate keeps wrong solves out of the material; then the distilled skill must replay its own answers standalone, with no model, before it enters the library.
• Gets stronger with use. New solves widen a skill in place; regression-replay keeps a later change from breaking coverage a skill already had.

Two gates, and a grade that means something

• Input gate — a solve becomes material only if its answer is trustworthy: gold (a known answer), or the default self_verify (shape + non-empty + the agent's own success report).
• Output gate — the distilled skill is replayed standalone, with no model, on its own training taskspecs, and admitted only if it reproduces the recorded answers.

The output gate is what lets a landed skill carry a real grade: executable (replay ran and reproduced — run it directly), reference (replay ran and failed — kept as a labelled prior the agent reads), or unverified (replay skipped).

Modules

Each has a stable interface and a swappable implementation:

module	role
skill_factory/library.py	store — Skill + Library, skills on disk (skill.py + meta.json)
skill_factory/retrieve.py	retrieve — task → most relevant candidates
skill_factory/decide.py	decide — candidates → use / adapt / skip
skill_factory/gate.py	input gate — gold / self_verify / none (keeps wrong solves out)
skill_factory/update.py	evolve — incremental growth; refine distils, parameterizes, decomposes into primitives, and replay-verifies
skill_factory/learn.py	learn <runs_dir> — auto-group runs into templates, gate, evolve; no manifest to write
skill_factory/init.py	init "<need>" — draft a skill.yaml skeleton to fill with ground truth
skill_factory/build.py	build <spec> — solve N instances then learn (parallel, resumable)
skill_factory/llm.py	backend-agnostic LLM via Webwright's Model (no endpoint/key hardcoded)
skill_factory/prompt.py	non-invasive task-prompt hint (with_skill_hint)
tools/skill_use.py	solve-time tool the agent calls from bash (retrieve + decide)

How it plugs in (no change to the agent loop or default config)

1. Reuse at solve time — the skill_use tool, invoked from bash like self_reflection / image_qa:

   python -m webwright.tools.skill_use --task "<the task>" --library "$SKILL_LIBRARY_ROOT"

   returns {verdict: use|adapt|skip, skill_id, source_path, how_to_reuse}.
2. Growth after solving — three ways in, by what you already have:

   python -m webwright.skill_factory init  "<one-line need>"          # draft a spec to fill
   python -m webwright.skill_factory build skill.yaml --library ./library   # solve N + learn
   python -m webwright.skill_factory learn outputs/  --library ./library    # distil runs you have

   The manual manifest path (update) remains for benchmarks and logged-in sites.

examples/learned_library/ checks in the skill this produced from 3 real Google Flights solves — five parameters lifted (origin/destination city+code, date), verified on an unseen route three independent ways (from scratch / reuse / standalone, same answer) — with a CI test locking it.

Validation

WebArena: 10 templates across 3 sites — reuse lifts held-out accuracy +15pp and saves steps

10 retrieve-type task templates across shopping-admin / gitlab / map. Per template: 3 train solves build the library (admitted only if gold-verified), 2 held-out instances (unseen parameter values) measure reuse. Every task is solved both with the library and from scratch. Model: gpt-5.4. 100 runs total.

	WITH library	from scratch	Δ
held-out accuracy (20)	70%	55%	+15 pp
held-out avg steps	14.7	17.1	−2.4
train accuracy (30)	86.7%	76.7%	+10 pp
train avg steps	13.7	15.9	−2.2

• Reuse helps most when solving from scratch is expensive. Of 20 held-out tasks, 4 were rescued (failed from scratch, solved with the library) and 6 more solved in fewer steps. Biggest single win: 33 steps → 10.
• Retrieval stayed reliable as the library grew to 10 skills: all 20 held-out solves retrieved their own template's skill, including two near-duplicate gitlab commit skills.
• These numbers are the agent-in-loop path — the agent reading a skill as a prior. The Quick Start's flights example shows the other end: a strict-verified executable skill rerunning an unseen route standalone, with no model.

[adamlu123]

update src/webwright/skills/README.md

[adamlu123]

I did not find a skill_factory/README.md?

[adamlu123]

detailed comments inline.