English · 简体中文
Features · Quickstart · Tracks · Reports · Versions · Related projects · Contributing
Run benchmark comparisons and turn the results into reports you can inspect, rebuild and share.
| Feature | What you get |
|---|---|
| Run to report | Run a configured comparison and generate an HTML report with one CLI command. |
| Multiple benchmarks | Use integrated benchmarks or bring your own tasks, with the selected benchmark's scoring rules. Benchmark integration |
| Extensible participants | Compare local browsers, hosted APIs, extraction tools and web agents. Add participants through the existing adapter contracts. |
| Inspectable evidence | Follow aggregate results down to individual responses, action traces and failure logs. |
| Saved-run workflows | Rebuild reports and rescore saved outputs without repeating provider calls. |
| Quality, performance and cost | Compare task success alongside latency, resource usage and recorded billing. |
| Reproducible comparisons | Record task inputs, benchmark and software revisions, scoring code and run settings. |
| Reports you can share | Publish through Artifact Site and choose who can access the report. Create and share a report |
Choose a workflow for its task contract, setup and runnable example.
| Track | What it tests | Benchmarks and task suites | Guide |
|---|---|---|---|
| Web Fetch | Retrieve the requested page content, records, resources or files | LexBench Web Fetch, WebMainBench, WCXB | Local example · Full guide |
| Web Search (preview) | Compare search results, relevance, source retrieval, latency and billing | LexBench Web Search: scenario queries and a SimpleQA Verified retrieval adaptation | Local example · Full guide |
| Browser Trajectory Replay | Complete the same recorded browser actions across implementations | Repository-curated fixed-action suite, adapted from Lexbench-Headless-Browser scenario designs; qualified WebArena human recordings, checked with the WebArena-Verified evaluator | Curated suite · WebArena replay |
| Browser Use | Let an Agent use different browsers to complete tasks | WebArena-Verified, using the pinned evaluator fork and declared task selection | CowAgent + WebArena |
Install uv, then run the offline example:
git clone https://github.com/lexmount/browser-eval.git
cd browser-eval
uv sync --frozen
uv run --frozen browser-harness-eval validate
uv run --frozen browser-harness-eval run --delay 0Open the report path printed in the terminal. The example has one expected pass and one expected failure; it needs no browser, Docker or API key. Python 3.11–3.14 is supported; development uses 3.11.
Next, try HTTP, Moli or Chromium on a local article. Optional adapters, service credentials and live evaluation commands are covered in the track guides above.
To create and share a report, open Artifact Site (Mainland China · Global) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it.
Reports include comparisons, task results, full responses and failure evidence. Runs are saved under ~/.browser-eval/runs/; use --output /absolute/path/new-run to choose another directory.
Rebuild from saved evidence without repeating calls:
uv run --frozen browser-harness-eval report /absolute/path/to/runEnglish is the default; add --language zh when running or rebuilding for Chinese. A rebuild without that option keeps the saved language.
Each new report records its Browser Eval build and selected benchmark edition, with exact code and input hashes. See versioning for comparison rules and reference inputs for historical data.
See the report guide for local builds and formal benchmark releases.
Published versions, changes and dependency snapshots are listed in GitHub Releases. See version rules for release policy and report provenance.
| Repository | Role |
|---|---|
| LexBench Web Fetch | Versioned Web Fetch tasks, acceptance scoring and frozen input bindings |
| LexBench Web Search | Versioned search queries, source references and retrieval scoring |
| Moli | Browser evaluated through the same task contracts as other participants |
| WebFetch CLI | CLI client for the Lexmount WebFetch service; separate from the evaluation harness |
| Artifact Site (Mainland China · Global) | Report creation and sharing with access controls |
| Lexbench-Headless-Browser | Scenario design source for curated replay; separate protocol, driver and web-platform benchmarks |
| WebMainBench fork · upstream | Frozen-HTML extraction tasks and declared scoring corrections |
| WebArena-Verified fork · upstream | Tasks and evaluator for Browser Use and WebArena trajectory replay |
| WCXB | Independent content-extraction benchmark |
| CowAgent, OpenClaw, DeepSeek Harness | Agent execution and Web Fetch tool baselines |
Dependency sources and pins cover Playwright, extraction libraries and supporting runtimes. Third-party notices cover attribution and data rights. Cite the benchmark and upstream data used alongside Browser Eval.
Read CONTRIBUTING for code ownership, tests and PR conventions, or open an issue with a reproducible example.
make setup-test
make ciArchitecture · Security · Third-party content · Apache 2.0
Project citation: CITATION.cff.
@software{browser_eval_2026,
author = {{Browser Eval contributors}},
title = {Browser Eval},
year = {2026},
url = {https://github.com/lexmount/browser-eval}
}