Score the content a Web Fetch tool returns against explicit task requirements.
| Feature | What you get |
|---|---|
| Content and file coverage | Evaluate page text, structured records with usable links, discovered resources and delivered file bytes. |
| Provider-independent rules | Apply the same task requirements to browsers, extraction tools and hosted services. |
| Acceptance and quality | Keep task pass/fail decisions separate from content-quality diagnostics. Evaluation protocol |
| Offline scoring | Score saved responses and files through the CLI or Python API, without calling providers. |
| Verified input exports | Check frozen input hashes and export only declared files for a selected task suite. Data and access |
| Reproducible editions | Bind task definitions, reference hashes and scoring code in one bench release. |
| Runner integration | Supply tasks and scoring to your own runner or Browser Eval, which handles execution, evidence and reports. |
| Suite | Tasks | Scope |
|---|---|---|
webfetch |
150 | Web pages, documents, images, video and challenge pages |
dynamic-content |
100 | Content and records loaded after the initial response |
popular-sites |
100 | Content on widely used websites |
access-restrictions |
100 | URLs with previously observed access restrictions |
articles |
100 | Article bodies, initial comments and post collections |
The catalog contains 550 task entries across five suites. Public definitions include task instructions and acceptance rules; complete frozen reference inputs must be obtained separately. See data and access and article requirements.
Install uv, then:
git clone https://github.com/lexmount/lexbench-webfetch.git
cd lexbench-webfetch
uv sync --frozen
uv run --frozen lexbench-webfetch list
uv run --frozen lexbench-webfetch validateThese commands check the bundled definitions offline, without a browser or API key.
To export a suite, obtain the frozen files listed in its input manifest and run:
uv run --frozen lexbench-webfetch export \
--suite webfetch --source /absolute/path/approved-inputs \
--output /absolute/path/new-input-directoryExport verifies the input hashes and writes the declared files to a new directory. Required reference files and access are described in the data guide.
Try an accepted and a rejected saved response:
uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/answer.txt
uv run --frozen lexbench-webfetch score --task examples/task.json --response examples/missing.txtFor real evaluations, use the complete task from a verified dataset. Add --metadata metadata.json for structured response fields, or --file /absolute/path/file for delivered files. Scoring requirements and result reporting are described in the evaluation protocol.
Follow the Browser Eval integration guide to run exported tasks against browsers or services and build comparison reports. Provider accounts, API keys and measured outputs belong to the execution environment. The commands above validate and score saved inputs without calling providers.
Use the release history to choose a frozen benchmark. Run uv run --frozen lexbench-webfetch version to inspect the installed catalog and scoring state. See version rules for result comparisons.
| Repository | Role |
|---|---|
| Browser Eval | Browser/provider execution, saved evidence and comparison reports |
| LexBench Web Search | Search queries, source references and retrieval scoring |
| Moli | Browser that can be evaluated with these tasks |
| WebFetch CLI | CLI client for the Lexmount WebFetch service |
| Artifact Site (Mainland China · Global) | Report sharing and access controls |
| Lexbench-Headless-Browser | Protocol, automation-driver and web-platform benchmarks |
Third-party notices list scoring dependencies and source rights. WebMainBench, WCXB and WebArena are separate benchmarks; they are not part of this 550-entry catalog.
Read CONTRIBUTING before changing task requirements or scoring. Run make check to validate definitions and tests.
Security · Third-party content · Apache 2.0
Project citation: CITATION.cff.
@software{lexbench_webfetch_2026,
author = {{LexBench Web Fetch contributors}},
title = {LexBench Web Fetch},
year = {2026},
url = {https://github.com/lexmount/lexbench-webfetch}
}