From 1e7825174c3cc0f99a1e26ddf7c248b5c47d3d52 Mon Sep 17 00:00:00 2001 From: lanyue-llk <270302213+lanyue-llk@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:15:23 +0800 Subject: [PATCH] docs(readme): highlight evaluation and reporting features --- README.md | 23 ++++++++++++++++------- docs/README.zh-CN.md | 23 ++++++++++++++++------- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 928919095..1d0ab38ad 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,28 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](pyproject.toml) -[Quickstart](#quickstart) · [Tracks](#evaluation-tracks) · [Reports](#reports) · [Versions](#versions) · [Related projects](#related-repositories) · [Contributing](CONTRIBUTING.md) +[Features](#features) · [Quickstart](#quickstart) · [Tracks](#evaluation-tracks) · [Reports](#reports) · [Versions](#versions) · [Related projects](#related-repositories) · [Contributing](CONTRIBUTING.md) -Browser Eval is an evaluation harness for browsers and web agents. Connect a benchmark and a browser or service adapter to run tasks, record responses and timing, apply the benchmark's scoring rules, and generate comparison reports with the original evidence. +Run benchmark comparisons and turn the results into reports you can inspect, rebuild and share. + +## Features + +| Feature | What you get | +| --- | --- | +| Run to report | Run a configured comparison and generate an HTML report with one CLI command. | +| Multiple benchmarks | Use integrated benchmarks or bring your own tasks, with the selected benchmark's scoring rules. [Benchmark integration](docs/benchmarks.md) | +| Extensible participants | Compare local browsers, hosted APIs, extraction tools and web agents. Add participants through the existing [adapter contracts](docs/provider-contract.md). | +| Inspectable evidence | Follow aggregate results down to individual responses, action traces and failure logs. | +| Saved-run workflows | Rebuild reports and rescore saved outputs without repeating provider calls. | +| Quality, performance and cost | Compare task success alongside latency, resource usage and recorded billing. | +| Reproducible comparisons | Record task inputs, benchmark and software revisions, scoring code and run settings. | +| Reports you can share | Publish through Artifact Site and choose who can access the report. [Create and share a report](#reports) | ## Evaluation tracks -The main branch includes all four tracks below. Each track provides a runnable example and a guide to its benchmark inputs and environment. +Choose a workflow for its task contract, setup and runnable example. | Track | What it tests | Benchmarks and task suites | Guide | | --- | --- | --- | --- | @@ -25,10 +38,6 @@ The main branch includes all four tracks below. Each track provides a runnable e | Browser Trajectory Replay | Complete the same recorded browser actions across implementations | [Repository-curated fixed-action suite](examples/trajectory_replay/README.md), adapted from [Lexbench-Headless-Browser scenario designs](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios); [qualified WebArena human recordings](evaluation/browser-use-public/webarena-human-suite.json), checked with the [WebArena-Verified evaluator](https://github.com/lexmount/webarena-verified/tree/lex-main) | [Curated suite](examples/trajectory_replay/README.md) · [WebArena replay](examples/trajectory_replay/public_webarena/README.md) | | Browser Use | Let an Agent use different browsers to complete tasks | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified), using the [pinned evaluator fork](https://github.com/lexmount/webarena-verified/tree/lex-main) and [declared task selection](evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](examples/browser_use/README.md) | -Web Fetch supports local browsers, hosted services and extraction tools. See [adapters](docs/provider-contract.md) for available methods and [dependencies](docs/dependencies.md) for installation requirements. Replay and Browser Use have separate runners and environments. - -See [benchmark integration](docs/benchmarks.md) for task and scoring ownership, pinned versions and input requirements. The replay suites adapt fixed actions for browser comparison; Browser Use evaluates autonomous Agent decisions. - ## Quickstart Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then run the offline example: diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md index c21f44a3c..92d59f44a 100644 --- a/docs/README.zh-CN.md +++ b/docs/README.zh-CN.md @@ -8,15 +8,28 @@ [![License: Apache 2.0](https://img.shields.io/badge/code-Apache--2.0-51735b)](../LICENSE) [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-356a91)](../pyproject.toml) -[快速开始](#快速开始) · [评测流程](#评测流程) · [报告](#报告) · [版本](#版本) · [相关仓库](#相关仓库) · [参与贡献](../CONTRIBUTING.md) +[特性](#特性) · [快速开始](#快速开始) · [评测流程](#评测流程) · [报告](#报告) · [版本](#版本) · [相关仓库](#相关仓库) · [参与贡献](../CONTRIBUTING.md) -Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation harness）。接入评测集和浏览器或服务适配器后，就能运行任务、记录响应和耗时、按评测集的规则评分，并生成带原始证据的对比报告。 +运行 benchmark 对比，生成带逐题证据的报告，后续可重建或分享。 + +## 特性 + +| 特性 | 能做什么 | +| --- | --- | +| 从评测到报告 | 配置任务和评测对象后，一条命令完成评测并生成 HTML 报告。 | +| 接入多种 benchmark | 使用已接入的 benchmark，也可提供自己的任务，按所选 benchmark 的规则评分。[接入指南](benchmarks.md) | +| 扩展评测对象 | 比较本地浏览器、在线 API、内容提取工具和 Web Agent，通过已有[适配接口](provider-contract.md)接入新的参与方。 | +| 查看逐题证据 | 从汇总指标查看对应的逐题响应、操作轨迹和失败日志。 | +| 复用已有结果 | 从保存的输出重建报告、重算评分，无需重复调用参与方。 | +| 比较质量、性能和成本 | 同时查看任务通过情况、延迟、资源使用和已记录的计费数据。 | +| 追溯实验配置 | 保留任务输入、benchmark 和软件版本、评分代码及执行配置。 | +| 分享报告并控制权限 | 通过 Artifact Site 发布报告，并设置谁能访问。[创建与分享报告](#报告) | ## 评测流程 -主分支已包含下列四种评测流程。各流程均提供可运行的示例，以及评测输入和环境配置指南。 +按任务选择评测流程，各指南提供任务约定、环境配置和可运行示例。 | 流程 | 测什么 | 对应评测集与任务集 | 使用指南 | | --- | --- | --- | --- | @@ -25,10 +38,6 @@ Browser Eval 是面向浏览器和 Web Agent 的评测脚手架（evaluation har | 轨迹重放 | 比较不同浏览器能否完成同一组录制动作 | [仓库自建固定动作任务集](../examples/trajectory_replay/README.md)，改编自 [Lexbench-Headless-Browser 场景设计](https://github.com/lexmount/Lexbench-Headless-Browser/tree/3f1580a20922fc3f339b5fc279d9bbdb06d5d194/tasks/scenarios)；[经过资格验证的 WebArena 人工录制轨迹](../evaluation/browser-use-public/webarena-human-suite.json)，由 [WebArena-Verified 评测器](https://github.com/lexmount/webarena-verified/tree/lex-main)验收 | [自建任务集](../examples/trajectory_replay/README.md) · [WebArena 重放](../examples/trajectory_replay/public_webarena/README.md) | | Browser Use | 让 Agent 使用不同浏览器完成任务 | [WebArena-Verified](https://github.com/ServiceNow/webarena-verified)，使用[固定版本的评测器分支](https://github.com/lexmount/webarena-verified/tree/lex-main)和[明确选定的任务](../evaluation/browser-use/webarena-verified-pilot.json) | [CowAgent + WebArena](../examples/browser_use/README.md) | -Web Fetch 支持本地浏览器、在线服务和提取工具。可用方法见[适配器说明](../docs/provider-contract.md)，安装要求见[依赖说明](../docs/dependencies.md)。轨迹重放和 Browser Use 各有自己的运行入口与环境。 - -题目与评分的维护归属、固定版本和输入要求见[评测集接入指南](../docs/benchmarks.md)。轨迹重放使用固定动作比较浏览器，Browser Use 则评测 Agent 自主决策后的任务完成情况。 - ## 快速开始 安装 [uv](https://docs.astral.sh/uv/getting-started/installation/)，运行离线示例：