Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ Next, [try HTTP, Moli or Chromium on a local article](examples/web_fetch/README.

## Reports

To create and share a report, open Artifact Site ([Mainland China](https://yours.lexmount.cn/artifact-site) · [Global](https://yours.lexmount.com/artifact-site)) and install the skill offered there. Follow the skill to create and publish your report, then choose who can access it.

Reports include comparisons, task results, full responses and failure evidence. Runs are saved under `~/.browser-eval/runs/`; use `--output /absolute/path/new-run` to choose another directory.

Rebuild from saved evidence without repeating calls:
Expand All @@ -59,7 +61,7 @@ English is the default; add `--language zh` when running or rebuilding for Chine

Each new report records its Browser Eval build and selected benchmark edition, with exact code and input hashes. See [versioning](docs/benchmarks.md#benchmark-editions) for comparison rules and [reference inputs](docs/reference-inputs.md) for historical data.

To share a report online, publish its `report/` directory through [Artifact Site](https://github.com/lexmount/artifact-site), following the [hosting guide](docs/report-hosting.md). Choose access permissions there. Formal benchmark releases also require the checks in their track guide.
See the [report guide](docs/report-hosting.md) for local builds and formal benchmark releases.

## Versions

Expand All @@ -73,7 +75,7 @@ Published versions, changes and dependency snapshots are listed in [GitHub Relea
| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | Versioned search queries, source references and retrieval scoring |
| [Moli](https://github.com/lexmount/moli) | Browser evaluated through the same task contracts as other participants |
| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | CLI client for the Lexmount WebFetch service; separate from the evaluation harness |
| [Artifact Site](https://github.com/lexmount/artifact-site) | Report hosting, immutable versions and access controls |
| Artifact Site ([Mainland China](https://yours.lexmount.cn/artifact-site) · [Global](https://yours.lexmount.com/artifact-site)) | Report creation and sharing with access controls |
| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | Scenario design source for curated replay; separate protocol, driver and web-platform benchmarks |
| [WebMainBench fork](https://github.com/lexmount/WebMainBench/tree/lex-main) · [upstream](https://github.com/opendatalab/WebMainBench) | Frozen-HTML extraction tasks and declared scoring corrections |
| [WebArena-Verified fork](https://github.com/lexmount/webarena-verified/tree/lex-main) · [upstream](https://github.com/ServiceNow/webarena-verified) | Tasks and evaluator for Browser Use and WebArena trajectory replay |
Expand Down
6 changes: 4 additions & 2 deletions docs/README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ uv run --frozen browser-harness-eval run --delay 0

## 报告

创建和分享报告,请打开 Artifact Site([中国大陆](https://yours.lexmount.cn/artifact-site) · [全球](https://yours.lexmount.com/artifact-site)),安装网站提供的 skill,按引导创建、发布报告,并设置分享权限。

报告包含结果对比、逐题结果、完整响应和失败证据。运行结果默认保存在 `~/.browser-eval/runs/`,也可用 `--output /absolute/path/new-run` 指定新目录。

从保存的证据重建报告,无需重新调用浏览器或服务:
Expand All @@ -59,7 +61,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run

新报告记录 Browser Eval 构建版本和选定的评测版本,并保留代码与输入哈希。结果比较规则见[版本说明](../docs/benchmarks.md#benchmark-editions),历史数据见[参考输入](../docs/reference-inputs.md)。

在线分享时,按[托管指南](../docs/report-hosting.md)将 `report/` 目录发布到 [Artifact Site](https://github.com/lexmount/artifact-site),并设置访问权限。正式评测发布还需通过对应流程的检查。
本地构建与正式评测发布要求见[报告指南](report-hosting.md)。

## 版本

Expand All @@ -73,7 +75,7 @@ uv run --frozen browser-harness-eval report /absolute/path/to/run
| [LexBench Web Search](https://github.com/lexmount/lexbench-websearch) | 搜索问题、来源参考和检索评分 |
| [Moli](https://github.com/lexmount/moli) | 与其他参与者按相同任务规则评测的浏览器 |
| [WebFetch CLI](https://github.com/lexmount/webfetch-cli) | Lexmount WebFetch 服务的命令行客户端 |
| [Artifact Site](https://github.com/lexmount/artifact-site) | 报告托管、不可变版本和访问权限 |
| Artifact Site([中国大陆](https://yours.lexmount.cn/artifact-site) · [全球](https://yours.lexmount.com/artifact-site)) | 报告创建、分享和权限控制 |
| [Lexbench-Headless-Browser](https://github.com/lexmount/Lexbench-Headless-Browser) | 自建轨迹的场景设计来源;另有独立的协议、自动化驱动和 Web 平台评测 |
| [WebMainBench 分支](https://github.com/lexmount/WebMainBench/tree/lex-main) · [上游](https://github.com/opendatalab/WebMainBench) | 冻结 HTML 提取任务与有记录的评分修正 |
| [WebArena-Verified 分支](https://github.com/lexmount/webarena-verified/tree/lex-main) · [上游](https://github.com/ServiceNow/webarena-verified) | Browser Use 和 WebArena 轨迹重放的题目与评测器 |
Expand Down
50 changes: 4 additions & 46 deletions docs/report-hosting.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Publish a report
# Create and share a report

Publish the generated `report/` directory. Keep the rest of the run directory locally so you can rebuild and audit it.
Open Artifact Site ([Mainland China](https://yours.lexmount.cn/artifact-site) · [Global](https://yours.lexmount.com/artifact-site)) and install the skill provided on the site. Follow it to create and publish a report, then set sharing permissions there.

For a local build, publish the generated `report/` directory. Keep the rest of the run locally for rebuilding and auditing.

## Report instances

Expand Down Expand Up @@ -39,50 +41,6 @@ Upload the entire report directory, including the English notice or explicitly g

Only `en` and `zh` are supported. Unsupported codes fail before a run starts or an existing report is replaced. Browser or operating-system language does not silently change a report. To add a language, extend the owning renderer's translation catalog and route, value and evidence-link checks; see [contributing](../CONTRIBUTING.md).

## 1. Choose a site

[Artifact Site](https://github.com/lexmount/artifact-site) hosts multi-page HTML reports and keeps version history. Sign in or create an account on one of these deployments:

| Deployment | Website and CLI base | Publishing guide |
| --- | --- | --- |
| China | [https://artifact-site.lexmount.cn](https://artifact-site.lexmount.cn/) | [Guide](https://artifact-site.lexmount.cn/for-agents.md) |
| Global | [https://yours.lexmount.com/artifact-site](https://yours.lexmount.com/artifact-site) | [Guide](https://yours.lexmount.com/artifact-site/for-agents.md) |
| Your server | Follow the [self-hosting setup](https://github.com/lexmount/artifact-site#quick-start-local-deployment) | `<your-base>/for-agents.md` |

Use the complete base URL, including `/artifact-site` on the global site. The same deployment serves its agent guide at `/for-agents.md` and MCP at `/mcp`.

## 2. Upload the report

With Node.js and npm installed, authorize the CLI and upload:

```bash
npx --yes @artifact-site/cli@0.4.0 login --base https://artifact-site.lexmount.cn
npx --yes @artifact-site/cli@0.4.0 publish /absolute/path/to/run/report \
--share none --base https://artifact-site.lexmount.cn
```

Follow the login page to approve access. For the global site, use `--base https://yours.lexmount.com/artifact-site` in both commands; for self-hosting, use your server's base URL.

These examples pin the tested CLI 0.4.0. `--share none` prevents that older client from creating an extra share link. New artifacts are private by default; check the returned owner, version, URL and access settings.

An agent can also publish through MCP using the deployment's guide. To update an existing report, supply its expected current version as the guide specifies. If an operation is interrupted, recover that upload rather than creating a duplicate.

## 3. Set access and check the report

Choose access in the website's sharing controls:

| Setting | Who can read it? |
| --- | --- |
| `private` | Authorized readers only |
| `unlisted` | Anyone with the main link; absent from public discovery |
| `public` | Publicly readable and discoverable |

CLI 0.4.0 has no `visibility` command. Use the website or the current API/MCP operations documented by your deployment.

Open a task result and follow its response, log and return links on the hosted report. Check that everything you share is intended to be public. Another static host works too if it preserves relative routes and evidence files.

Example runs remain diagnostic after upload. Publishing does not certify benchmark results.

## Maintained benchmark releases

Formal releases use the selected track's package/publish commands after certification, required review and release checks. See [architecture](architecture.md#reports-and-publication) and the [track guides](../README.md#evaluation-tracks).
Expand Down
Loading