Skip to content

Add no-time-limit scores to the Pro leaderboard #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xujustinj merged 1 commit into from
Oct 22, 2026
Merged

Add no-time-limit scores to the Pro leaderboard #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Scores are percentages; costs are USD. The best score in each metric is **bold**
- **Grounding F1:** requires both a correct value and correct evidence: the predicted box must overlap an accepted evidence box at IoU 0.5 or higher (word level). Grounding is evaluated on the subset with verified ground-truth evidence for the corresponding metric.
- **% Perfect:** percentage of documents with 100% Value F1.

Failed documents and Pro documents exceeding the 30-minute limit score zero.
In the official benchmark, documents that fail or exceed a 30-minute limit score zero. However, due to the additional thinking required to handle Pro documents, we also maintain an unofficial leaderboard where completed outputs count regardless of elapsed time.

</details>

Expand Down Expand Up @@ -108,14 +108,14 @@ Rows are ordered by % Perfect, then Value F1, then lower cost.
| 9 | Codex GPT-6.1 Sol | Coding Agents | 12 | 74.06 | <u>63.96</u> | 13.86¢ |
| 10 | Reducto Deep | Specialized APIs | 3 | 56.35 | 42.20 | 6.18¢ |

Top 10 of 13 systems — full table in [leaderboard-pro.csv](leaderboard-pro.csv).
Top 10 of 13 systems — full table in [leaderboard-pro.csv](leaderboard-pro.csv). Scores with no time limit are also available there.
<!-- PRO_LEADERBOARD:END -->

### Submitting results

Open a PR updating [leaderboard.csv](leaderboard.csv) or [leaderboard-pro.csv](leaderboard-pro.csv), with your pipeline configuration and evaluation reports. Scores are percentages; costs are USD per page. Regenerate the tables with `uv run python scripts/update_readme.py`.

The rows each CSV needs are described in [ExtractBench: submitting results](docs/extractbench.md#submitting-results) and [ExtractBench Pro: submitting results](docs/extractbench-pro.md#submitting-results). For Pro, include the overall (`all`) row and all six challenge rows.
The rows each CSV needs are described in [ExtractBench: submitting results](docs/extractbench.md#submitting-results) and [ExtractBench Pro: submitting results](docs/extractbench-pro.md#submitting-results). For Pro, include the overall (`all`) row and all six challenge rows, with scores for both time-limit views.

<details>
<summary><strong>Inclusion criteria</strong></summary>
Expand Down
15 changes: 13 additions & 2 deletions docs/extractbench-pro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ Seven of the eight short documents are single plotter sheets 13 to 17 feet long

Every document is synthetic: its values are fictional and fixed before the page is rendered, so the expected output is exact by construction. Each value's scoring rule lists its accepted readings with their source locations (`page`, `bbox`, `quote`, `value`), and any accepted reading counts as correct.

Grounding F1 (box level) and page-level grounding F1 are computed on the 81 documents with verified ground-truth evidence. Value F1 and % Perfect use all 100 documents. A document that fails, or exceeds the 30-minute limit, scores zero.
Grounding F1 (box level) and page-level grounding F1 are computed on the 81 documents with verified ground-truth evidence. Value F1 and % Perfect use all 100 documents.

Both leaderboard views use the same runs and document denominators. The **30-minute limit** view gives zero to documents taking longer than 1,800 seconds. The **no time limit** view scores all completed outputs regardless of elapsed time. Failed extractions, including attempts that timed out without a usable output, score zero in both views. Removing the scoring cutoff does not recover an output from a terminated attempt. Costs are shared between the two views.

## Local layout

Expand Down Expand Up @@ -92,4 +94,13 @@ These figures apply the reported mean cost per page to 2,622 pages. Per-system p

## Submitting results

Add seven rows per system to [leaderboard-pro.csv](../leaderboard-pro.csv): the overall row (`Challenge` = `all`) and one row for each of the six challenges. Each row carries `Documents`, `Grounding_Documents`, `Perfect_Documents` (% Perfect), `Value_F1`, `Word_Grounding` (Grounding F1), `Page_Grounding` (page-level grounding F1), and `Cost_Per_Page`. Then follow the shared steps in [Submitting results](../README.md#submitting-results).
Add seven rows per system to [leaderboard-pro.csv](../leaderboard-pro.csv): the overall row (`Challenge` = `all`) and one row for each of the six challenges. Each row carries `Documents`, `Grounding_Documents`, `Cost_Per_Page`, and both score views:

| Metric | 30-minute limit | No time limit |
|---|---|---|
| % Perfect | `Perfect_Documents` | `Perfect_Documents_No_Time_Limit` |
| Value F1 | `Value_F1` | `Value_F1_No_Time_Limit` |
| Grounding F1 | `Word_Grounding` | `Word_Grounding_No_Time_Limit` |
| Page-level grounding F1 | `Page_Grounding` | `Page_Grounding_No_Time_Limit` |

Scores are percentages. Use an empty cell for an unreported metric; zero means a measured zero. Then follow the shared steps in [Submitting results](../README.md#submitting-results).
Loading
Loading