Skip to content

docs: add a guide to verifying a submitted leaderboard row - #97

Closed
Nikit117 wants to merge 1 commit into
run-llama:mainfrom
Nikit117:docs/verifying-a-submission
Closed

Nikit117 wants to merge 1 commit into
run-llama:mainfrom
Nikit117:docs/verifying-a-submission

Conversation

@Nikit117

Copy link
Copy Markdown

The row schema and the shared steps are already documented, in
docs/extractbench.md#submitting-results
and README#submitting-results.
What those pages do not cover is the other half of a submission: how a row
gets checked, what that check costs a reviewer, and what a submitter can do to
make it cheap.

This adds docs/verifying-a-submission.md, written from the submitter's side:

  • what checking a row costs: scoring the submitted predictions takes minutes,
    running the pipeline over the corpus is the expensive part
  • publish the per-document predictions and scores, so a reviewer can recompute
    the score without running anything and without spending an API key
  • state the endpoint, every service the run needs (not only the model), the
    configuration switches and the cost derivation before the reviewer has to ask
  • pin the revision by a tag as well as by a commit, so a force-push cannot move
    the numbers a row was produced from
  • offer a one-command subset run that prints the published per-document score
    next to each document that just ran
  • what actually caused a round trip on our own submission, and a checklist for
    submitters

The running example is our own submission (#84), which is why the "what caused
a round trip" list is specific rather than hypothetical. I am happy to cut that
section if a more general list is preferred, and to fold the checklist into
docs/extractbench.md instead of keeping a separate page.

No change to the inclusion criteria, the schema or the scoring - this is a
submission-side guide only.

The row schema is documented in docs/extractbench.md#submitting-results and the
shared steps in README#submitting-results. Neither covers the other half of a
submission: how a row gets checked, and what a submitter can do to make that
check cheap.

This adds docs/verifying-a-submission.md, written from the submitter's side:
what checking a row costs, publishing the per-document predictions and scores
so a reviewer can recompute without running the pipeline, stating the endpoint
and every service and the cost derivation up front, pinning the revision by tag
as well as by commit, offering a one-command subset run, and what caused a
round trip on our own submission (run-llama#84).

No change to the inclusion criteria, the schema or the scoring.
@Nikit117

Copy link
Copy Markdown
Author

Closing this for now. The submission side is documented properly in
docs/extractbench.md, and this page turns into an opinionated guide that the
repository did not ask for - and pushing a docs pull request into a repository
where my other pull request is already open is not the right move.

The file stays in my fork (docs/verifying-a-submission, commit 03c1440) if
it is ever useful.

@Nikit117 Nikit117 closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants