Repository navigation
Conversation
The row schema is documented in docs/extractbench.md#submitting-results and the shared steps in README#submitting-results. Neither covers the other half of a submission: how a row gets checked, and what a submitter can do to make that check cheap. This adds docs/verifying-a-submission.md, written from the submitter's side: what checking a row costs, publishing the per-document predictions and scores so a reviewer can recompute without running the pipeline, stating the endpoint and every service and the cost derivation up front, pinning the revision by tag as well as by commit, offering a one-command subset run, and what caused a round trip on our own submission (run-llama#84). No change to the inclusion criteria, the schema or the scoring.
Author
|
Closing this for now. The submission side is documented properly in The file stays in my fork ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The row schema and the shared steps are already documented, in
docs/extractbench.md#submitting-results
and README#submitting-results.
What those pages do not cover is the other half of a submission: how a row
gets checked, what that check costs a reviewer, and what a submitter can do to
make it cheap.
This adds
docs/verifying-a-submission.md, written from the submitter's side:running the pipeline over the corpus is the expensive part
the score without running anything and without spending an API key
configuration switches and the cost derivation before the reviewer has to ask
the numbers a row was produced from
next to each document that just ran
submitters
The running example is our own submission (#84), which is why the "what caused
a round trip" list is specific rather than hypothetical. I am happy to cut that
section if a more general list is preferred, and to fold the checklist into
docs/extractbench.mdinstead of keeping a separate page.No change to the inclusion criteria, the schema or the scoring - this is a
submission-side guide only.