Repository navigation
Add GPT-6.1 Terra Codex pipelines and GPT-6.1 Terra/Sol rows - #98
Merged
Merged
Conversation
Register codex_code_extract_gpt_6_1_terra_{low,medium}_evidence in the
GPT-6 evidence sweep, price gpt-6.1-terra at $1.00 input, $0.05 cached
input, $1.25 cache write and $5.00 output per 1M tokens, and add the
low-effort run to the leaderboard.
Low-effort evidence run over all 370 documents.
boyang-zhang1
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds GPT-6.1 Terra (
gpt-6.1-terra) as two Codex evidence pipelines, and leaderboard rows for Codex GPT-6.1 Terra and Codex GPT-6.1 Sol.Why
GPT-6.1 Terra is OpenAI's new fast, low-cost model, priced at half of GPT-6.1 Sol.
How
codex_code_extract_gpt_6_1_terra_low_evidenceandcodex_code_extract_gpt_6_1_terra_medium_evidencejoin the GPT-6 evidence sweep with the same config as GPT-6.1 Sol (unrestricted shell, no cost cap, page and box citations).codex_code_extract_gpt_6_1_terra_low_evidence,codex_code_extract_gpt_6_1_sol_low_evidence) over all 370 documents (252 short / 98 medium / 20 long). The GPT-6.1 Sol pipelines were already registered.Results
GPT-6.1 Sol and Terra take the top two word-grounding scores on the leaderboard.
Changes
src/extract_bench/inference/pipelines/extract.py: Terra entry in the GPT-6 evidence sweepsrc/extract_bench/inference/providers/extract/codex_code_extract.py: pricing rowsdocs/pipelines.md: Terra pipelinesleaderboard.csv: two rowsREADME.md: regenerated byscripts/update_readme.pyTesting
extract-bench pipelineslists both pipelinesruff checkon the changed source files passesscripts/update_readme.pychanges only the two generated tablesNotes
Requires the Codex CLI and
OPENAI_API_KEY. These pipelines run local shell commands; see the README warning.