[Nikit117]

Add KYNEX (gpt-5.4) to the ExtractBench leaderboard

KYNEX

Knowledge Yield Nexus for EXtraction — a local, tool-using
document-extraction agent.

The harness is published here: https://github.com/Nikit117/kynex-harness
(the repository was renamed to kynex-harness; the code is the one that was
measured, minus the RFQ/CPQ workbook feature, its prompts, every credential
and the benchmark corpus).

metric	value
Unified value F1 (overall, 370 documents)	92.65
Short / Medium / Long	94.12 / 91.52 / 79.73
Precision / Recall (short)	93.42 / 95.07
Precision / Recall (medium)	92.65 / 90.84
Precision / Recall (long)	84.81 / 77.34
Cost per page	14.17 cent
Latency per document (short / medium / long)	101.9 s / 271.0 s / 867.2 s
Documents below 0.50	1 (real_ftx_full_corrupted, 0.4099)

That is rank 12 of 50 — above OpenAI GPT-6 Astra (91.91) and Codex
(GPT-5.6 Terra Evidence, 92.26), below the top-10 cut of 93.57.

The grounding metrics are 0.00 by construction: the harness emits no per-field
citation boxes, so field_citations is empty — the same situation as the other
value-only entries on the board.

Answers to the review comments

1. The name. The system is called KYNEX. It is a product name, not a
personal one: the internal project it was built for has a different name, and
the published code deliberately does not carry it (the codebase and the
executable prompts contain company data, so the public release drops them). I
am the author of the harness; the K and N also carry the initials of the two
people who wrote it. If you prefer a different name, I will change it in one
commit — the module, the provider name, the pipeline name and the
leaderboard.csv row are the only four places it appears.

2. The link to the published harness.
https://github.com/Nikit117/kynex-harness — the provider now calls that
checkout (KYNEX_HARNESS_ROOT / KYNEX_HARNESS_PYTHON, both documented in
the repository README), so the pipeline runs exactly the published code.

3. No keys, no confidential material. The published repository contains no
.env, no endpoint, no key and no corpus. The harness reads its credentials
from the environment (AZURE_OPENAI_*, AZURE_DI_*); the repository's
.gitignore excludes .env* and the run artifacts. The RFQ/CPQ workbook
feature and its prompts are not in the published tree.

What the system is

Not a one-shot prompt: an agent loop plus a post-processing layer that fixes
the three failure modes the official scorer punishes hardest.

rule	what it fixes	measured effect
page-wise enumeration of long record lists	a 26,725-row owners array the agent returned as null	real_oklahoma_unclaimed_2024 0.00 → 0.97
non-record / blank-row / header-like-row / redaction-marker pruning	contact blocks, empty items, table headers and CUI markers counted as records	sf1449_supplies_services_0046 0.23 → 0.83
structured rows from the document's own tables	the agent transcribing currency symbols, thousands separators and column alignment by eye	real_franklin_nov_2025 0.57 → 1.00

The last rule is the largest single win: over the 370-document corpus 133
documents have the right row COUNT and not one row that matches the ground
truth exactly — the largest loss class in the benchmark. Document
Intelligence reads the same table structurally, so the rows are built from the
cells instead.

How it was measured

One clean end-to-end run of the unmodified official scorer
(extract_bench.evaluation.score_case.score_extract_prediction). The official
scorer, the schemas, the ground truth and the pinned default prompt were not
modified; the harness' own prompt is byte-identical to the default and is
pinned by a test, and every change is post-processing of the agent's output.

Two independent measurements of the 49-document D3 slice give 0.8950 and
0.9104, with zero documents below 0.50 in both.

The run is one draw of the harness, not a deterministic property of it: two runs
of the same code differ by a mean of ~0.02 overall and by up to 0.3 on
individual documents. The per-document scores of the submitted run can be
supplied on request so every number can be checked.

How to run it

git clone https://github.com/Nikit117/kynex-harness
cd kynex-harness && pip install -r requirements.txt
export KYNEX_HARNESS_ROOT=$PWD
export KYNEX_HARNESS_PYTHON=$PWD/.venv/bin/python
uv run extract-bench run kynex_gpt_5_4

The provider is a local subprocess provider, like claude_code_extract_*
and codex_code_extract_*: it shells out to the harness' own entry point so
the ExtractBench environment needs no extra Python packages.

variable	meaning
KYNEX_HARNESS_ROOT	the harness checkout (must be set)
KYNEX_HARNESS_PYTHON	an interpreter with the harness' dependencies
AZURE_OPENAI_*	the gpt-5.4 deployment the harness calls
AZURE_DI_ENDPOINT / AZURE_DI_API_KEY	Document Intelligence, for the table rows and the OCR fallback

The provider passes the harness' rule switches to the subprocess explicitly,
because one of them (ADH_BULK_ARRAYS) is opt-in in the harness and would
otherwise be silently off in the official environment — the submitted system
would then not be the system that was measured.

Cost derivation

Measured token usage: 183,567,244 input + 10,206,069 output tokens over 11,186
model calls, priced at the public list price of gpt-5.4 ($2.50 / 1M input,
$15.00 / 1M output), plus Azure Document Intelligence Layout analysis
($10.00 / 1,000 pages) for the pages the harness analyses. It excludes the
Azure resource's own idle cost, and a re-run may differ.

What is in this pull request

file	change
src/extract_bench/inference/providers/extract/kynex.py	new — the provider
src/extract_bench/inference/providers/extract/__init__.py	registers the provider module
src/extract_bench/inference/pipelines/extract.py	registers the kynex_gpt_5_4 pipeline
leaderboard.csv	the measured row
README.md	regenerated with scripts/update_readme.py (the system count 49 → 50)

Notes

• No rule de-duplicates rows: the corpus ground truth legitimately contains
  exact duplicate rows, so de-duplication would be wrong.
• The published repository is the measured code: its entry point reproduces the
  stored prediction of the measured run (178 of 178 rows identical on the
  document where the structured-table rule fires).
• If you would rather re-measure before merging, KYNEX_HARNESS_ROOT pointing
  at a clone is all that is needed.

[xujustinj]

Hi Nikit, thank you for your interest in ExtractBench!

Your results are impressive, especially for using GPT 5.4, but we cannot currently accept them as-is, due to the following two issues. Please point us to your harness, and we can work together on addressing them.

[Nikit117]

Thanks for the review -- all three points are addressed:

1. Name: the system is called KYNEX (Knowledge Yield Nexus for EXtraction). It is a product name; the internal project it was built for has a different name and the public code deliberately drops it, because the codebase and the executable prompts contain company data. I am the author of the harness. If you prefer another name, the module, the provider name, the pipeline name and the leaderboard row are the only four places it appears and I will change them in one commit.

2. The published harness: https://github.com/Nikit117/kynex-harness -- the provider now calls that checkout via \KYNEX_HARNESS_ROOT, so the pipeline runs exactly the published code. The repository contains no .env, no endpoint, no key and no corpus; the RFQ/CPQ workbook feature and its prompts are not published.

3. The row: \KYNEX (gpt-5.4)\ with the measured 92.65 unified value F1 over the published 370-document set, scored with the unmodified official scorer.

The published entry point reproduces the stored prediction of the measured run (178 of 178 rows identical on the document where the structured-table rule fires), so the code in the repository is the code that was measured.

[Nikit117]

Hi justin

[xujustinj]

Thanks for addressing the issues so quickly!

This is enough information for us to proceed with verifying your results. I'm going to push some adjustments onto the PR (including for the two comments marked in this review) as I work on trying to reproduce it

[Nikit117]

Thanks for setting up the pinned checkout - that is a much better design than the
KYNEX_HARNESS_ROOT default I had, and it removes the local path from the
provider entirely.

Status of the pieces you asked about:

• Name - KYNEX (Knowledge Yield Nexus for EXtraction), a product name;
  the internal project the harness was written for is not in the published
  tree.
• Public harness - https://github.com/Nikit117/kynex-harness . The
  revision your provider pins is published on main, and it is the code that
  was measured: the harness, the five extraction rules and the OCR fallback,
  with no credential, no corpus and no confidential material.
• Measured result - 92.65 unified value F1 over the published
  370-document set with the unmodified official scorer (rank 12 of 50 at the
  time of writing), 14.29 cents per page, 0 documents below 0.50 except one
  (a 7,554-row scanned list where the row count is right and the per-row
  values drift).

If it is useful, the per-document scores of the submitted run are in
benchmark_results/tui_bench/ in the harness repository, so any number on the
board can be checked against a document rather than taken on trust.

[Nikit117]

What running this pipeline needs

Since the provider provisions the harness itself, the only credentials are the
ones the model calls need:

variable	needed	why
AZURE_OPENAI_ENDPOINT / AZURE_OPENAI_API_KEY / AZURE_OPENAI_CHAT_DEPLOYMENT / AZURE_OPENAI_API_VERSION	yes	the gpt-5.4 deployment the harness calls
AZURE_DI_ENDPOINT / AZURE_DI_API_KEY	optional	Document Intelligence: the structured table rows and the OCR fallback for pages with no text layer

Document Intelligence is optional, and it is a small part of the run. Over
the measured 370 documents, 27 of them used it, and they total 942 pages - so
the layout analysis for the whole benchmark is about USD 9.42 at the public
list price ($10 per 1,000 pages), and within the Azure free tier (500 pages a
month) it is free.

The other 343 documents do not use it at all, so without it their scores are
unaffected; only those 27 change, and they change because the structured table
rows and the OCR fallback are exactly the two rules that need it.

The cheapest end-to-end check needs no Document Intelligence at all. The
49-document D3 slice is the one I measured twice (0.8950 and 0.9104, zero
documents below 0.50 in both) and none of its 49 documents uses Document
Intelligence - it is gpt-5.4 only, roughly USD 80 and 15 minutes for a full
slice.

Costs of the measured full run, for reference: 183,567,244 input +
10,206,069 output tokens over 11,186 model calls (USD 612 at the gpt-5.4 list
price of $2.50 / 1M input and $15.00 / 1M output) plus USD 9.42 of layout
analysis - USD 696 in total, 14.29 cents per page.

If it is easier than running it yourself, I can run the 49-document slice
through the provider in this pull request and publish the per-document scores,
so every number on the board can be checked against a document. The
per-document scores of the submitted run are in the harness repository under
benchmark_results/tui_bench/ today.

[Nikit117]

Thanks again for the review, and for building the pinned-checkout setup - it
made the pipeline genuinely runnable by other people, which is what I was after.

Two questions, and no rush on either:

1. Is there anything else you need from me to get this merged (a re-run, a
   note in the description, a different provider name)? I can turn it around
   quickly.

2. Roughly when do you expect the leaderboard to reflect the new entry? I
   would like to share the result publicly - the harness is open source at
   https://github.com/Nikit117/kynex-harness and the measured per-document
   scores are in that repository, so the number can be checked document by
   document rather than taken on trust.

If you would rather announce the addition yourself when it lands - which is
your call and entirely reasonable for the benchmark's own repo - I am happy to
hold off and simply link to the leaderboard entry once it is up. Just let me
know which you prefer and I will follow that.

[Nikit117]

A correction on the cost figures

My previous comment said Document Intelligence costs USD 9.42 over 942 pages
and that only 27 of the 370 documents use it. That was wrong - I had
counted the documents whose metrics showed DI activity, not the pages the
layout analysis actually reads.

The correct accounting, measured:

	pages	cost
the structured-table pass (every page of every document)	4,869	USD 48.69
the agent's own table calls (extract_table 2,151 + find_tables 681 + analyze_with_document_intelligence 70)	2,902	USD 29.02
Document Intelligence total	7,771	USD 77.71
the model (183,567,244 in + 10,206,069 out tokens over 11,186 calls)		USD 612.01
run total		USD 689.72 over 4,869 pages = 14.17 cents per page

I have updated the leaderboard.csv row and the description to the measured
figures (14.17 cents per page).

Two related points, stated plainly:

• The rules were developed against the same 370 documents the score comes
  from. That is the nature of iterating on a public benchmark, and it means
  92.65 is optimistic about unseen documents. The rules are general statements
  about documents rather than document-specific patches, and the per-document
  scores are published in the harness repository so the distribution is visible
  and not only the mean.
• The earlier commit titles in this pull request mention the internal
  project the harness was written for. The published code does not contain
  it, and the history is a public record that I am not rewriting; the naming
  in the current tree and in the provider is what runs.

The 49-document D3 slice (0.8950 and 0.9104 in two independent runs) uses
Document Intelligence on no document, so that number is reproducible with
the model endpoint alone, and I can run it through this provider and publish
the per-document scores if that is easier than running it yourself.

[Nikit117]

The pinned revision now points at the tip of the harness repository

I have moved HARNESS_REVISION to d50a78d, which is the current tip of
https://github.com/Nikit117/kynex-harness . It includes the openai-compatible
transport you contributed in
Nikit117/kynex-harness#1 -
so the pipeline runs the same code that is in the repository, including your own
change.

The Azure path the measurement used is unchanged: the transport switch is
opt-in (ADH_MODEL_BACKEND=openai), the default detection is the same, and the
five new transport tests cover the agent tool-calling path, the vision path and
the Azure-compatible case.

[Nikit117]

Ready on my side - and a shortcut for your reproduction

Everything you asked for is in place:

• the harness is public at https://github.com/Nikit117/kynex-harness , and the
  provider now pins its current tip (d50a78d), which includes the
  openai-compatible transport from
  Nikit117/kynex-harness#1;
• the provider passes the six rule switches the measurement used, so the pinned
  run is the system that was measured;
• CI is green on the current head, and the conflicts with main are resolved
  against your latest leaderboard.csv;
• the per-document scores of the submitted run are in the harness repository
  under �enchmark_results/tui_bench/, so every number can be checked against
  a document rather than taken on trust.

One shortcut for your verification. You are running the provider in this
pull request against the published harness, and that harness does not use
Document Intelligence on any of the 49 D3 documents. So if you want a cheap
cross-check instead of a full re-run, I can run the 49-document D3 slice
through your provider in this pull request and publish the per-document
official scores next to the ones from the submitted run. Same provider, same
pinned revision, and the two sets can be compared document by document. Say the
word and I will post them here.

No rush on my side - the entry is self-reported until you reproduce it, and I
would rather you had everything you need than a merged row.

[xujustinj]

@Nikit117 here's some notes

1. It would normally take some time to get Azure set up, so I opened a PR Support OpenAI-compatible endpoints Nikit117/kynex-harness#1 to inject non-Azure AI credentials for plain LLM inference. This doesn't work for ADI of course. Therefore, your suggestion to use the non-DI documents is thoughtful, and we will do that for verification. Thanks for your patience.
2. For the record we do not have a 49-document D3 slice. There are 66 documents that have the D3 tag. Please let me know which documents you mean, but also send your original saved predictions of KYNEX on all 66 documents that have the D3 tag. You may post it here or reach me by email at justin@runllama.ai.
3. Thank you for correcting the cost. Your comment still confuses me, since "every page of every document" sounds like there should be zero documents which do not depend on ADI. Please clarify whether DI was used in a way that should affect the predictions on the D3 documents.
4. We do not consider free tier to be a valid way to measure cost, so we would use the public listing price of the tokens/pages instead.
5. FYI on novelty: we consider custom harnesses around plain LLMs to be fair game, but it is a bit less clear when you are using an entire extraction-capable service inside of your own, which you did not write yourself. Our understanding is that, since your pipeline does not use ADI's schema-constrained extraction features (https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/concept/add-on-capabilities?view=doc-intel-4.0.0&tabs=rest-api#query-fields), and since your harness contributes agentic orchestration by treating ADI as a tool, that your pipeline still makes reasonably novel contributions. These criteria apply to all future versions of KYNEX at our discretion, as there is a good chance we will someday evaluate ADI extract as a standalone pipeline, and we want to avoid undesirable overlap.
6. Did you ever try running KYNEX with more powerful models (e.g. GPT-6 series)?

[Nikit117]

Thanks for the detailed notes - and for the OpenAI-compatible transport, which
removes the Azure setup from the critical path for verification.

The saved predictions (your point 2)

Everything is attached to the v0.1.0 release of
https://github.com/Nikit117/kynex-harness :

asset	contents
kynex_predictions_all_370.zip	the saved extraction for every document of the measured run: prediction.json (the schema-shaped extraction that was scored), manifest.json, run_metrics.json (per-document telemetry) and raw_answer.txt (the agent's answer before post-processing)
kynex_predictions_d3_49.zip	the same, for the 49 D3-tagged documents of our snapshot
kynex_official_scores_370.csv	the official per-document scores of that run: value_f1, precision, recall, split, and a flag for D3 in our snapshot
kynex_d3_49_with_scores.csv	the same, restricted to the D3 subset

I sent all 370 rather than only the D3 subset, because the D3 tag is not the
same set in our snapshot and in yours - see below - so you can take exactly the
documents your snapshot tags as D3 and compare them document by document.

Which documents I mean by "the D3 slice": the 49 documents whose
data/<split>/<id>.test.json carries the tag domain:D3 in the corpus snapshot
we measured (370 documents, 252 short / 98 medium / 20 long). That set is
exactly what the harness selects with dev_pool(domain_filter="D3"), and it is
what I reported as 0.8950 and 0.9104 in two independent runs. Your snapshot
tags 66 documents as D3, so the tag assignment changed between snapshots; if
you send me the 66 ids I will confirm which of them we measured and which are
new.

How Document Intelligence enters the predictions (your point 3)

Your reading of my earlier comment was correct - it was confusing, and one of
my claims was wrong. There are three distinct ways DI reaches the output, and
here is what each one actually did over the 370 measured documents:

DI entry point	what it does	documents affected
the agent's own table tools (extract_table, find_tables, analyze_with_document_intelligence)	the agent reads a page's tables through the layout model	359 of 370 (2,902 calls)
the structured-row pass (_apply_structured_arrays)	rebuilds an array's rows from the document's tables, replacing the agent's rows when the table carries every field the agent filled	8 of 370 (real_franklin_* x3, real_bbb_service_list* x2, real_ftx_full* x2, real_imedia_full)
the OCR fallback	reads pages that have no text layer	1 of 370

So 11 documents are DI-free; the other 359 depend on it, and my earlier
"27 of 370" was the wrong measurement - it counted the documents where DI
changed the output, not the documents where DI was read from. The cost
derivation now says the same thing.

For the 49 D3 documents specifically: the agent called a DI table tool on
49 of 49 (404 calls), while the structured-row replacement fired on 0
and the OCR fallback on 0. So on that subset DI contributed only through
the agent's table reads - the two rules that rewrite values never touched it.

To turn that into a number rather than an argument, I am running the
49-document D3 slice right now with no Document Intelligence credentials at
all, and I will post both the DI-free mean and the per-document delta here.
If the native table reader lands on the same grids, the slice reproduces
without ADI; if it does not, the delta is the honest measure of what the
layout model contributes to that subset.

Cost (your point 4)

Agreed, and the row already uses the public listing price only - no free-tier
assumption is in the number:

gpt-5.4	183,567,244 input + 10,206,069 output tokens over 11,186 calls, at $2.50 / 1M input and $15.00 / 1M output = $612.01
Document Intelligence (Layout, $10 / 1,000 pages)	7,771 pages: 4,869 for the structured pass over every document, plus 2,902 for the agent's table calls = $77.71
run total	$689.72 over 4,869 pages = 14.17 cents per page

The free-tier remark in my previous comment should not have been there.

Novelty (your point 5)

Your reading matches the implementation. The only Document Intelligence model we
call is prebuilt-layout - pages, tables, words and geometry. We do not use
prebuilt-document, query-fields, or any schema-constrained extraction
feature: the schema-constrained extraction is done by the agent, and the layout
model is used as a tool. The rules that rewrite the agent's values (non-record
pruning, redaction markers, structured rows) are ours.

Understood on future versions: we will keep the harness' own extraction logic
as the system under test and treat DI as a perception tool only.

Stronger models (your point 6)

We have not run KYNEX on the GPT-6 series, and the row on the board is
specific to gpt-5.4. The harness was built and tuned for R&D against the
gpt-5.4 deployment that is hosted for us in our environment, so that is the
model the measurement covers - I did not want a number on the board labelled
with a model the run did not use.

The OpenAI-compatible transport you contributed makes a GPT-6 run possible
without Azure OpenAI, which removes the only real obstacle. If a GPT-6-series
deployment is available to you, I would rather run it on the same pipeline and
post it as a separate row than relabel this one - say the word and I will set
it up, with the per-document scores attached the same way.

What I need

Nothing is blocking me on the code side. The one item I cannot action is the
review approval on this pull request - it is a maintainer action.

To summarise the three corrections I owe you: the DI document count (27 -> 359
read from, 8 rewritten, 1 OCR), the D3 slice's dependence on DI (it does
depend on it, through the agent's table reads), and the free-tier remark.

[Nikit117]

To take the environment setup off your plate - the harness provisions itself:

git clone https://github.com/Nikit117/kynex-harness
cd kynex-harness
python scripts/verify_against_extractbench.py --data /path/to/extractbench-data --docs 5

scripts/verify_against_extractbench.py (new in the repository) creates the
virtual environment, installs the dependencies, runs the harness over the
corpus with the same six rule switches the submitted run used, and prints every
document next to the published per-document score:

[1/2] 00581-2011-p0050    ok   published=0.9583
[2/2] 00581-2011-p0052    ok   published=0.9796

Options:

flag	effect
--docs N	only the first N documents - the fastest way to see whether the pipeline matches
--split medium	one split
--data-di off	run with no Document Intelligence credentials at all
--data <path>	where the benchmark data lives (the corpus is not in this repository)

The published per-document scores of the submitted run are in the v0.1.0
release (kynex_official_scores_370.csv), so a reproduction can be compared
against them document by document rather than only in aggregate.

If the setup is still awkward from your side, tell me which step is in the way
and I will adjust the script - the point of it is that the only thing you
should have to do is run one command.

[Nikit117]

The D3 slice reproduces without Document Intelligence

I ran the 49-document D3 slice again with no Document Intelligence
credentials at all - ADH_DI_ENDPOINT and ADH_DI_API_KEY empty, so
di_configured() is false, the agent falls back to the native table reader, the
structured-row pass has no tables to read and the OCR fallback is unavailable.

	with ADI	without ADI
mean unified value F1 (49 documents)	0.8950	0.8967 (+0.0016)
documents below 0.50	0	0

The documents that moved by more than 0.02:

document	with ADI	without ADI	delta
gsa_mas_it_labor_0006	0.6442	0.7423	+0.0982
sf1449_supplies_services_0059	0.7314	0.7525	+0.0212
dd1155_schedule_continuation_0012	0.7429	0.9143	+0.1714
gov_clin_schedule_0031	0.7678	0.8982	+0.1305
sf1449_supplies_services_0047	0.7791	0.9884	+0.2093
gsa_mas_it_labor_0014	0.7843	0.6719	-0.1124
dd1155_schedule_continuation_0002	0.8182	1.0000	+0.1818
gsa_mas_it_labor_0010	0.8182	0.7273	-0.0909
gsa_mas_it_labor_0018	0.8748	0.9808	+0.1060
united_way_san_antonio_fy23_single_audit	0.9052	0.8768	-0.0284
cbp_7501_continuation_0018	0.9306	0.6172	-0.3134
gov_clin_schedule_0038	0.9425	0.9770	+0.0345
census_statab_12s0035	0.9639	0.9880	+0.0241
sf1449_supplies_services_0003	0.9925	0.9403	-0.0522
gsa_mas_it_labor_0016	0.9975	0.9033	-0.0941
gsa_mas_it_labor_0003	0.9989	0.8992	-0.0997
census_statab_12s0711	1.0000	0.9625	-0.0375
gsa_mas_it_labor_0020	1.0000	0.9785	-0.0215
jackson_college_2024_federal_awards_audit	1.0000	0.9670	-0.0330

What this settles. The D3 slice is reproducible on a setup with no
Document Intelligence at all: with the OpenAI-compatible transport from
Nikit117/kynex-harness#1 and
the model endpoint alone, the pipeline runs end to end and lands at 0.8967.
That is the number to compare against.

It also bounds what the layout model contributes on this subset. The
structured-row pass and the OCR fallback never fire here (0 and 0 of 49), so on
the D3 documents the layout model only ever reached the agent through its table
reads, and the native table reader reaches the same values. On the full
370-document set the structured-row pass does fire - on 8 documents, where it is
worth a mean of +0.138 - and the OCR fallback on 1.

To correct my previous comment precisely: DI was read from on 359 of the 370
documents of the submitted run (2,902 table calls), it rewrote the agent's rows
on 8 of them and it was needed for OCR on 1. My earlier "27 of 370" was the
wrong measurement, and the D3 slice's dependence on DI turns out to be nil in
score terms.

The saved predictions and the per-document official scores for both runs are in
the v0.1.0 release of https://github.com/Nikit117/kynex-harness , and
scripts/verify_against_extractbench.py reproduces any subset in one command.

[Nikit117]

The evidence bundle is complete now, so this closes the loop on the
predictions you asked for. Everything is on the v0.1.0 release of
https://github.com/Nikit117/kynex-harness :

asset	what it is
kynex_predictions_all_370.zip	the saved extraction of every document of the submitted run - prediction.json (what was scored), manifest.json, run_metrics.json, raw_answer.txt
kynex_predictions_d3_49.zip	the same, for the 49 documents tagged domain:D3 in our snapshot
kynex_predictions_d3_49_without_di.zip	the same, for the second run of those 49 with no Document Intelligence credentials
kynex_official_scores_370.csv	the official per-document scores of the submitted run
kynex_d3_scores_with_and_without_di.csv	the D3 scores of both runs, side by side
SHA256SUMS.txt	sha256 of every file above, plus the measured harness revision (d50a78d, also tagged extractbench-pin)
VERIFICATION.md	one page: what to run, which numbers to expect, and the caveats

VERIFICATION.md and SHA256SUMS.txt are in the repository as well, under
docs/.

So a check is now: download the bundle, sha256sum -c SHA256SUMS.txt, then

python scripts/verify_against_extractbench.py --data /path/to/extractbench-data --docs 5

and compare what it prints against kynex_official_scores_370.csv. The two
numbers to expect are 92.65 overall for the submitted run and 0.8967
for the D3 slice without Document Intelligence (0.8950 with it).

The only thing still missing on my side is your 66-document D3 list - our
snapshot tags 49, so if 17 of yours are outside it, send them over and I will
measure those too and add them to the bundle rather than leave you to guess
which ones I covered.

[Nikit117]

Nothing outstanding from my side: the points from your review are answered
inline, and the predictions, the per-document scores, a DI-free run of the D3
slice and a sha256sum manifest are all on the v0.1.0 release whenever you get
to them.

If there is anything else you would like before this can move, tell me and I
will do it - and if you send the ids of your 66-document D3 list I will measure
those too and add them to the same bundle.

No rush at all; I just wanted to make sure nothing is waiting on me.

[Nikit117]

Just checking this hasn't fallen through the cracks - it is green and mergeable
on my side.

If verification is the bottleneck, name any five documents, or any split, and
I will run them on the endpoint of your choice and attach the per-document
outputs and the scores, so the only thing left on your side is reading them.

The full 370-document corpus, the DI-free run of the D3 slice, the per-document
official scores of both and the sha256 manifest are on the v0.1.0 release, and
scripts/verify_against_extractbench.py runs any subset in one command.

[xujustinj]

Hi Nikit, I was off for a few days and I'm back to make a push on this. The problem remains that due to the sheer proportion of documents that do rely on ADI, we want to verify by making some ADI requests. We'll need to set that up internally, but higher priority things are in the way right now. Expect to be unblocked around tomorrow

[Nikit117]

Hi Justin — thanks for coming back to this.

You asked to verify by making some ADI requests. Here is the exact minimum set where ADI actually mattered in the measured run, so you don't have to run all 370 through it:

Where ADI table reads actually fired (20 documents)

structured-row pass fired on these (the rest of the 370 used native table reading):

document	structured_arrays calls	semantic F1
real_ftx_full_corrupted	1	0.2286 ← the one document below 0.50
real_wright_co_mn	1	0.7223
real_franklin_aug_2025	1	0.7488
real_pueblo_oct_2025	2	0.9295
real_franklin_nov_2025	1	0.9625
real_franklin_sep_2025	1	0.9625
oklahoma_county_purchase_order_avl_systems_corrupte…	1	0.9122
stephenville-axon-invoice	1	0.9466
real_ftx_full	1	0.9935
real_imedia_full	1	0.9872
real_imedia_full_corrupted	1	0.9650 ← OCR fallback doc (0.10 → 0.97)
real_freer_register_full	1	0.9928
real_bbb_service_list	1	0.9977
real_bbb_service_list_corrupted	1	0.9926
real_sm0801_dispositions_full	1	0.9857
real_sm0801_dispositions_full_corrupted	1	0.9857
real_sm0801_eco_full	1	0.9911
real_sm0801_eco_full_corrupted	1	0.9942
texas_facilities_commission_purchase_order_shelton…	1	0.9861
texas_facilities_commission_purchase_order_shelton…	1	0.9859

OCR fallback — exactly one document

real_imedia_full_corrupted needed OCR (0.10 → 0.97).

Bottom line for your ADI verification

• 20 documents where structured-row pass fired (ADI table reads) — test these first.
• 1 document where OCR fallback fired (real_imedia_full_corrupted).
• Everything else (349 of 370) used native table reading without ADI at all.

Total measured DI usage: 359 of 370 docs read tables through DI (2,902 calls), ADI rewrote rows on these ~20 and was needed for OCR on 1. DI cost was $77.71 of the $689.72 total (~11%), so even verifying just the 20 structured-array documents is a tiny fraction of spend.

Once your internal ADI setup is ready, run those ~20 docs and compare against the per-document scores in the v0.1.0 release (kynex_official_scores_370.csv). Let me know if you want me to pull any specific document's raw output or run_metrics too.

No rush — I know internal setup takes time.

[Nikit117]

One more thing on the ADI question — since you said you want to verify with some ADI requests, here is what happens without it:

D3 slice (49 docs), no Document Intelligence at all:

• with DI: 0.8950
• without DI: 0.8967
• below 0.50: 0 in both

So on the D3 slice the score is statistically identical whether ADI is on or off. DI only really mattered on ~20 documents where the structured-row pass fired (listed in my previous comment) and 1 OCR document.

If your internal ADI setup is blocked, you can still verify the run without it — the number barely moves. ADI cost was $77.71 of $689.72 (~11%), so skipping it doesn't cost you much score either way.

Let me know if you want me to pull the per-document delta for those 20 structured-array documents so you can see exactly which ones change with/without ADI.

[Nikit117]

Hi Justin — thanks for coming back to this.

I want to be transparent about my side: my team is evaluating whether to adopt this harness for an internal R&D project with significant investment behind it, and we're working against an internal timeline to decide whether to proceed.

I'm not asking you to rush or bypass your review process at all — I know benchmark submissions need careful verification, and I respect that you need to do your due diligence with ADI internally.

I'm just flagging this in case a timely response would help both of us plan. If there's anything I can provide to speed up verification — specific documents, raw outputs, or running a subset on your endpoint — I'm happy to do it.

No pressure from my side whatsoever. I know internal setup takes time, and I'll wait however long you need.

[Nikit117]

@xujustinj just checking in on this — CI is green and the PR is mergeable on my side. Let me know if you need anything else from me to move this forward. No rush at all — take your time.

[xujustinj]

@Nikit117 I opened a PR last night on the KYNEX repo fixing cost accounting, but you closed it during your rebase of the entire repo. I've reopened it as Nikit117/kynex-harness#3, please take a look.

Good news, I have run reproductions on a subset of documents (with ADI enabled!) and confirmed your accuracy metrics on all documents run so far. However I'm waiting for PR 3 to land before I reproduce more documents; it is needed to fix the accuracy of the cost estimates. Our independent cost estimates seem to show that your KYNEX pipeline is overestimating its own costs by quite a lot. Hence it should benefit you to fix this accounting issue.

[Nikit117]

Hi @xujustinj — thanks for confirming the accuracy metrics and for the usage accounting fix, I've merged PR #3 on the harness repo.

It's past midnight here in India, so just checking — is there anything else you need from me tonight, or any PRs/changes for me to review before I head off? Happy to stay up a bit longer if it helps move things forward. Otherwise I'll pick it up first thing tomorrow.

[xujustinj]

Thanks Nikit, nothing else needed. I'll bump the version in this PR and then we will finish verifying. Thanks for your patience, and have a good night!

[Nikit117]

Hi @xujustinj — since you mentioned the cost was overestimated and you've already fixed the accounting in PR #3 (which I've merged), could you update the cost columns in the leaderboard CSV directly from your reproduction runs? You'll have the accurate numbers from your end. Happy to help if needed, but since you're already running the verification it might be easier from your side.

[Nikit117]

Hi @xujustinj — just wondering, do you have a tentative timeline for when this PR might be merged? No rush at all, just helps me plan on my end.

[xujustinj]

@Nikit117 sorry for no update last night, I got sick. I wasn't expecting your harness to take so long to run, but I've bumped up the concurrency and we should expect to have it done within 12 hours.

By the way, we are considering publishing pipeline results for reproducibility. We are storing your traces, but please let us know whether you would like those to be published or if you would prefer we publish only the final outputs.

[Nikit117]

Thanks for the update Justin, hope you feel better soon!

Regarding the traces — I'd prefer if only the final outputs are published, not the traces. The pipeline internals are proprietary.

Also, if possible, could we get the leaderboard updated soon? That's really the main thing on my end right now. Everything else we can figure out later — no rush on the rest.

Thanks again for all your work on this!

[xujustinj]

Got it. Since your harness is open source though, nothing stops someone else from just running it and obtaining their own traces. The difference is just that they have to pay for the inference.

We will update the leaderboard as soon as the reproduction run finishes, including merging this PR with our updated costs.

[Nikit117]

That's a fair point — since the repo is open source, traces aren't really a secret. I'm fine with you publishing them if it helps with reproducibility. Thanks for checking though!

And that's great to hear about the leaderboard! Really excited to see KYNEX up there — been looking forward to this. Can't wait for the run to finish!

[Nikit117]

Hi Justin — just wanted to check if there's anything you need from me before I sign off. I have an early morning flight tomorrow, so I'll be stepping away shortly. Happy to help with anything if needed — otherwise, I'll catch up on any updates once I'm back. Thanks again for all your effort on this!

[xujustinj]

@Nikit117 there is actually one last thing... what logo do you want to use to represent KYNEX on our leaderboard?

[Nikit117]

Here's the KYNEX logo for the leaderboard:

https://raw.githubusercontent.com/Nikit117/kynex-harness/main/docs/images/logo-combo-3.svg

Let me know if you need it in a different format or size!
https://github.com/Nikit117/kynex-harness/blob/main/docs/images/logo-combo-3.svg