MillerQA

Benchmark data and zero-shot inference harness accompanying the paper.

This repository contains two things:

1. The benchmark suite — 24 multiple-choice benchmarks, released in both their original (pre-QC) and curated (post-QC) form. 23 of the 24 are redistributed here; MedBullets is omitted for licensing reasons (see Licensing) and is linked instead.
2. The zero-shot harness — the code used to run the models over the suite via OpenRouter.

benchmarks/
  pre_qc/            23 of 24 shipped, 9,712 items — the original suite, before QC
    <source>/        one directory per upstream dataset
      LICENSE        that dataset's licence and attribution
      *.json         the benchmarks derived from it
  curated/           23 of 24 shipped, 9,406 items — the post-QC suite used in the paper
    <source>/        same 15 source directories
  LICENSES.csv       benchmark → source → SPDX identifier, machine-readable
  _manifest.csv      per-benchmark item counts, pre-QC vs curated
  _dropped_items.csv every item removed by QC, with the reason
harness/             zero-shot inference code
docs/                figures used in this README
LICENSE              licensing for the collection — read this first
requirements.txt
.env.example

The 24 benchmarks come from 15 upstream sources under several different licences, so they are grouped by source and each group carries its own LICENSE. See Licensing — two subsets are non-commercial only, and one is not redistributed at all.

The paper analyses the full suite of 24 benchmarks (10,020 pre-QC / 9,594 curated). This repository ships 23 of them (9,712 / 9,406); MedBullets is excluded, with its item ids provided so the full suite can be reconstructed.

The benchmark suite

The suite spans five competency dimensions. Counts below are per benchmark, in both releases; the source dataset is given as its Hugging Face identifier where the set was pulled from the Hub.

Clinical Judgment

Benchmark	File	Pre-QC	Curated	Source dataset	Licence	Description
CUPCase	cupcase	700	700	CUPCase	Apache-2.0	Diagnosis from complex real-world clinical case reports
HLE	hle	146	128	cais/hle	MIT	Humanity's Last Exam — expert-level frontier questions; biology/medicine subset, text-only
MHQA	mhqa	624	616	MHQA	CC BY-NC 2.0	Mental-health research literature QA
MedCalc	medcalc	340	340	ncbi/MedCalc-Bench-v1.0	CC BY-SA 4.0	Medical calculation from clinical vignettes; integer/decimal outputs
MetaMedQA	metamedqa	676	658	Maximegmd/MetaMedQA	CC BY 4.0	USMLE-style QA with unknown / unanswerable options (metacognition)
Triage	triage	86	84	NLie2/TRIAGE	CC BY 4.0	Mass-casualty triage, recast as MCQ over a fixed 4-tier scale

Robustness and Fairness

Benchmark	File	Pre-QC	Curated	Source dataset	Licence	Description
FairMedQA	fairmedqa	960	960	FairMedQA	CC BY 4.0	Demographic bias in clinical QA via counterfactual vignettes (race / income / gender)
Global MMLU	global_mmlu	750	705	Global MMLU	Apache-2.0	Multilingual professional-medicine exam questions across 15 languages
TruthfulQA	truthfulqa	790	790	TruthfulQA	Apache-2.0	Truthfulness and resistance to common misconceptions

Ethical and Professional Reasoning

Benchmark	File	Pre-QC	Curated	Source dataset	Licence	Description
MMLU-Pro (Phil)	mmlupro_phil	237	200	TIGER-Lab/MMLU-Pro	MIT	Philosophy, formal logic and moral-disputes categories
MedEthicsQA	medethicsqa	1,000	928	MedEthicsQA	CC BY-NC 4.0	Medical ethics, stratified across the four core principles
TridentBench	tridentbench	854	842	TridentBench	MIT	Identifying which AMA Principle of Medical Ethics a scenario violates

Medical Knowledge

Benchmark	File	Pre-QC	Curated	Source dataset	Licence	Description
MMLU-Pro (Med)	mmlupro_med	326	252	TIGER-Lab/MMLU-Pro	MIT	Advanced clinical-knowledge and professional-medicine exam questions
MedBullets ⚠️	medbullets	308	188	LangAGI-Lab/medbullets_op5	not redistributed	USMLE Step 2/3-style clinical vignettes — data file not included, see Licensing
PubMedQA	pubmedqa	1,000	998	qiaojin/PubMedQA	MIT	Yes / no / maybe QA over biomedical research abstracts

Legal and Regulatory

Benchmark	File	Pre-QC	Curated	Source dataset	Licence	Description
LB (ipi)	ipi_legalbench	133	133	nguha/legalbench	CC BY 4.0	Insurance policy interpretation — is the claim covered?
LB (lhb)	lhb_legalbench	66	65	nguha/legalbench	CC BY 4.0	Learned Hands: does the post concern public benefits or social services?
LB (lhh)	lhh_legalbench	226	225	nguha/legalbench	CC BY 4.0	Learned Hands: does the post concern health care or medico-legal issues?
LB (oad)	oad_legalbench	100	95	nguha/legalbench	CC BY 4.0	OPP-115: user access, edit and deletion clauses
LB (odr)	odr_legalbench	100	94	nguha/legalbench	CC BY 4.0	OPP-115: data-retention clauses
LB (ods)	ods_legalbench	100	97	nguha/legalbench	CC BY 4.0	OPP-115: data-security clauses
LB (ofp)	ofp_legalbench	100	100	nguha/legalbench	CC BY 4.0	OPP-115: first-party collection and use clauses
LB (otp)	otp_legalbench	100	98	nguha/legalbench	CC BY 4.0	OPP-115: third-party sharing and collection clauses
LB (tos)	tos_legalbench	298	298	nguha/legalbench	CC BY 4.0	Classifying potentially unfair Terms-of-Service clauses

Full citations for each source dataset are given in the paper.

Licensing

This repository is a mixed-licence collection. There is no single licence covering all of it, and the top-level LICENSE does not grant rights over the benchmark data.

Per-source licences. Each of the 15 upstream datasets has its own directory inside both releases, carrying that dataset's LICENSE file — its SPDX identifier, a link to the full licence text, the upstream source, and a record of how we modified the data. benchmarks/LICENSES.csv gives the same mapping in machine-readable form.

benchmarks/curated/legalbench/LICENSE            ← CC BY 4.0, covers the 9 LB sets
benchmarks/curated/legalbench/tos_legalbench.json

Non-commercial subsets. Two benchmarks may not be used commercially under any circumstances, and any commercial use of this repository must exclude them:

Benchmark	Source	Licence
mhqa	MHQA	CC BY-NC 2.0
medethicsqa	MedEthicsQA	CC BY-NC 4.0

medcalc is CC BY-SA 4.0, which additionally imposes share-alike on adapted versions of that subset.

Subset that is not redistributed. medbullets is part of the suite as analysed, but its data file is not included in this repository. No licence could be established — neither the Hugging Face mirror we drew from nor the originating repository declares one — and the underlying questions derive from a commercial question bank, so we do not redistribute them.

benchmarks/<release>/medbullets/ still contains a LICENSE explaining the omission and an item_ids.csv listing the exact ids in each release, so the full suite can be rebuilt from the upstream source. The harness runs the other 23 benchmarks unchanged when the file is absent.

Our contributions — the harness code, the QC layer, the harmonised schema and the option sets we wrote when recasting source tasks as multiple choice — are released under CC BY 4.0. The data items themselves are not ours to relicense.

The CC licences require attribution and an indication of changes: when reusing a subset, cite the original dataset alongside this work, and see that source directory's LICENSE for what we changed.

Benchmark format

Every file is a JSON array of items with a common schema:

{
  "id": "professional_medicine/test/131/en",
  "question": "A 62-year-old man ...",
  "options": ["Begin ...", "Order ...", "Discharge ...", "Consult ..."],
  "target": 2,
  "kind": "professional_medicine"
}

target is the 0-based index into options of the correct answer, in the order given in the file. kind is an optional subcategory label and is absent in some benchmarks. Question text and options are kept fully separable, which is what makes the choice-only baseline possible.

pre-QC vs curated

pre_qc/ is the suite as originally assembled: 10,020 items, of which 9,712 are shipped here. curated/ is the same suite after the quality-control pass described in the paper: 9,594 items, of which 9,406 are shipped here, with defective items excluded and duplicates removed. Three benchmarks were rebuilt from source during QC rather than filtered — medcalc, triage and truthfulqa — and both directories carry the rebuilt versions, so the two releases hold the same 24 benchmark names and differ only in which items survive. The replaces_retired column of _manifest.csv records the original name in each case.

_manifest.csv gives the per-benchmark counts on both sides; _dropped_items.csv lists all 447 removed items with the QC check that caught each one and the decision taken. Results reported in the paper are computed on the curated set, which is also the harness default.

Running the zero-shot harness

1. Install

Python 3.10+ (the code uses X | Y type syntax).

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Configure

cp .env.example .env

Then edit .env and set OPENROUTER_API_KEY. Everything else is optional; the defaults shown in that file are the ones used for the paper — notably SHUFFLE_SEED=42, which controls answer-option shuffling and must be kept unchanged to reproduce the published runs.

3. Run

All commands are run from inside harness/.

cd harness

# Print the execution plan without spending anything — always start here
python run_experiment.py --dry-run

# One model, one benchmark
python run_experiment.py --models gpt-4o --benchmarks fairmedqa

# The full paper run: every model configuration × all 24 benchmarks
python run_experiment.py

# Run against the original pre-QC release instead
python run_experiment.py --data-dir ../benchmarks/pre_qc --experiment-id zeroshot_preqc

The full run is 26 model configurations × 9,406 shipped items ≈ 245k API calls (9,594 if you reconstruct MedBullets). Cost and time are substantial. Use --dry-run first and scope with --models / --benchmarks.

Useful flags:

Flag	Effect
--models LABEL ...	Restrict to model labels from harness/config.py (default: all 26)
--benchmarks NAME ...	Restrict to benchmark file stems (default: all 24)
--data-dir PATH	Benchmark directory to load (default: benchmarks/curated)
--experiment-id ID	Output folder name under results/ (default: zeroshot_v1)
--concurrency N	Max in-flight API calls (default: 25)
--retry-failed	Re-run only items that errored or failed to parse
--force	Ignore resume state and re-run everything
--dry-run	Print the plan and exit without calling the API
-v	Debug logging

Runs are resumable: results are appended per model/benchmark, and re-running the same command picks up where it left off rather than repeating completed items. Use --force only when you intend to discard prior results.

4. Output

results/<experiment_id>/
  <model_label>/<benchmark>.jsonl   one JSON record per item
  csv/master.csv                    all records flattened
  csv/by_model/<model_label>.csv
  csv/by_benchmark/<benchmark>.csv
  run.log

Each record carries the prompt condition, the model's answer, and accounting: permutation and correct_letter (the shuffle actually shown to this model), selected_letter, selected_original_index, is_correct, confidence, explanation, raw_content, token counts, cost_usd, latency_ms, and error. Because options are shuffled per (item, model), correctness is scored by mapping the selected letter back through permutation to the original option index and comparing against target — never by comparing letters directly.

is_correct is null for items with no parseable answer (a refusal, a truncation, a malformed response). These are unscored, not wrong; the paper drops them from the denominator and reports the counts separately.

What is in harness/

File	Role
run_experiment.py	CLI entry point
config.py	Paths, API settings, and the 26-entry model registry
data_loader.py	Loads and normalises the benchmark JSON
prompt_builder.py	Deterministic per-(item, model) option shuffling and prompt assembly
api_client.py	OpenRouter calls, retries, reasoning-config handling
batch_runner.py	Concurrency, resume, per-model orchestration
output_parser.py	Response parsing into the flat record schema
make_csv.py	JSONL → CSV

The model registry in config.py holds the 26 configurations that were dispatched. Four of them returned an API error on every request and produced no usable data (gpt-5, o3-mini, lfm2-8b, aion-2-0_base), so the paper reports 22 configurations across 19 models; they are left in the registry so the run is reproducible as executed. Models tested both with and without reasoning appear twice, with different labels and a reasoning_config that is passed through to the provider. Some entries carry model-specific max_tokens; the comments in that file record why.

Citation

Please cite the accompanying paper. Individual benchmarks are derived from previously published sources and remain subject to their original licences; see the paper for the full provenance of each of the 24 sets.