Measured, pre-registered evaluation of AI systems.
An independent, pre-registered evaluation of TypeSafe AI's Jev — 5,721 calls, 21 experiments, $0.176.
Model typesafe/jev-1.13-20260917, accessed through OpenRouter, measured from
Western Europe on 20 September 2026.
📄 Read the full report → 📊 All 50 predictions and their outcomes →
-
Latency is a fixed cost, not a workload cost. ~430 ms floor. 800 typed judgements in one call: 985 ms, $0.00075. Chaining two Jev calls wastes 430 ms for nothing.
-
95.9 % zero-shot on our 400-item benchmark, versus 77.2 % for hand-written keywords and 66.0 % for supervised TF-IDF + logistic regression that saw every label.
-
TypeSafe's own documentation understates the model. Its "jaggedness" page says Jev cannot compare numbers and that date ordering is unreliable. We measure 99.6 % across 13 designs. Negation, another listed failure mode, scores 100 %.
-
It always answers. A cake recipe is classified as a technical issue at 0.94 confidence; random letters at 0.97. Without an explicit "none of these" option, 0 of 30 out-of-scope messages were flagged — at 0.99 confidence.
-
Gate at 0.99 or not at all. Accuracy above threshold is flat from 0.50 to 0.95, then jumps to 100 % at 0.99, covering 60.2 % of traffic.
-
Eight concurrent requests is the operating point. Throughput saturates near 11 req/s.
-
Wrong criteria descriptions are catastrophic: 16.7 %, below the 25 % random floor. Missing descriptions cost only 0.8 points.
Everything here was produced in a single evening. It is a wide, shallow pass: 21 experiments across the whole model, none of them taken as far as it could go. Every number below is a first measurement, not a final one.
This repository is updated daily. The plan, in order:
- Separate the model from the gateway. The ~430 ms floor is the single biggest open question in this report, and native API access settles it.
- Go deeper on every axis. Counting and cardinality are mapped coarsely;
the calibration curve deserves ten times the items; the
scoreprimitive has had one experiment and needs five. - Automate the whole suite. The harness is already parameterised. The goal is a single command that re-runs all 21 experiments against any model version and diffs the results — so a silent model update shows up as a diff, not as a surprise.
- More languages, more adversarial surface, more failure modes. Ten of the vendor's documented weaknesses have been tested. There are more, and there are failure modes nobody has written down yet.
- Other models. The method in
METHOD.mdis not Jev-specific. Jev is the first subject, not the only one.
Corrections and replications are the fastest way to make this better. If a number here is wrong, the harness is built so you can prove it.
- One call, many questions — never chain Jev calls.
- Eight concurrent requests; beyond that you buy latency, not throughput.
- Gate at 0.99, or do not gate.
- Always offer an explicit "none of these" option.
- Use
noulwhen you need a number — twice as well calibrated as achoice's confidence. But do not threshold it at 0.5; it is not reproducible there. - Keep criteria short or omit them — but never let them be wrong.
- Fix your option order for ambiguous tasks (up to 13 points of movement).
- Do not put it in a sub-300 ms hot loop from Europe.
- Do not use it to count beyond a handful of items.
- Test the documentation's warnings yourself — several did not reproduce.
Twelve rules, stated in full in the report. The three that matter most:
- Every hypothesis written before collection. 50 predictions, timestamped in
PREREGISTRATION.md. 26 confirmed, 21 falsified. - Falsified predictions are published, including the systematic bias they reveal: 18 of our 21 misses were in the same direction — we kept predicting failure and kept being wrong.
- Every surprising result replicated with a different design before publication. The "documentation understates the model" claim was re-tested across 13 independent designs before we wrote it down.
An independent recount script (analysis/recount.py) recomputes every headline
figure from raw JSONL without importing any analysis code.
git clone https://github.com/priorbench/jev
cd jev
python3.11 -m venv .venv
./.venv/bin/pip install -r requirements.txt
./.venv/bin/python setkey.py # paste an OpenRouter key; never echoed
./.venv/bin/python latency/bench.py # 491 calls, ~4 min
./.venv/bin/python probes/t6_banc.py # 800 calls, ~1 min
./.venv/bin/python analysis/recount.py # verify the totalsFull suite: 5,721 calls, $0.176, ~25 minutes at 8-way concurrency. Seeds fixed
at 20260920. Every raw response is in */raw/*.jsonl.
paper/REPORT.md the report
paper/tables/predictions.md all 50 predictions and outcomes
PREREGISTRATION.md hypotheses, timestamped before each wave
bench/reference_set.py the 400-item benchmark
baselines/no_ai.py keyword, TF-IDF+LR and random baselines
probes/ the experiments
latency/ latency harness
analysis/ cross-cutting analysis and independent recount
*/raw/*.jsonl every raw response (published)
- Everything goes through OpenRouter. We cannot separate model latency from gateway latency. The ~430 ms fixed cost is measured; its cause is hypothesis. Anyone with native API access can settle this in twenty minutes.
- Our benchmarks are ours — reproducible, but not recognised standards.
- One location, one day, one model version.
- Some classification ground truth is disputable — flagged, reported separately, never folded into headline figures.
Corrections welcome. The harness is built to be re-run against them.
The twelve rules are documented once, separately, in
METHOD.md — they apply to every PriorBench evaluation, not just
this one.
1j6c (2026). PriorBench — Jev: an independent, pre-registered evaluation
of TypeSafe AI's System One model. https://github.com/priorbench/jev
A machine-readable CITATION.cff is included.
1j6c@proton.me — corrections, replications and disagreements all welcome.
MIT