Add reproducible Kraken olmOCR-bench run and results

experiments/kraken-olmocr-bench/README.md

@@ -0,0 +1,134 @@
+# Kraken on olmOCR-bench
+
+This experiment runs bundled BLLA segmentation and the PP-OCRv6 medium recognizer
+through Kraken's Python API on Hugging Face Jobs, then scores the raw outputs with
+the pinned official olmOCR-bench implementation. It includes a 21-page smoke
+test (three fixed pages per category) and the full 1,403-page benchmark.
+
+The [completed 9 September 2026 run](RESULTS-2026-09-09.md) scored **36.8% overall**
+(95% CI 35.9–37.7), including **74.4% on long/tiny text**. Its
+[public artifact bundle](https://huggingface.co/buckets/davanstrien/kraken-olmbench-results-20260909)
+contains scores and execution metadata; source document images and text remain
+in a separate archive.
+
+## Method
+
+- Pin Kraken to `3e6893a0d5eb06273494bfcdb9d3cac6159e0611`, which includes the GPU
+  tensor-placement fix. The bundled segmenter's identity follows this pin.
+- Pin the model and dataset revisions in `runner.py`; verify recognition weight
+  bytes against SHA-256 before loading.
+- Render PDF pages to RGB at 300 DPI using pypdfium2. No PDF text extraction.
+- The pinned `old_scans/48.pdf`, `51.pdf` and `57.pdf` pages render to
+  253,864,905, 331,857,945 and 323,914,932 pixels at this DPI. Their cached PNGs
+  have explicit Pillow size allowances and dimension checks; other pages retain
+  Pillow's default limit. No page is downscaled to bypass it.
+- Use fp32, batch size 32, in-process line extraction, default reading order,
+  bidi reordering and 16-pixel recognition padding. Torch CPU threads are limited
+  to four on A10G-small.
+- Select smoke pages by a deterministic hash of the page identifier, before
+  observing predictions. Assign the full manifest to four category-balanced
+  shards. There is no official modern-only subset.
+- The stock arm invokes Kraken's high-level segmentation and recognition methods
+  for each page. The staged arm configures each model once, runs segmentation
+  across the manifest, and then recognizes the saved segmentations. It retains
+  the high-level merge, reading-order and decoding paths.
+- Compare every coordinate and ordering field plus raw line-joined text. Random
+  line/region UUIDs are canonicalized while preserving identity relationships;
+  image filenames are excluded from geometry comparison. Persist geometry without
+  discarding any fields, and check its JSON round-trip before recognition.
+- Exercise resume with prediction methods replaced by failure sentinels: completed
+  pages must be loaded without invoking either model again.
+
+Preparation is implemented with an instance-local guard around
+`prepare_for_inference`; a different configuration raises an error. This does not
+change network weights, image scaling, thresholds, polygonization or decoding.
+Both models currently remain loaded during the staged passes.
+
+## Run on Jobs
+
+All runtime checks and tests run on Jobs, including Ruff. The image is pinned by
+digest in `launch.sh` and in provenance. The committed `uv.lock` and
+`scoring/uv.lock` were resolved and tested on Jobs. Bootstraps prefer these locks
+and use `uv sync --locked`. Their fallback, for initial development without a
+committed lock, saves a Jobs-generated lock to the Bucket. The compatible pre-1.0
+Hub client is used only for downloads; mounted Bucket I/O does not depend on its
+newer Bucket APIs.
+
+```sh
+experiment=experiments/kraken-olmocr-bench
+bucket=YOUR_NAME/YOUR_BUCKET
+hf buckets create "$bucket" --private
+sh "$experiment/launch.sh" smoke "$bucket" smoke-v1
+# Replace these uppercase Job ID placeholders with the IDs printed by launch.sh.
+hf jobs wait SMOKE_JOB_ID
+sh "$experiment/launch.sh" score-smoke "$bucket" smoke-v1
+hf jobs wait SMOKE_SCORER_JOB_ID
+sh "$experiment/launch.sh" full "$bucket" full-v1
+hf jobs wait SHARD_0_JOB_ID SHARD_1_JOB_ID SHARD_2_JOB_ID SHARD_3_JOB_ID
+sh "$experiment/launch.sh" score-full "$bucket" full-v1
+hf jobs wait FULL_SCORER_JOB_ID
+```
+
+Each phase is asynchronous. Require a `COMPLETED` Job state, passing smoke checks
+and zero scorer candidate errors before moving on; waiting alone is not evidence
+of success. The full entrypoint uses the same `runner.py` with `--mode full
+--shard 0` through `3`. Give each new configuration a new run identifier; resume
+only identical code and settings. The smoke's `--verify-only` option compares
+previously saved outputs and tests resumption without repeating predictions.
+
+Defaults cap the GPU smoke at 50 minutes, each full shard at three hours, and each
+CPU scoring Job at one hour. These are per-Job limits. Account for all attempts
+when applying a cumulative budget; do not blindly relaunch all shards after one
+fails. Inspect `errors.json` and resume only the incomplete shard with the same
+run identifier and inference settings. Read retries are bounded at 30 seconds for
+mounted-Bucket visibility delays; missing/corrupt artifacts remain hard failures.
+For example, `sh "$experiment/launch.sh" resume-0 "$bucket" full-v1` resumes
+shard zero with a 45-minute cap. A successful complete resume preserves the earlier
+error ledger as `errors-before-successful-resume.json` before clearing its active
+`errors.json` marker. Include both Job attempts in the cost total.
+Hub downloads retry transient transport errors and HTTP 408/429/500/502/503/504 responses
+up to five attempts, with bounded backoff. Permanent errors still fail the Job;
+successful retries retain the same revision and subsequent byte-identity checks.
+
+## Artifacts and interpretation
+
+The Bucket stores the manifest, environment/package versions, input hashes,
+geometry, raw candidate Markdown, line predictions, stage timings, errors and
+fixed-sample overlays. Candidate names follow the official scorer convention,
+`{pdf_stem}_pg{page}_repeat1.md`. Empty text is valid if segmentation found no
+lines; exceptions are separately reported and make the run fail.
+
+`summary.json` records inference-stage totals and elapsed time. The first page
+also has a CPU profile, so its runtime includes profiling overhead. Stock runs
+before staged in this smoke: cold-start and cache effects mean the result cannot
+isolate the benefit of model preparation or support a precise speedup claim.
+Use Job running durations, including setup and failed attempts, for compute cost.
+
+`score.py` uses official scorer revision
+`f7cfe4c22098b154c76b6ec950d1c0a464eecf8d` in `allenai/olmocr`, including baseline
+tests and its category-aware bootstrap. It checks complete candidate coverage,
+raw-text identity and PDF hashes before calling the official `main()` function.
+Thin observers capture its returned test results and confidence interval without
+replacing scoring code. The official overall is the mean of per-JSONL category
+scores, including the automatic baseline category; it is not the micro-average.
+Both are exported, with raw test results and the complete official console output.
+NumPy/Python seeds are fixed at 20260909 for 1,000 bootstrap samples.
+
+Scoring artifacts live under `scores/RUN_ID/`: `summary.json`,
+`test-results.jsonl`, `coverage.json`, `candidate-errors.json`,
+`official-output.txt`, a PNG/SVG category chart, and bounded diagnostic examples
+with original page, line overlay, prediction and test explanation. Diagnostic
+selection happens after scoring and never changes the scored set. Full math/table
+tests assess structured output that plain line text does not provide. Smoke
+scores validate execution and should not be presented as benchmark estimates.
+
+To retrieve private artifacts after a successful run:
+
+```sh
+hf buckets sync "hf://buckets/$bucket/scores/full-v1" ./kraken-results
+```
+
+The recognizer contains approximately 15.92 million parameters; the complete
+pipeline also includes a segmenter. Its training card mentions historical,
+contemporary and born-digital material without proportions, so results should
+not be framed as proof of transfer from exclusively or mostly historical data.

experiments/kraken-olmocr-bench/RESULTS-2026-09-09.md

@@ -0,0 +1,93 @@
+# Full Kraken run — 9 September 2026
+
+Kraken's bundled BLLA segmentation and PP-OCRv6 medium recognition scored
+**36.8% overall** on all **1,403 olmOCR-bench pages**, with a 95% bootstrap
+confidence interval of **35.9–37.7%**. All pages completed; the official scorer
+evaluated **8,413 tests** with zero unresolved producer or candidate errors.
+
+The [public results bundle](https://huggingface.co/buckets/davanstrien/kraken-olmbench-results-20260909)
+contains the exact aggregate result, per-test outcomes, per-page timings and
+hashes, environment/source snapshots, a category chart and the complete compute
+cost ledger. [Method and launch commands](README.md).
+
+| Category | Passed / tests | Score |
+|---|---:|---:|
+| Long / tiny text | 329 / 442 | 74.4% |
+| Multiple columns | 512 / 884 | 57.9% |
+| Headers / footers | 285 / 760 | 37.5% |
+| Old scans | 127 / 526 | 24.1% |
+| Tables | 2 / 1,022 | 0.2% |
+| Old scans: math | 0 / 458 | 0.0% |
+| ArXiv math | 0 / 2,927 | 0.0% |
+| Automatic baseline category | 1,394 / 1,394 | 100.0% |
+
+The official macro score averages these eight rows. Baseline checks overlap the
+page categories. There are nine additional baseline checks in the manifests:
+seven in headers/footers and two in tables. All 1,403 baseline checks passed;
+**zero of 1,020 actual table-structure tests** and **zero of 3,385 math tests**
+passed. Micro-average: 31.49% (2,649 / 8,413).
+
+## Configuration and validation
+
+The recognizer has approximately 15.92M parameters; the segmenter is additional.
+Pages were rendered to RGB at 300 DPI and processed in CUDA fp32, batch size 32,
+with four Torch CPU threads and four A10G-small shards. Output was raw
+newline-joined text without dehyphenation, header suppression, table reconstruction
+or LaTeX generation. The official scorer ran on CPU-upgrade.
+
+All inference, runtime checks and scoring ran on Hugging Face Jobs. The 21-page
+smoke verified identical ordered geometry and text for stock/staged inference
+and completed-page resumption without model calls. Staging did not establish a
+meaningful speedup.
+
+| Component | Pinned revision |
+|---|---|
+| Reproducer code | `8ea25cb7426a657947b94c413cf0e52bfcd066e1` |
+| Kraken | `3e6893a0d5eb06273494bfcdb9d3cac6159e0611` |
+| `small-models-for-glam/kraken-ppocrv6-medium` | `b26109b54a73e27b91c3fa83558ccba1ed485e03` |
+| `allenai/olmOCR-bench` | `54a96a6fb6a2bd3b297e59869491db4d3625b711` |
+| Official `allenai/olmocr` scorer | `f7cfe4c22098b154c76b6ec950d1c0a464eecf8d` |
+
+The bundle's `run.json` records configuration and the recognition weight hash.
+Container digest, package versions, dependency locks and source hashes are in
+the environment records. Bootstrap: seed 20260909, 1,000 samples, 95% confidence.
+
+## Runtime and cost
+
+- Segmentation: **24,390.1s**, or **85.5%** of aggregate measured inference time.
+- Recognition: **4,139.8s**. Total aggregate inference: **7.925 hours**.
+- Full inference including recovery: **2h 26m 25s** elapsed; through scoring
+  completion: **2h 53m 28s**.
+- **$9.56** estimated list-rate compute, or **$9.65** with each Job rounded up to
+  a minute, including all 15 setup/smoke/full/recovery/scoring attempts. All stopped.
+  Costs are before credits and are not an invoice.
+
+Three very large old scans needed page-specific Pillow size allowances and
+resumed inference after the original attempt completed 1,400 pages. Their
+resolution and model settings stayed unchanged. The first full scorer attempt
+failed on a transient Hub HTTP 502; bounded download retries allowed scoring to
+complete. Both source revisions and every attempt remain accounted for.
+
+## Interpretation and artifacts
+
+These are pipeline test pass rates, not CER or isolated reading-order accuracy.
+Headers/footers mainly test omission, and order tests require matching text
+anchors. In multiple columns, 356 of 372 failures lacked an anchor, preventing
+the test from assessing order. The model's training card does not quantify the
+historical share of its training data; this run does not isolate a domain-transfer
+effect.
+
+```sh
+hf buckets sync hf://buckets/davanstrien/kraken-olmbench-results-20260909 ./kraken-run-results
+```
+
+The public bundle includes all test outcomes with reference passages, original
+source URLs and detailed explanations removed. Page statistics include input and
+prediction hashes without the predictions themselves. Source document images,
+transcriptions, geometry and detailed scorer logs remain in a separate private
+archive. Public metadata supports aggregate checks; independently rescoring
+predictions requires regenerating them with the pinned scripts and upstream dataset.
+
+Contains information from [allenai/olmOCR-bench](https://huggingface.co/datasets/allenai/olmOCR-bench),
+made available under the [ODC Attribution License 1.0](https://opendatacommons.org/licenses/by/1-0/).
+The bundle includes `NOTICE.md` and a SHA-256 manifest.

experiments/kraken-olmocr-bench/bootstrap.sh

@@ -0,0 +1,18 @@
+#!/bin/sh
+set -eu
+mkdir -p /tmp/kraken-bench
+cp /input/*.py /input/pyproject.toml /tmp/kraken-bench/
+cd /tmp/kraken-bench
+if [ -f /input/uv.lock ]; then
+    cp /input/uv.lock uv.lock
+elif [ -f /bucket/environment/uv.lock ]; then
+    cp /bucket/environment/uv.lock uv.lock
+else
+    uv lock
+    mkdir -p /bucket/environment
+    cp uv.lock /bucket/environment/uv.lock
+    cp pyproject.toml /bucket/environment/pyproject.toml
+fi
+uv sync --locked
+uv run --locked ruff check runner.py
+uv run --locked python runner.py "$@"

experiments/kraken-olmocr-bench/launch.sh

@@ -0,0 +1,52 @@
+#!/bin/sh
+# Submit the reproducible experiment. Inference and all runtime checks run on Jobs.
+set -eu
+if [ "$#" -ne 3 ]; then
+    echo "Usage: sh launch.sh smoke|full|resume-0|resume-1|resume-2|resume-3|score-smoke|score-full NAMESPACE/BUCKET RUN_ID" >&2
+    exit 2
+fi
+phase=$1
+bucket=$2
+run_id=$3
+source_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+image='ghcr.io/astral-sh/uv@sha256:85d4cb1afa769a7338e095b927bee941cf5ec92266c7424b3f6c0f2748567248'
+
+submit() {
+    flavor=$1
+    timeout=$2
+    name=$3
+    shift 3
+    hf jobs run --detach --name "$name" --label experiment=kraken-olmbench \
+        --label "run=$run_id" --label "phase=$phase" \
+        --flavor "$flavor" --timeout "$timeout" --secrets HF_TOKEN \
+        --env PYTHONUNBUFFERED=1 --env OMP_NUM_THREADS=4 --env OPENBLAS_NUM_THREADS=4 \
+        --volume "$source_dir:/input:ro" \
+        --volume "hf://buckets/$bucket:/bucket:rw" "$image" "$@"
+}
+
+case "$phase" in
+    smoke)
+        submit a10g-small 50m "kraken-$run_id-smoke" \
+            sh /input/bootstrap.sh --mode smoke --run-id "$run_id"
+        ;;
+    full)
+        for shard in 0 1 2 3; do
+            submit a10g-small 3h "kraken-$run_id-shard-$shard" \
+                sh /input/bootstrap.sh --mode full --run-id "$run_id" --shard "$shard"
+        done
+        ;;
+    resume-0|resume-1|resume-2|resume-3)
+        shard=${phase#resume-}
+        submit a10g-small 45m "kraken-$run_id-resume-$shard" \
+            sh /input/bootstrap.sh --mode full --run-id "$run_id" --shard "$shard"
+        ;;
+    score-smoke|score-full)
+        mode=${phase#score-}
+        submit cpu-upgrade 1h "kraken-$run_id-score" \
+            sh /input/score-bootstrap.sh --mode "$mode" --run-id "$run_id"
+        ;;
+    *)
+        echo "Unknown phase: $phase" >&2
+        exit 2
+        ;;
+esac

experiments/kraken-olmocr-bench/pyproject.toml

@@ -0,0 +1,13 @@
+[project]
+name = "kraken-olmocr-bench"
+version = "0.1.0"
+requires-python = ">=3.12,<3.13"
+dependencies = [
+  "kraken @ git+https://github.com/mittagessen/kraken@3e6893a0d5eb06273494bfcdb9d3cac6159e0611",
+  "huggingface-hub>=0.34,<1.0",
+  "pypdfium2>=4.30",
+  "ruff==0.15.6",
+]
+
+[tool.ruff]
+line-length = 100