Document benchmark metrics and publish checkpoint PSR reconstruction

README.md

@@ -42,6 +42,7 @@ versioned tasks, and evaluation utilities so you can follow an experiment from
 | **Benchmark a model** | Use an OpenAI-compatible endpoint with the [reference runner](docs/benchmark/quickstart.md). |
 | **Benchmark my agent harness** | Bring your planner, memory, or framework through a [custom adapter](docs/benchmark/custom-agents.md). |
 | **Understand a run** | Inspect actions and observations with [trajectory visualizations](docs/benchmark/visualization.md). |
+| **Understand the metrics** | Read the [SR, PSR, and CCE definitions and scoring examples](docs/benchmark/metrics.md). |
 | **Contribute or review results** | Read the [contributor guide](CONTRIBUTING.md) and [submission requirements](docs/benchmark/scoring.md). |
 
 ## Inside the benchmark
@@ -152,7 +153,7 @@ harness through the game API.
 | :--- | :--- |
 | **Success rate · SR** | Offline, task-specific verification with coverage, errors, and per-task outcomes. Full-suite SR is reported only when coverage is complete and there are no scoring errors. |
 | **Causal Collaboration Effectiveness · CCE** | Research analysis using an LLM judge. Record the judge, configuration, and protocol alongside results. |
-| **Partial success rate · PSR** | A validated implementation reproducing the paper's metric is not yet available in this checkout. |
+| **Partial success rate · PSR** | [Checkpoint reconstruction and worked example](docs/benchmark/metrics.md#partial-success-rate-psr); not cross-validated against the historical implementation. |
 
 The checkout contains 200 augmented task files; the paper reports experiments on
 100 augmented variants. Record the exact task manifest when comparing results.

docs/benchmark/metrics.md

@@ -0,0 +1,200 @@
+# Evaluation metrics
+
+This document specifies the checkpoint PSR reconstruction used for the September
+2026 additional-model experiments, alongside SR, CCE, Rounds, and Chats.
+The implementation is [scripts/score_psr.py](../../scripts/score_psr.py), with
+offline [regression tests](../../tests/benchmark/test_psr.py).
+
+**Reproducibility boundary:** this is method `checkpoint_fraction_v1`. It
+reconstructs the paper's verbal checkpoint definition, but has not been
+cross-validated against the unavailable historical PSR program. Label its output
+**reconstructed PSR**, rather than claiming it reproduces every published PSR
+value. It uses fractional credit for numeric quantities; it is not the proportion
+of checkpoints whose targets have been fully met.
+
+## Partial success rate (PSR)
+
+For task *i* with *Kᵢ* verifier checkpoints, assign each checkpoint a credit:
+
+| Checkpoint | Credit | Example |
+| --- | --- | --- |
+| Quantity with target *r* and observed amount *a* | `min(a / r, 1)` | `Logs: 3/5` → 0.6 |
+| Boolean condition | 1 for `True`, 0 for `False` | `All alive: True` → 1 |
+| Upper bound on rounds | 1 if `actual <= limit`, otherwise 0 | `Rounds: 30/25` → 0 |
+
+The per-task score is:
+
+```text
+PSR_i = 1                                      if the task verifier reports success
+        (credit_i1 + ... + credit_iK) / K_i     otherwise
+
+PSR_percent = 100 × (PSR_1 + ... + PSR_N) / N
+```
+
+Each checkpoint has equal weight within its task, and each task has equal weight
+within a reported split. We do not pool all checkpoints across tasks: a task with
+more checkpoints does not receive more weight. A successful task receives 1 even
+if its diagnostic message omits checkpoints or describes intermediate states.
+
+For example, for an unsuccessful task with this verifier message:
+
+```text
+Logs: 3/5, Kills: 0/3, All alive: True
+```
+
+the checkpoint credits are 0.6, 0, and 1. Its PSR is `(0.6 + 0 + 1) / 3`, or
+**53.33%**, while its binary success score is 0. If another task succeeds, the
+two-task mean PSR is **76.67%** and SR is **50%**.
+
+The script reads comma-separated `label: value` messages from the verifier.
+Numeric targets normally appear as `actual/target`. For legacy messages that
+omit a denominator, the explicit `BARE_TARGETS` table in the script supplies
+targets taken from the corresponding verifier predicates. Unknown formats,
+unknown bare targets, and nonpositive targets raise errors; they are not replaced
+with zero. If a verifier changes its message or criteria, review this mapping
+and version the scoring protocol before comparing results.
+
+## Task success rate (SR)
+
+```text
+SR_percent = 100 × number of successful tasks / number of tasks
+```
+
+For new runs, [benchmarks/score.py](../../benchmarks/score.py) calls the checked-in
+task-specific verifiers. See the [scoring guide](scoring.md) for commands.
+
+The historical additional-model collector in `scripts/score_psr.py` deliberately
+uses [task_verifier.py](../../task_verifier.py): version 0 for Main, version 1 for
+`_v1`, and version 2 for `_v2`. This preserves that experiment's verifier choice.
+It does **not** silently switch historical results to the newer adjacent-task
+verifier path in `benchmarks/score.py`. Record the verifier and code revision;
+these paths are not certified equivalent. For augmented results, do not trust a
+saved main-task success flag without verifying the variant.
+
+## Causal Collaboration Effectiveness (CCE)
+
+[compute_cce_v2.py](../../analysis/scripts/compute_cce_v2.py) first asks an LLM
+judge which actions directly achieved the objective. It then traces backward by
+round to identify actions that enabled contributing actions in later rounds.
+The union of those actions is the contributing set *C*.
+
+```text
+CCE_i = |C_i| / |T_i|     for a successful task with recorded actions
+        0                 for a failed task or a task with no recorded actions
+
+CCE = (CCE_1 + ... + CCE_N) / N
+```
+
+`T` contains the nonempty action strings extracted from the trajectory. The
+implementation rounds each per-task CCE to four decimal places before saving
+`ce`; the experiment summary averages these saved values. The reported mean
+includes failed tasks with zero CCE. It is neither a successful-task-only mean
+nor a ratio formed by pooling action counts across tasks.
+
+CCE requires a configured judge API for successful tasks; PSR does not use an
+LLM. See the [CCE commands](scoring.md#causal-collaboration-effectiveness-cce).
+Keep the judge model/version, prompts, endpoint, and per-task output with the
+results. The additional-model experiments used Vertex Gemini 2.5 Flash; the
+paper's original table used GPT-4.1. Scores from different judges must be labeled
+and are not directly interchangeable. A judge/API failure is a pending score,
+not a model failure or a zero CCE.
+
+## Rounds and Chats
+
+These are arithmetic task means of the trajectory's `metrics.total_rounds` and
+`metrics.total_chats`, respectively, including valid failed tasks. They are not
+per-agent means. Report Main Rounds/Chats when reproducing the table layout used
+for the additional-model experiments.
+
+## Run the PSR script
+
+Python 3.10+ is sufficient; this script uses only the standard library and the
+repository verifier. Run commands from the repository root.
+
+### From checkpoint inputs
+
+The input JSON maps model names to task rows. Every row has `stage` (`main` or
+`augmented`), the full task filename stem, boolean `success`, verifier `message`,
+saved `CCE` in [0, 1], and nonnegative `rounds` and `chats`. For example:
+
+```json
+{
+  "example-model": [
+    {
+      "stage": "main",
+      "task": "task_01_magic_staff",
+      "success": false,
+      "message": "Logs: 3/5, Kills: 0/3, All alive: True",
+      "CCE": 0.0,
+      "rounds": 30,
+      "chats": 8
+    }
+  ]
+}
+```
+
+This is a synthetic parser example, not task 01's actual verifier output. Save
+it as `runs/checkpoint-inputs.json`, then run:
+
+```bash
+python scripts/score_psr.py runs/checkpoint-inputs.json runs/psr-results.json
+```
+
+The command writes the inputs back to the input path and writes the full report
+to the output path. Use separate paths and copies if you want to preserve an
+earlier input snapshot. The result includes per-checkpoint credits, per-task PSR,
+the method ID, `historical_implementation_reproduced: false`, and split summaries.
+
+### From the audited additional-model result layout
+
+The collector supports any model directory name and both original and copied
+result trees:
+
+```text
+RESULTS_ROOT/MODEL/main/TASK/completed.json
+RESULTS_ROOT/MODEL/main/TASK/cce.json
+RESULTS_ROOT/MODEL/main/TASK/ATTEMPT/task_NN_trajectory.json
+RESULTS_ROOT/MODEL/augmented/TASK_v1/...  (or TASK_v2)
+```
+
+```bash
+python scripts/score_psr.py runs/checkpoint-inputs.json runs/psr-results.json \
+  --collect-root . \
+  --results-root /path/to/additional-model-results
+```
+
+`--collect-root` selects the repository's verifier and task catalog. Without
+`--results-root`, collection retains the historical default
+`agents/logs/additional_20260925` under that repository. The collector prefers the
+sibling attempt trajectory in a copied folder over the original absolute path
+stored in a completion marker. It re-verifies outcomes, checks task identity and
+marker metrics, and requires agreement with saved CCE success. Cached rows are
+invalidated when the completion marker, CCE record, verifier, or trajectory changes.
+
+This collector requires `completed.json` and `cce.json`, which were produced by
+the additional-model experiment harness. It does not discover arbitrary raw
+runner outputs or generate CCE scores. A missing CCE file leaves that task out
+of the collected rows, so its PSR/SR summaries are also partial until scoring
+finishes. The collector is not a replacement for initialization and trace audits.
+
+### Coverage and reporting
+
+- Main has 100 tasks; Augmented v1 and v2 have 100 each; their union has 200.
+- The script checks task identities against the repository YAML catalog and
+  rejects duplicate rows, unexpected tasks, and invalid numeric metrics.
+- Every summary includes `scored`, `expected`, `missing_tasks`, and `final`.
+  When `final` is false, numeric means describe only the scored subset. Do not
+  present them as complete benchmark scores or impute missing tasks as zero.
+- Audit complete team initialization, initial observations, inference failures,
+  and one selected valid attempt per task before collection. Preserve interrupted
+  attempts; infrastructure failures remain pending rather than model failures.
+- Score trials separately and state the trial aggregation policy. Do not select
+  the best attempt or pool repeated trials as if they were distinct tasks.
+- Report Main, v1, and v2 separately. The additional-model v1 split is not
+  established as equivalent to the original paper's historical augmented subset.
+
+Run the offline regression tests with:
+
+```bash
+python -m unittest discover -s tests/benchmark -p 'test_psr.py' -v
+```

docs/benchmark/scoring.md

@@ -40,11 +40,18 @@ than assuming their saved success flags remain comparable.
 
 ## Partial success rate (PSR)
 
-A validated, portable implementation reproducing the paper's PSR is not present
-in this upstream checkout. The separately maintained local experiment copy has
-a reconstruction, but it has not been cross-validated against the historical
-implementation. Do not label a replacement calculation as the published metric.
-Report PSR as unavailable until its implementation and protocol are established.
+The [metrics guide](metrics.md#partial-success-rate-psr) specifies the checkpoint
+reconstruction used for the additional-model experiments, with a worked example
+and commands for [scripts/score_psr.py](../../scripts/score_psr.py). Quantities
+receive fractional credit capped at 1; booleans and round limits receive 0/1;
+checkpoints are averaged within each task and tasks are averaged equally.
+Successful tasks receive 1.
+
+Report this as **reconstructed PSR (`checkpoint_fraction_v1`)**. It has not been
+cross-validated against the unavailable historical implementation and does not
+certify reproduction of the paper's PSR. The additional-model collector retains
+its historical `task_verifier.py` path; it does not silently replace the newer
+task-specific verifier protocol used by `benchmarks/score.py` above.
 
 ## Causal Collaboration Effectiveness (CCE)