E57: on a clean environment, faults the curve checks miss leave policies as good as fault-free ones; every wrecked policy was already caught

experiments/e57-clean-eval/AMENDMENT.md

@@ -0,0 +1,18 @@
+# E57 amendment
+
+## 2026-10-03 00:25 EDT: summarise.py read a blank cell as a number
+
+All 834 registered evaluations had finished and none had failed, before
+anything was summarised. Then `summarise.py` stopped with
+`ValueError: invalid literal for int() with base 10: ''`.
+
+The cause is the six runs of E54's two loud faults, `indices:reorder` and
+`reward:short-array`. They raised an error, logged no return, and so have a
+blank `strict` in E54's `runs.csv`. `summarise.py` converted `rule` and
+`strict` to integers without allowing for a blank.
+
+The fix is in one line: a blank cell becomes `None`. A run with `None` is
+not "silent on the curve". The protocol leaves the loud faults out of every
+count, so the fix changes no count, no rule and no prediction. Nothing else
+in `summarise.py`, `verdicts.py` or `evaluate.py` changed. No result had been
+seen when the fix was made: the crash came before any output.

experiments/e57-clean-eval/FINDINGS.md

@@ -0,0 +1,110 @@
+# E57: on a clean environment, the policies behind the faults that both curve checks miss are as good as fault-free ones; every policy the faults wrecked, the curve checks had already caught
+
+2026-10-03 · guangzhao (E54's `bridges-venv`) · protocol:
+[`PROTOCOL.md`](PROTOCOL.md), after the pilot in `results/pilot/`, before
+the registered runs · amended once, see [`AMENDMENT.md`](AMENDMENT.md): a
+parsing fix in `summarise.py`, made before any output, that changes no count
+
+---
+
+## The result
+
+Each of the 834 final policies of E54 and E56 was run on a fresh gymnasium
+environment for 50 deterministic episodes, all policies on the same seeds.
+"Harmed" means its mean return, the clean return, fell more than 3 standard
+deviations below the fault-free band (`none`, seeds 10-19).
+
+Boundary-fault runs, seeds 0-2:
+
+| env | dose | harmed | silent on the curve | of them harmed |
+| --- | --- | --- | --- | --- |
+| Pendulum | one | 1/63 | 63 | 1 |
+| | 0.001 | 2/63 | 58 | 1 |
+| | 0.01 | 1/63 | 59 | 1 |
+| | 0.1 | 6/63 | 50 | 2 |
+| | 1.0 | 24/63 | 25 | 0 |
+| HalfCheetah | one | 0/63 | 60 | 0 |
+| | 0.01 | 0/63 | 57 | 0 |
+| | 1.0 | 28/63 | 31 | 0 |
+| Hopper | one | 0/69 | 69 | 0 |
+| | 0.01 | 0/69 | 66 | 0 |
+| | 1.0 | 19/69 | 35 | 0 |
+
+"Silent on the curve" means the run passed both the gain threshold and
+seed spread.
+
+Every check and prediction holds (`results/verdicts.txt`):
+- **V1:** 834 of 834 evaluations, each of 50 episodes.
+- **V2:** the 12 weights shared by two or more runs got identical clean
+  returns, so evaluation is deterministic.
+- **P1:** the harm rule flagged none of the ten fault-free band seeds in any
+  environment.
+- **P2:** at one value, 1 of 63, 0 of 63 and 0 of 69 boundary-fault runs
+  were harmed.
+- **P3:** Pendulum's `reward:zero` at every value logged 0, the best return
+  Pendulum gives, at all three seeds. Its policies scored -1196, -1550 and
+  -1639 on the clean environment, against a band of -186 ± 26.
+
+---
+
+## What it means
+
+**Every policy the faults wrecked, the curve checks had already caught.** At
+every value, the faults harmed 24, 28 and 19 runs. None of them passed
+both curve checks. At HalfCheetah and Hopper, no run that passed both checks
+was harmed, at any dose.
+
+**The faults the curve checks miss leave policies as good as fault-free
+ones.** This is an exploratory analysis, not in the protocol (`explore.py`,
+`results/explore.txt`). For each environment and dose, the runs silent on
+the curve have a mean clean return 0.03-0.65 band s.d. *above* the band.
+None is significantly below the 13 fault-free runs (two-sided Mann-Whitney
+U). The one significant difference is upward: Pendulum at every value,
+p = 0.003. At their own seed, they beat the fault-free twin by a median of 7-9 on
+Pendulum, 63-101 on HalfCheetah and 142-368 on Hopper. That is consistent with a
+change of seed, not with harm.
+
+**The exception is five Pendulum runs.** They passed both curve checks but
+scored below every one of the 13 fault-free runs, whose worst was -238:
+- `reward:f16` at 0.1%: -744;
+- `obs:swap` at 1%: -506;
+- `obs:stale` at one value: -430;
+- `final-obs:f16` at 10%: -333;
+- `obs:stale` at 10%: -277.
+
+That is 5 of 255 silent runs on Pendulum, against 0 of 13 fault-free.
+Thirteen fault-free runs cannot tell a rare harm from a rare bad seed.
+
+**So a silent fault is silent on a clean evaluation too.** In these tasks,
+with SB3's PPO, a fault that leaves the curve looking right also leaves
+the policy looking right. What it changes is the run. The weights differ
+from the run the authors think they trained, and with log faults or reward
+faults, so do the numbers they report. Pendulum's `reward:zero` is the
+extreme case: its log read 0, the best possible, while its policy scored
+-1196 to -1639. Only a check of the boundary itself, the step ledger or the
+episode record, tells such a run from a correct one.
+
+**On trusting the trainer's log.** For each run, the logged final return
+minus the clean return is in `results/summary.txt`. The fault-free runs'
+own gap is wide: Hopper's ranges from -1667 to +806. Training returns come
+from stochastic actions and the clean return from deterministic ones. A
+gap this wide hides all but gross reward faults.
+
+---
+
+## What E57 does not show
+
+* **Any trainer but SB3's PPO, or any task but these three.** On a task
+  where small changes in the data matter more, such as sparse rewards,
+  long horizons or real robots, the same faults may harm the policy.
+* **Harm too rare for 13 fault-free runs to bound.** The five Pendulum
+  outliers above are the edge of what this design can see.
+* **Harm in a comparison between methods.** E57 compares policies with the
+  fault-free band, not one method with another.
+
+Kept here, in `results/`: `clean.csv` (`summarise.py`, one row per run, with
+its clean return, harm, curve checks and logged return), `summary.txt`,
+`verdicts.txt` (`verdicts.py`), `explore.txt` (`explore.py`), `jobs.txt`,
+`load.txt`, `registered.out` and the source hashes. Each run's
+`clean_eval.json`, with all 50 episodes, is kept on guangzhao. Evaluation
+is deterministic (V2), so `run.sh` reproduces it.

experiments/e57-clean-eval/PROTOCOL.md

@@ -0,0 +1,118 @@
+# E57 measurement protocol (pre-registered)
+
+**Written 2026-10-03, after the pilot in `results/pilot/`, before the
+registered runs.**
+
+This file must not be edited after the first registered data point. Anything
+learned afterwards goes in `AMENDMENT.md`, dated.
+
+---
+
+## The question
+
+E54 and E56 judged each run by what the trainer saw: its own log, its
+weights, or the two sides' records. None of them asks what a faulted run
+leaves behind, the policy. Reviewers asked whether the faults matter at all.
+
+**Evaluated on a clean environment, is a policy trained through a fault
+worse than one trained without it? In particular, are the faults that both
+curve checks miss?** And does the trainer's log agree with the clean
+evaluation?
+
+---
+
+## Declared in advance: what was already known
+
+1. **The policies.** Every E54 and E56 run saved its final policy
+   (`model.zip`): 590 runs of E54 (Pendulum, HalfCheetah) and 244 of E56
+   (Hopper). Their outcomes under the gain threshold, seed spread, byte
+   identity, record and ledger are published (`runs.csv`, `ledger.csv`). No
+   policy had been evaluated outside training before the pilot.
+2. **The pilot** (`results/pilot/`), 20 episodes each, on guangzhao:
+
+   | env | `none` seed 0 | `none` seed 10 | `obs:f16` 1.0 seed 0 |
+   | --- | --- | --- | --- |
+   | Pendulum | -147.0 | -139.1 | -139.7 |
+   | HalfCheetah | 647.1 | 779.8 | 851.2 |
+   | Hopper | 3368.9 | 3651.3 | 1633.5 (mean length 482) |
+
+   Pendulum's episodes vary by about 100 (initial states); HalfCheetah's by
+   about 130; Hopper's by 10-580. An evaluation took 0.3-2.5 s.
+
+---
+
+## Design
+
+`evaluate.py RUN_DIR OUT_DIR --episodes 50`. It does the following:
+- loads the run's `model.zip` on CPU, with one torch thread;
+- makes a fresh `gymnasium.make(id)` at default settings (Pendulum-v1,
+  HalfCheetah-v5, Hopper-v5), with no bridge, client, wrapper or fault;
+- plays 50 episodes with `deterministic=True`, reset with seeds 10000-10049,
+  which are the same for every policy;
+- writes `clean_eval.json`: each episode's return and length, and their mean,
+  the **clean return**.
+
+Every run directory of E54 and E56 that holds a `model.zip` is evaluated:
+834 runs, 12 at a time on guangzhao (`run.sh`). Then `python summarise.py`
+and `python verdicts.py`.
+
+---
+
+## How each run is judged
+
+- **The fault-free band**, per environment: the clean returns of `none` at
+  seeds 10-19. The band is its mean and standard deviation.
+- **Harmed:** a run's clean return is more than 3 standard deviations below
+  the band's mean. The rule is the seed-spread check's, applied to the clean
+  return, and only on the low side. A fault-free seed among 10-19 is judged
+  against the other nine.
+- **Silent on the curve:** the run passed both the gain threshold and seed
+  spread (`rule` = 0 and `strict` = 0 in E54's or E56's `runs.csv`).
+- **The log's error:** the trainer's logged final return (`last` in
+  `runs.csv`) minus the clean return.
+
+Boundary faults are E54's 21 client faults (23 on Hopper). Log faults,
+controls (`none`, `delay-1ms`, `wire-float64`) and the two loud faults
+(`indices:reorder`, `reward:short-array`) are reported separately. The
+loud faults raised an error, so they are left out of every count.
+
+---
+
+## Checks that the runs are valid
+
+- **V1:** all 834 evaluations complete, each with 50 episodes.
+- **V2:** two runs with the same weights hash get the same clean return. The
+  controls and log faults ended on `none`'s weights, so this tests that
+  evaluation is deterministic.
+
+---
+
+## Predictions, and what falsifies each
+
+**P1 - The harm rule raises few false alarms.** It flags at most 1 of the
+ten fault-free seeds 10-19 in each environment, each judged against the
+other nine. Falsified by 2 or more in any environment.
+
+**P2 - One changed value does no measurable harm.** At dose `one`, at most
+10% of boundary-fault runs are harmed, in each environment. Falsified by
+more.
+
+**P3 - A zeroed reward leaves a policy that does not work, behind a log
+that looks perfect.** On Pendulum at dose 1.0, `reward:zero` is harmed at
+all three seeds, while its log reads 0, the best return Pendulum can give.
+Falsified by any seed not harmed.
+
+> Grounds: P1, E54's and E56's seed-spread false alarms (0, 1, 0 of ten). P2,
+> one changed value moved the logged return about as much as a change of
+> seed (E54, E56). P3, with every reward zero PPO's advantages are zero
+> after normalisation, and nothing is learned.
+
+**Reported, not predicted:**
+- **The main count.** Among boundary-fault runs, at each dose:
+  - how many are harmed;
+  - how many are harmed while silent on the curve.
+- **The faults behind it.** Every fault and dose that was silent on the curve
+  at all three seeds and harmed at two or more.
+- **The log's error.** For each kind, against the fault-free runs' range.
+- **The pilot's lead.** Whether Hopper's `obs:f16` at 1.0 is harmed at seeds 1
+  and 2. The pilot saw it at seed 0 only.

experiments/e57-clean-eval/evaluate.py

@@ -0,0 +1,64 @@
+"""E57: evaluate one run's final policy on a clean environment.
+
+    python evaluate.py RUN_DIR OUT_DIR [--episodes N]
+
+RUN_DIR is an E54 or E56 run directory (model.zip, result.json). The policy
+acts deterministically in a fresh gymnasium environment of the same id, at
+its default settings, for N episodes reset with seeds 10000, 10001, ... No
+bridge, no env client, no fault. Writes OUT_DIR/clean_eval.json.
+"""
+
+import argparse
+import json
+import pathlib
+import time
+
+import gymnasium as gym
+import numpy as np
+import torch
+from stable_baselines3 import PPO
+
+IDS = {"pendulum": "Pendulum-v1", "halfcheetah": "HalfCheetah-v5", "hopper": "Hopper-v5"}
+SEED0 = 10_000
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("run", type=pathlib.Path)
+    ap.add_argument("out", type=pathlib.Path)
+    ap.add_argument("--episodes", type=int, default=20)
+    a = ap.parse_args()
+    torch.set_num_threads(1)
+
+    res = json.loads((a.run / "result.json").read_text())
+    env = gym.make(IDS[res["env"]])
+    model = PPO.load(a.run / "model.zip", device="cpu")
+
+    returns, lengths = [], []
+    t0 = time.time()
+    for k in range(a.episodes):
+        obs, _ = env.reset(seed=SEED0 + k)
+        done, ret, n = False, 0.0, 0
+        while not done:
+            act, _ = model.predict(obs, deterministic=True)
+            obs, r, term, trunc, _ = env.step(act)
+            ret += float(r)
+            n += 1
+            done = term or trunc
+        returns.append(ret)
+        lengths.append(n)
+
+    out = dict(
+        env=res["env"], fault=res["fault"], dose=res["dose"], seed=res["seed"],
+        weights_sha256=res["weights_sha256"], episodes=a.episodes,
+        returns=returns, lengths=lengths, mean=float(np.mean(returns)),
+        eval_s=round(time.time() - t0, 3),
+    )
+    a.out.mkdir(parents=True, exist_ok=True)
+    (a.out / "clean_eval.json").write_text(json.dumps(out))
+    print(json.dumps({k: out[k] for k in ("env", "fault", "dose", "seed", "mean", "eval_s")}))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

experiments/e57-clean-eval/explore.py

@@ -0,0 +1,51 @@
+"""E57, exploratory (not in PROTOCOL.md): are the runs both curve checks miss
+shifted as a group, though no single one is harmed?
+
+    python explore.py [results]
+
+For each env and dose, the clean returns of boundary-fault runs that are
+silent on the curve, in units of the fault-free band (none, seeds 10-19):
+their mean z, and a two-sided Mann-Whitney U test against the 13 fault-free
+runs (seeds 0-2 and 10-19). Also the clean-return gap between each such run
+and the fault-free run at its own seed. Writes results/explore.txt.
+"""
+
+import csv
+import pathlib
+import statistics as st
+import sys
+
+from scipy.stats import mannwhitneyu
+
+HERE = pathlib.Path(__file__).resolve().parent
+
+
+def main() -> int:
+    root = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else HERE / "results"
+    rows = list(csv.DictReader(open(root / "clean.csv")))
+    for r in rows:
+        r["seed"], r["clean"] = int(r["seed"]), float(r["clean"])
+    out = ["exploratory: boundary-fault runs silent on the curve, against fault-free runs",
+           "env          dose    n   mean z   median z   U-test p   twin gap median (clean - none at same seed)"]
+    for env in sorted({r["env"] for r in rows}):
+        band = [r["clean"] for r in rows if r["env"] == env and r["fault"] == "none" and r["seed"] >= 10]
+        mu, sd = st.mean(band), st.stdev(band)
+        free = [r["clean"] for r in rows if r["env"] == env and r["fault"] == "none"]
+        twin = {r["seed"]: r["clean"] for r in rows if r["env"] == env and r["fault"] == "none"}
+        for dose in ("one", "0.001", "0.01", "0.1", "1.0"):
+            s = [r for r in rows if r["env"] == env and r["dose"] == dose and r["kind"] == "boundary"
+                 and r["seed"] < 10 and r["silent_curve"] == "1"]
+            if not s:
+                continue
+            z = [(r["clean"] - mu) / sd for r in s]
+            p = mannwhitneyu([r["clean"] for r in s], free, alternative="two-sided").pvalue
+            gap = st.median(r["clean"] - twin[r["seed"]] for r in s)
+            out.append(f"{env:12} {dose:6} {len(s):3}   {st.mean(z):6.2f}   {st.median(z):8.2f}   {p:8.3f}   {gap:9.1f}")
+    text = "\n".join(out)
+    (root / "explore.txt").write_text(text + "\n")
+    print(text)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())