E55: a step ledger on both sides of the boundary catches and locates every fault in what crosses, with no reference run, across machines too

experiments/e55-step-ledger/FINDINGS.md

@@ -0,0 +1,92 @@
+# E55: a step ledger on both sides of the boundary catches and locates every fault in what crosses, with no reference run, across machines too
+
+2026-10-02 · guangzhao (E54's `bridges-venv`) and a Windows 11 laptop
+(Python 3.11.14, numpy 2.3.2, gymnasium 1.3.0) over Tailscale · protocol:
+[`PROTOCOL.md`](PROTOCOL.md), after the pilots in `results/pilot/`, before
+the registered runs · no amendment
+
+---
+
+## The result
+
+E54's faults on Pendulum, at doses one, 0.01 and 1.0. Each environment was
+wrapped in `LedgerEnv`, which records what it took and gave. The trainer
+recorded what it sent and received. `ledger.py` compared the two, step by
+step, with no fault-free run.
+
+| | runs | client faults caught and located | false alarms | ledger or record |
+| --- | --- | --- | --- | --- |
+| one machine, seeds 0-2 | 216 | **189 of 189** | 0 of 27 | 207 of 207 |
+| across machines, seed 0 | 24 | **21 of 21** | 0 of 3 | 23 of 23 |
+
+Every check and prediction holds (`results/verdicts.txt`):
+- **V1:** all 240 runs completed; every client exited 0.
+- **V2:** `none` ended on E50's weights at seeds 0-2. The wrapper changes
+  nothing that crosses.
+- **P1:** on one machine, all 189 client-fault runs with a changed value
+  were caught. Each first difference was the fault's first changed cell,
+  in the field the fault touches.
+- **P2:** no difference in any of the 30 control and log-fault runs, on one
+  machine or across two.
+- **P3:** across machines, all 21 client faults at one value were caught
+  and located; `none` showed no difference.
+- **P4:** across machines, `none` ended on `d7a19cc0`, not guangzhao's
+  `ec5b8226`.
+- **P5:** the ledger or E54's record caught every run with a changed value:
+  207 of 207, and 23 of 23.
+
+---
+
+## What it means
+
+**Compare the two sides, and nothing else is needed.** A boundary fault is a
+difference between what the environment took and gave and what the
+trainer sent and received. Recording both and comparing them finds the
+fault where it is: the step, the env and the field. The ledger counted
+exactly the values a fault changed: 1,068 for `obs:f16` at 1%, the same
+count the client kept. It does not need a fault-free run, a seed that
+reproduces, or the two sides on one machine.
+
+**Across machines it is the only check of the four that sees every fault
+in what crosses.**
+- Byte identity cannot be used there. A fault-free run across machines
+  ended on other weights than the same seed on guangzhao, as E43 and E50
+  found: it would have called that run faulty.
+- The status rule caught none of the 24 runs.
+- E54's record caught 8, the faults that change the books.
+- The ledger caught all 21 client faults and raised no alarm on the three
+  others.
+
+**What it leaves to the record.** A fault in the trainer's own log changes
+nothing that crosses, so the ledger cannot see it. E54's record, kept at the
+environment, catches those. The two together caught every fault, on one
+machine and across two.
+
+**What it costs.** The ledger is a wrapper around each environment, plus a
+trace on the trainer's side. Neither changed a byte of what crossed (V2).
+A run across machines took 42-55 s, against 20 s on one machine. That is
+the link, not the ledger: E43 measured the crossing's cost.
+
+---
+
+## What E55 does not show
+
+* **A ledger checked while the run goes on.** The two sides were compared
+  after each run. A digest of each step carried in the protocol would let
+  the server check as it goes; that is a protocol feature still to design.
+* **Faults inside the environment, or before the wrapper.** The ledger sits
+  at the environment's own interface. A fault there, or in an environment
+  reached only through its own client, is beyond it.
+* **Any env but Pendulum, or any trainer but SB3.** E54 covers HalfCheetah
+  for the other checks.
+* **Faults it was not built to see.** Every field it compares is one E54
+  injected a fault into; a field that crosses and is not compared
+  (`info`) is not checked.
+
+Kept here, in `results/` and `results/cross/`: `ledger.csv` (`ledger.py`),
+`runs.csv` and `summary.txt` (`summarise.py`), `runs.jsonl` (`merge_runs.py`,
+every run's `result.json` and `fault.json`), and `verdicts.txt`
+(`verdicts.py`). The per-run directories are kept on guangzhao and the
+laptop and not committed: traces and ledgers, the episode logs and the
+final policies. A run on one machine is deterministic, so rerunning its line
+of `jobs.txt` there reproduces them.

experiments/e55-step-ledger/PROTOCOL.md

@@ -0,0 +1,144 @@
+# E55 measurement protocol (pre-registered)
+
+**Written 2026-10-02, after the pilots in `results/pilot/`, before the
+registered runs.**
+
+This file must not be edited after the first registered data point. Anything
+learned afterwards goes in `AMENDMENT.md`, dated.
+
+---
+
+## The question
+
+E54 found two checks that between them catch every fault it injected:
+- **byte identity** needs a fault-free run at the same seed on the same
+  machine, so it cannot judge a run across machines;
+- **reconciliation** with an env-side record of episodes cannot see faults
+  in observations or actions, which leave the episodes as the environment
+  produced them.
+
+A boundary fault is a difference between what the environment took and
+gave and what the trainer sent and received. So:
+
+**Does a step ledger kept on both sides of the boundary, compared step by
+step, catch every fault in what crosses, and locate it? Does it do so with
+no reference run, and with the two sides on different machines?**
+
+---
+
+## Declared in advance: what was already known
+
+1. **The code**, E54's with two additions.
+   - `faulty_client.py`: E54's client. Each environment is wrapped in
+     `LedgerEnv`, a gymnasium wrapper outside the client's own code. It
+     records, for each step:
+     - the observation the environment had given before it;
+     - the action it was handed;
+     - the reward, flags and observation it returned.
+     `--ledger` saves the record.
+   - `train.py`: E54's, plus two things:
+     - it passes `--ledger` to the client;
+     - `--external-client`: the bridge listens on every address and starts
+       no client, for runs whose client is on another machine (`cross.sh`).
+     Its `--trace` keeps the trainer's side, as in E54.
+   - `ledger.py` compares the two sides, env by env, step by step:
+     observation, action, reward, episode end, time limit, final
+     observation, and the number of steps. It needs no fault-free run.
+   - `jobs.py`, `run.sh` (same machine, ports 8870-8879), `cross.sh`
+     (across machines, port 8880), and E54's `summarise.py`.
+2. **The faults and doses** are E54's (its `PROTOCOL.md`), at doses one,
+   0.01 and 1.0.
+3. **The machines.**
+   - guangzhao: E54's `bridges-venv`.
+   - the laptop: a Windows 11 laptop, Python 3.11 with gymnasium 1.3.0 and
+     numpy 2.3.2. It reaches guangzhao over Tailscale at about 5 ms.
+4. **The pilots**, at seed 9, in `results/pilot/`.
+   - **Same machine** (`same-ledger.csv`, `same-summary.txt`): 72 runs.
+     - `none` ended on E50's pilot weights (`3721266c`), so the wrapper
+       changes nothing that crosses.
+     - The ledger caught all 63 client-fault runs with a changed value, at
+       their first changed cell and field.
+     - It found no difference in the 3 controls and the 6 log-fault runs:
+       72 of 72 right.
+     - One defect, fixed before this protocol: `ledger.py` crashed on a run
+       whose envs never stepped (`step:fill` at every value).
+   - **Across machines** (`cross-ledger.csv`): 7 runs, at one cell where
+     there was a fault:
+     - `none`, `obs:stale`, `action:zero`, `reward:stale`,
+       `truncated:as-term`, `step:fill` and `log:short`;
+     - 7 of 7 right.
+     - The fault-free run's weights were `be73067e`, not guangzhao's
+       `3721266c`. Across machines, byte identity would have called a
+       fault-free run faulty; the ledger found no difference.
+
+---
+
+## Design
+
+- **Same machine** (`bash run.sh results registered 10` on guangzhao):
+  Pendulum, seeds 0-2. The 3 controls, plus the 21 client faults and 2 log
+  faults at doses one, 0.01 and 1.0. Every run is traced and ledgered:
+  3 x (3 + 69) = 216 runs.
+- **Across machines** (`bash cross.sh results/cross 0` on the laptop):
+  Pendulum, seed 0. SB3 and the bridge run on guangzhao, the env client on
+  the laptop. `none`, then the 21 client faults and 2 log faults at one
+  cell: 24 runs, one at a time.
+
+Then `python ledger.py results` and `python ledger.py results/cross`, and
+`python summarise.py` on each, for E54's checks.
+
+---
+
+## How the ledger is judged
+
+A run is **caught** when anything differs between the two sides.
+
+Its first difference is the lowest step that differs, then the lowest env,
+then the first field in the order above. It is **right** when:
+- for a client fault that changed a value, that step and env are the
+  fault's first changed cell (`fault.json`), and the field is the one the
+  fault touches (`ledger.py` maps each);
+- for a control, a log fault, or a run whose fault changed no value,
+  nothing differs.
+
+---
+
+## Checks that the runs are valid
+
+- **V1:** every run reaches its 102,400 steps, and its client exits 0.
+- **V2:** on guangzhao, `none` ends on E50's weights at seeds 0, 1 and 2
+  (`ec5b8226`, `062e92e3`, `3a8d968f`): the ledger's wrapper changes nothing
+  that crosses.
+
+---
+
+## Predictions, and what falsifies each
+
+**P1 - On one machine, the ledger catches and locates every fault in what
+crosses.** Every client-fault run with a changed value is caught, at every
+dose, and its first difference is right. Falsified by one run that is not
+caught, or not right.
+
+**P2 - It raises no false alarm.** No control and no log-fault run shows a
+difference, on one machine or across machines. Falsified by one that does.
+
+**P3 - Across machines too.** With the env client on the laptop, all 21
+client faults at one cell are caught, and each first difference is right.
+`none` shows no difference. Falsified by one run against either.
+
+**P4 - Byte identity cannot judge across machines.** The fault-free run
+across machines ends on weights other than guangzhao's fault-free run at
+seed 0 (`ec5b8226`). Falsified if they are the same.
+
+**P5 - The ledger and the record catch every silent fault, on any
+machine.** Every client- and log-fault run with a changed value is caught
+by the ledger or by E54's record, on one machine and across machines.
+Falsified by one run that neither catches.
+
+> Grounds: the pilots, above. P1-P3 follow from what a boundary fault is,
+> if the ledger sits outside the client's code: the pilots test that it
+> does. P4 is E43's and E50's finding that a seed belongs to a machine.
+
+**Reported, not predicted:**
+- the time each run took across machines;
+- E54's checks on every run: rule, weights and record.

experiments/e55-step-ledger/cross.sh

@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+# E55's runs across machines, from the laptop: SB3 and the bridge on guangzhao
+# (train.py --external-client), the env client here, over Tailscale. One run at
+# a time: none, then E54's 21 client faults and 2 log faults at one cell.
+#   bash cross.sh results/cross [SEED]       # SEED defaults to 0
+#   FAULTS="none obs:stale" bash cross.sh pilot/cross 9   # only these faults
+# Each run's files from both machines end up in DIR/pendulum/<fault>/<dose>/seed<s>/.
+set -uo pipefail
+unset HTTPS_PROXY HTTP_PROXY ALL_PROXY
+H=guangzhao-memeseeks
+R=zuogou/plugrl/exp/e55-step-ledger
+GZ=100.75.226.89
+PORT=8880
+PY=/c/Users/75128/.claude/jobs/1fea619d/tmp/plugrl-rlinf/.venv/Scripts/python.exe
+export OMP_NUM_THREADS=1
+cd "$(dirname "$0")"
+DIR="${1:?results dir}"; SEED="${2:-0}"
+CLIENT_FAULTS=$("$PY" -c "from faulty_client import SILENT; print(' '.join(SILENT))")
+LOG_FAULTS="log:drop-first log:short"
+mkdir -p "$DIR"
+"$PY" --version > "$DIR/laptop-python.txt" 2>&1
+"$PY" -c "import numpy, gymnasium; print('numpy', numpy.__version__, 'gymnasium', gymnasium.__version__)" >> "$DIR/laptop-python.txt"
+
+for fault in ${FAULTS:-none $CLIENT_FAULTS $LOG_FAULTS}; do
+  dose=one; [ "$fault" = none ] && dose=1.0
+  out="$DIR/pendulum/${fault//:/__}/$dose/seed$SEED"
+  mkdir -p "$out"
+  client_fault=$fault; case $fault in log:*) client_fault=none ;; esac
+  echo "== $fault $dose seed $SEED $(date +%T)"
+  ssh -n -o BatchMode=yes "$H" "cd ~/$R && mkdir -p $out && (OMP_NUM_THREADS=1 nohup ~/zuogou/plugrl/bridges-venv/bin/python train.py --env pendulum --fault $fault --dose $dose --seed $SEED --out $out --port $PORT --trace --external-client > $out/train.log 2>&1 &)"
+  "$PY" faulty_client.py --host "$GZ" --port "$PORT" --num-envs 16 --seed "$SEED" \
+    --env-id Pendulum-v1 --action-high 2.0 --fault "$client_fault" --dose "$dose" \
+    --record "$out/env_record.csv" --ledger "$out/ledger.npz" > "$out/client.log" 2>&1
+  echo "client exit $?" >> "$out/client.log"
+  tail -n 2 "$out/client.log"
+  waited=0
+  until ssh -n -o BatchMode=yes "$H" "test -f ~/$R/$out/result.json"; do
+    sleep 2; waited=$((waited + 2))
+    if [ "$waited" -ge 300 ]; then echo "no result.json after 300 s: $out"; break; fi
+  done
+  ssh -n -o BatchMode=yes "$H" "cd ~/$R/$out && tar -cf - result.json trace.npz monitor.csv episodes.csv train.log" | tar --force-local -xf - -C "$out"
+done
+echo E55_CROSS_DONE