exp: E29 - the logprob clamp is not why fpo-policy stands still under DPPO; its narrow noise is part of it

experiments/e29-clamp-or-noise/.gitignore

@@ -0,0 +1,3 @@
+# Checkpoints and tensorboard event files; summarise.py reads out what they hold.
+results/*/dppo/
+pilot/*/dppo/

experiments/e29-clamp-or-noise/FINDINGS.md

@@ -0,0 +1,106 @@
+# E29: the log-probability clamp is not why `fpo-policy` stands still under DPPO; its narrow noise is part of it
+
+2026-09-25 · Linux workstation (`guangzhao`), CPU only · `fpo-policy` · DPPO ·
+Hopper-v5, three arms, three seeds, 100 iterations · protocol:
+[`PROTOCOL.md`](PROTOCOL.md) (`20484bf`, after the pilot in
+[`results/pilot.txt`](results/pilot.txt))
+
+---
+
+## The result
+
+`fpo-policy` learns Hopper under FPO and not under DPPO (E28), where DPPO's
+own policy improves it by +260 to +417. Reading the code found two places
+where the two policies meet DPPO's loss differently: the clamp of every
+log-probability to [-5, 2], which binds on half of `fpo-policy`'s elements
+and none of `dppo-policy`'s, and the per-step noise, about a tenth of
+`dppo-policy`'s. E29 changed each, one arm apiece, on E28's cell.
+
+| arm | change | gain, seeds 0 / 1 / 2 | verdict |
+| --- | --- | --- | --- |
+| `control` | none | +10.6 / +4.5 / +9.7 | P2 holds: E28 was +10.6 / +4.9 / +9.7 |
+| `noclamp` | upper bound lifted | **+7.0 / +3.0 / +7.8** | **P3 falsified**: 0 of 3 past +100 |
+| `wide` | noise level ×10 | **+73.6 / +84.6 / +106.2** | **P4 falsified**: 1 of 3 past +100 |
+
+(Gain: mean return of iterations 91-100 minus the first iteration's.)
+
+* **P1 holds**, 3 of 3 arms, 9 of 9 seeds; 20 minutes each.
+* **V1 passes**: the logged entropy is −1.9198 in `control` and `noclamp`
+  and +0.3828 in `wide`, at every iteration - the expected −1.920 and
+  −1.920 + ln 10 = +0.383.
+
+**The clamp is ruled out.** Lifting it changes nothing measurable: the gains
+sit inside the control's, and the actor moves as much as the control's
+(15-20% from iteration 20 to 100, against 20-23%).
+
+**The noise is part of it, and not all of it.** By the registered bar - +100
+on two seeds of three - P4 is falsified, and the protocol's reading for two
+falsifications, "neither is sufficient", is what is claimed. Described
+beside it: every `wide` seed gained seven to nineteen times its control
+seed, and every one ended at least 60 above its control seed's best
+iteration.
+Mean return per ten iterations:
+
+| arm, seed | 1-10 | 21-30 | 41-50 | 61-70 | 81-90 | 91-100 |
+| --- | --- | --- | --- | --- | --- | --- |
+| `wide` 0 | 17 | 19 | 25 | 56 | 77 | 90 |
+| `wide` 1 | 21 | 23 | 30 | 54 | 89 | 106 |
+| `wide` 2 | 19 | 22 | 28 | 52 | 97 | 126 |
+| `control` 0-2 | 16-18 | 17-19 | 19 | 19-22 | 22-24 | 24-27 |
+
+`wide` barely moves for forty iterations, then climbs, and is still climbing
+at the end; episodes grew from about 23 steps to 58-76. At an entropy about
+`dppo-policy`'s, it gains about a quarter of what `dppo-policy` gained in
+E28.
+
+---
+
+## What I expected, and what the pilot misled me about
+
+After the pilot I registered both as expected to fail. By the rule both did;
+about `wide` I was wrong in substance. In eleven iterations `wide` had made
+`approx_kl` ten times *smaller* than the control's, and I read that as the
+ratio moving less. It is not comparable across noise levels: the KL between
+two normals for the same shift of the mean scales as 1/σ², so ten times the
+deviation divides it by a hundred for the same change. The actor moved
+**40-43%** in `wide` against the control's 20-23%.
+
+---
+
+## What this does and does not support
+
+**Supported:**
+
+* The log-probability clamp does not stop `fpo-policy` learning under DPPO.
+  #64 keeps it configurable and its default the reference's; E29 gives no
+  reason to change that default.
+* Ten times the flow policy's noise, for sampling and for the ratio alike,
+  lets DPPO move `fpo-policy`'s return on Hopper by +74 to +106 in 409,600
+  steps, where the unchanged policy moves +5 to +11.
+
+**Not supported:**
+
+* That `fpo-policy` learns Hopper under DPPO at any noise level: the bar was
+  not met, and one level was tried.
+* That the noise is the whole difference from `dppo-policy`. At matched
+  entropy the gap is still about fourfold. The two policies also differ in
+  network (`fpo-policy`'s four hidden layers of 32 against DPPO's own),
+  action chunk (1 against 4), denoising steps (10 against 20) and
+  `fpo-policy`'s output scale of 0.25; none is tested here.
+* Anything about how the noise helps - through exploration, through the
+  variance of the policy gradient, or through the ratio. E29 varies the
+  level once and does not separate them.
+
+---
+
+## Reproducing
+
+```bash
+bash run.sh                                  # three arms at once; about 20 minutes on 24 cores
+python summarise.py                          # P1, V1, P2-P4, the diagnostics; writes summary.tsv
+PYTHONPATH=src python clamp_probe.py 0.1     # the clamp's reach on fresh policies, per step
+```
+
+`results/verdicts.txt` is `summarise.py`'s output. The pilot's own logs stay
+on `guangzhao`; what they showed is in `results/pilot.txt`. Checkpoints and
+tensorboard files stay there too (`.gitignore`).

experiments/e29-clamp-or-noise/PROTOCOL.md

@@ -0,0 +1,154 @@
+# E29 measurement protocol (pre-registered)
+
+**Written 2026-09-25, after the pilot in `results/pilot.txt` and before the
+registered run.**
+
+This file must not be edited after the first registered data point. Anything
+learned afterwards goes in `AMENDMENT.md`, dated.
+
+---
+
+## The question
+
+`fpo-policy` learns HalfCheetah, Hopper and Walker2d under FPO (E6, E16, E23,
+E24) and none of them under DPPO (E17, E18, E28), where DPPO's own
+`dppo-policy` improves all three (E28). By the project's rule that is a
+defect of ours. Reading the code for where the two policies meet DPPO's loss
+differently found two concrete differences:
+
+1. **The clamp.** DPPO clamps every element's log-probability to [-5, 2]
+   before forming the ratio. At the MuJoCo variants' noise level, 0.1, the
+   upper bound binds on **50.7%** of `fpo-policy`'s elements - none at its
+   first three steps, 93% at its last - and on none of `dppo-policy`'s. A
+   clamped element carries no gradient and a ratio of exactly 1. #64 makes
+   the bounds configurable.
+2. **The noise.** `dppo-policy` floors every denoising step's deviation at
+   the level; `fpo-policy`'s flow steps run from 0.1 down to about 0.01. Per
+   element, the logged entropy is −1.920 for `fpo-policy` and 0.312 for
+   `dppo-policy` - about a tenth of the deviation.
+
+**Which of these stops `fpo-policy` learning under DPPO - either, both, or
+neither?**
+
+### What this cannot settle
+
+* Any cause beyond these two. If neither arm learns, the remaining
+  differences - network, chunk, number of denoising steps, output scale -
+  are the next question, not this one's.
+* Other tasks: E29 runs Hopper only, where E28's contrast is sharpest
+  (`dppo-policy` +260 to +417, `fpo-policy` +5 to +11).
+
+---
+
+## Declared in advance: what was already known
+
+1. **E28's `fpodppo-hopper`**, this experiment's cell unchanged: gain (mean of
+   iterations 91-100 minus the first) +10.6, +4.9, +9.7; `approx_kl` around
+   2.5 × 10⁻⁷ over the first ten iterations; entropy −1.920.
+2. **The probe** (`clamp_probe.py`, `results/pilot.txt`): the clamp's reach at
+   level 0.1 as above; at level 1.0, **no** `fpo-policy` element above 2
+   and 0.19% below −5 (`dppo-policy` at 0.1: 0.17%), entropy 0.383.
+3. **How far each actor moved in E28**, iteration 20 to 100, relative
+   distance of the actor's weights, seeds 0 / 1 / 2: `fpo-policy` **22.5,
+   20.0, 21.7%**, `dppo-policy` 11.9, 12.0, 11.6%. `fpo-policy`'s weights do move - further
+   than `dppo-policy`'s - while its action distribution barely changes.
+4. **The pilot** (`results/pilot.txt`), eleven iterations, seed 0: every arm
+   ran; each server logged the configuration intended; entropies −1.9198,
+   −1.9198, +0.3828. Against `control`, `noclamp` raised the mean
+   `approx_kl` 2.4-fold (5.9 × 10⁻⁷) and the actor's gradient norm 1.5-fold;
+   `wide` lowered `approx_kl` tenfold (2.7 × 10⁻⁸) - at an entropy close to
+   `dppo-policy`'s, whose `approx_kl` is about 10⁻⁵. No arm's return moved
+   in eleven iterations.
+
+---
+
+## Design
+
+| arm | extra algorithm flags | the clamp | the noise |
+| --- | --- | --- | --- |
+| `control` | none | binds on 50.7% | level 0.1 |
+| `noclamp` | `--algo.logprob-clamp-max inf` | lifted | level 0.1 |
+| `wide` | `--algo.sampling-noise-level 1.0 --algo.logprob-noise-level 1.0` | cannot bind | level 1.0, entropy ≈ `dppo-policy`'s |
+
+Every arm is E28's `fpodppo-hopper` cell: `fpo-policy`, `dppo hopper`,
+Hopper-v5, buffer 4,096, the variant's minibatch of 2,048, 100 iterations,
+seeds 0, 1, 2, checkpoints every 20 iterations. `run_cell.sh` is E24's with
+one addition, `EXTRA_ALGO_ARGS`. `wide` raises the level for both sampling
+and the log-probability, so the ratio is taken under the distribution the
+actions were drawn from. Only the upper bound is lifted in `noclamp`: it is
+the one that binds.
+
+`wide` changes the noise and, as a consequence, removes the clamp's reach
+(known item 2); `noclamp` removes the clamp's reach alone. The design cannot
+add noise while keeping the clamp's reach, so "the noise alone" is read
+from the two together, below.
+
+Machine `guangzhao`, CPU, one thread per process, the three arms at once.
+Code: `main` plus #64, run from this branch's worktree through `PYTHONPATH`;
+client plugrl-env-client at 931ab56.
+
+---
+
+## Checks
+
+* **V1 - the manipulation took.** Each arm's logged `losses/entropy`, which
+  depends only on the noise schedule, is within 0.005 of **−1.920**
+  (`control`, `noclamp`) and **−1.920 + ln 10 = +0.383** (`wide`) at every
+  iteration. An arm that fails V1 is not read.
+
+---
+
+## Predictions, and what falsifies each
+
+The gain is the mean return of iterations 91-100 minus the first
+iteration's.
+
+**P1 - every arm runs end to end**: 100 iterations logged, the checkpoint at
+100, no traceback, client exit 0, on every seed.
+
+**P2 - the control reproduces E28**: `control`'s gain is at most **+50** on
+every seed.
+
+> E28: +5 to +11. Falsified: the control moved, and nothing below is read.
+
+**P3 - lifting the clamp alone lets `fpo-policy` learn**: `noclamp`'s gain is
+at least **+100** on at least 2 of 3 seeds.
+
+**P4 - ten times the noise lets it learn**: `wide`'s gain is at least **+100**
+on at least 2 of 3 seeds.
+
+> +100 is about nine times the control's largest gain in E28 and under half
+> of `dppo-policy`'s smallest (+260).
+
+**What I expect, after the pilot: both P3 and P4 falsified.** In eleven
+iterations `noclamp` changed the update little and `wide` made the ratio
+move less, not more. The hypotheses are registered as the claims they would
+support, so that a falsification rules each one out.
+
+**Reading P3 and P4 together:**
+
+| P3 `noclamp` | P4 `wide` | reading |
+| --- | --- | --- |
+| holds | holds | the clamp is enough to stop it; the noise need not change |
+| holds | falsified | the clamp stops it, and ten times the noise hurts in some other way |
+| falsified | holds | the clamp alone is not it; the narrow noise is - with or without the clamp |
+| falsified | falsified | neither: both are ruled out as sufficient causes |
+
+**Reported, not predicted:** `approx_kl` and `clipfrac` over the first and
+last ten iterations; the actor's relative movement from iteration 20 to 100,
+beside E28's; each run's return and episode length.
+
+---
+
+## Declared deviations allowed in advance
+
+1. One restart of any run that dies for a reason outside the experiment,
+   recorded in `AMENDMENT.md`.
+2. A failure caused by how an arm was launched is fixed and the arm rerun
+   once, recorded in `AMENDMENT.md`.
+
+---
+
+## Reading order
+
+P1, V1, P2, then P3 and P4 and the table, then the reported figures.

experiments/e29-clamp-or-noise/clamp_probe.py

@@ -0,0 +1,45 @@
+"""How often does DPPO's logprob clamp [-5, 2] bind, per denoising step?
+
+Builds each policy fresh (random initialisation, as every E28 cell starts),
+samples a batch of actions with the Hopper variant's sampling_noise_level,
+and counts the per-element logprobs the loss would clamp. A clamped element
+contributes no gradient and a ratio of exactly 1.
+"""
+
+import sys
+
+import numpy as np
+import torch
+
+from plugrl_server.policy.dppo.dppo_policy import DPPOPolicy, DPPOPolicyConfigHopper
+from plugrl_server.policy.fpo.fpo_policy import FPOPolicy, FPOPolicyConfig
+
+LEVEL = float(sys.argv[1]) if len(sys.argv) > 1 else 0.1
+B = 4096
+torch.manual_seed(0)
+obs = {
+    "states": {
+        "obs": np.random.default_rng(0).standard_normal((B, 11)).astype(np.float32)
+    }
+}
+
+policies = {
+    "fpo-policy": FPOPolicy(FPOPolicyConfig(obs_dim=11, action_dim=3, device="cpu")),
+    "dppo-policy": DPPOPolicy(DPPOPolicyConfigHopper(device="cpu")),
+}
+print(f"sampling_noise_level {LEVEL}; fraction of logprob elements outside [-5, 2]")
+for name, policy in policies.items():
+    with torch.no_grad():
+        _, state = policy.get_action_and_runtime_state(obs, sampling_noise_level=LEVEL)
+    lp = state.logprob  # [B, steps, horizon, dim]
+    high = (lp > 2).float().mean(dim=(0, 2, 3))
+    low = (lp < -5).float().mean(dim=(0, 2, 3))
+    t = state.obs.t[0]
+    print(
+        f"{name}: {lp.shape[1]} steps, overall above 2: {float((lp > 2).float().mean()):.3f}, "
+        f"below -5: {float((lp < -5).float().mean()):.4f}, mean entropy {float(state.entropy.mean()):.3f}"
+    )
+    for i in range(lp.shape[1]):
+        print(
+            f"    step {i:2d} t={float(t[i]):6.3f}  above 2: {float(high[i]):.3f}  below -5: {float(low[i]):.4f}"
+        )

experiments/e29-clamp-or-noise/results/control.out

@@ -0,0 +1,11 @@
+cell:    control = fpo-policy/default x dppo/hopper x Hopper-v5
+iters:   100 x 4096  batch: variant default  replan: 1  seeds: 0 1 2
+extra:   none
+out:     /home/guangzhao/zuogou/plugrl/e29/experiments/e29-clamp-or-noise/results/control
+start 2026-09-25 21:36:26
+seed 0 finished rc=0 at 21:56:23
+seed 1 finished rc=0 at 21:56:35
+seed 2 finished rc=0 at 21:56:36
+end   2026-09-25 21:56:36
+failed seeds: 0
+CELL_DONE control