diff --git a/experiments/e57-clean-eval/AMENDMENT.md b/experiments/e57-clean-eval/AMENDMENT.md new file mode 100644 index 0000000..652cd86 --- /dev/null +++ b/experiments/e57-clean-eval/AMENDMENT.md @@ -0,0 +1,18 @@ +# E57 amendment + +## 2026-10-03 00:25 EDT: summarise.py read a blank cell as a number + +All 834 registered evaluations had finished and none had failed, before +anything was summarised. Then `summarise.py` stopped with +`ValueError: invalid literal for int() with base 10: ''`. + +The cause is the six runs of E54's two loud faults, `indices:reorder` and +`reward:short-array`. They raised an error, logged no return, and so have a +blank `strict` in E54's `runs.csv`. `summarise.py` converted `rule` and +`strict` to integers without allowing for a blank. + +The fix is in one line: a blank cell becomes `None`. A run with `None` is +not "silent on the curve". The protocol leaves the loud faults out of every +count, so the fix changes no count, no rule and no prediction. Nothing else +in `summarise.py`, `verdicts.py` or `evaluate.py` changed. No result had been +seen when the fix was made: the crash came before any output. diff --git a/experiments/e57-clean-eval/FINDINGS.md b/experiments/e57-clean-eval/FINDINGS.md new file mode 100644 index 0000000..7bd33af --- /dev/null +++ b/experiments/e57-clean-eval/FINDINGS.md @@ -0,0 +1,110 @@ +# E57: on a clean environment, the policies behind the faults that both curve checks miss are as good as fault-free ones; every policy the faults wrecked, the curve checks had already caught + +2026-10-03 · guangzhao (E54's `bridges-venv`) · protocol: +[`PROTOCOL.md`](PROTOCOL.md), after the pilot in `results/pilot/`, before +the registered runs · amended once, see [`AMENDMENT.md`](AMENDMENT.md): a +parsing fix in `summarise.py`, made before any output, that changes no count + +--- + +## The result + +Each of the 834 final policies of E54 and E56 was run on a fresh gymnasium +environment for 50 deterministic episodes, all policies on the same seeds. +"Harmed" means its mean return, the clean return, fell more than 3 standard +deviations below the fault-free band (`none`, seeds 10-19). + +Boundary-fault runs, seeds 0-2: + +| env | dose | harmed | silent on the curve | of them harmed | +| --- | --- | --- | --- | --- | +| Pendulum | one | 1/63 | 63 | 1 | +| | 0.001 | 2/63 | 58 | 1 | +| | 0.01 | 1/63 | 59 | 1 | +| | 0.1 | 6/63 | 50 | 2 | +| | 1.0 | 24/63 | 25 | 0 | +| HalfCheetah | one | 0/63 | 60 | 0 | +| | 0.01 | 0/63 | 57 | 0 | +| | 1.0 | 28/63 | 31 | 0 | +| Hopper | one | 0/69 | 69 | 0 | +| | 0.01 | 0/69 | 66 | 0 | +| | 1.0 | 19/69 | 35 | 0 | + +"Silent on the curve" means the run passed both the gain threshold and +seed spread. + +Every check and prediction holds (`results/verdicts.txt`): +- **V1:** 834 of 834 evaluations, each of 50 episodes. +- **V2:** the 12 weights shared by two or more runs got identical clean + returns, so evaluation is deterministic. +- **P1:** the harm rule flagged none of the ten fault-free band seeds in any + environment. +- **P2:** at one value, 1 of 63, 0 of 63 and 0 of 69 boundary-fault runs + were harmed. +- **P3:** Pendulum's `reward:zero` at every value logged 0, the best return + Pendulum gives, at all three seeds. Its policies scored -1196, -1550 and + -1639 on the clean environment, against a band of -186 ± 26. + +--- + +## What it means + +**Every policy the faults wrecked, the curve checks had already caught.** At +every value, the faults harmed 24, 28 and 19 runs. None of them passed +both curve checks. At HalfCheetah and Hopper, no run that passed both checks +was harmed, at any dose. + +**The faults the curve checks miss leave policies as good as fault-free +ones.** This is an exploratory analysis, not in the protocol (`explore.py`, +`results/explore.txt`). For each environment and dose, the runs silent on +the curve have a mean clean return 0.03-0.65 band s.d. *above* the band. +None is significantly below the 13 fault-free runs (two-sided Mann-Whitney +U). The one significant difference is upward: Pendulum at every value, +p = 0.003. At their own seed, they beat the fault-free twin by a median of 7-9 on +Pendulum, 63-101 on HalfCheetah and 142-368 on Hopper. That is consistent with a +change of seed, not with harm. + +**The exception is five Pendulum runs.** They passed both curve checks but +scored below every one of the 13 fault-free runs, whose worst was -238: +- `reward:f16` at 0.1%: -744; +- `obs:swap` at 1%: -506; +- `obs:stale` at one value: -430; +- `final-obs:f16` at 10%: -333; +- `obs:stale` at 10%: -277. + +That is 5 of 255 silent runs on Pendulum, against 0 of 13 fault-free. +Thirteen fault-free runs cannot tell a rare harm from a rare bad seed. + +**So a silent fault is silent on a clean evaluation too.** In these tasks, +with SB3's PPO, a fault that leaves the curve looking right also leaves +the policy looking right. What it changes is the run. The weights differ +from the run the authors think they trained, and with log faults or reward +faults, so do the numbers they report. Pendulum's `reward:zero` is the +extreme case: its log read 0, the best possible, while its policy scored +-1196 to -1639. Only a check of the boundary itself, the step ledger or the +episode record, tells such a run from a correct one. + +**On trusting the trainer's log.** For each run, the logged final return +minus the clean return is in `results/summary.txt`. The fault-free runs' +own gap is wide: Hopper's ranges from -1667 to +806. Training returns come +from stochastic actions and the clean return from deterministic ones. A +gap this wide hides all but gross reward faults. + +--- + +## What E57 does not show + +* **Any trainer but SB3's PPO, or any task but these three.** On a task + where small changes in the data matter more, such as sparse rewards, + long horizons or real robots, the same faults may harm the policy. +* **Harm too rare for 13 fault-free runs to bound.** The five Pendulum + outliers above are the edge of what this design can see. +* **Harm in a comparison between methods.** E57 compares policies with the + fault-free band, not one method with another. + +Kept here, in `results/`: `clean.csv` (`summarise.py`, one row per run, with +its clean return, harm, curve checks and logged return), `summary.txt`, +`verdicts.txt` (`verdicts.py`), `explore.txt` (`explore.py`), `jobs.txt`, +`load.txt`, `registered.out` and the source hashes. Each run's +`clean_eval.json`, with all 50 episodes, is kept on guangzhao. Evaluation +is deterministic (V2), so `run.sh` reproduces it. diff --git a/experiments/e57-clean-eval/PROTOCOL.md b/experiments/e57-clean-eval/PROTOCOL.md new file mode 100644 index 0000000..33043c8 --- /dev/null +++ b/experiments/e57-clean-eval/PROTOCOL.md @@ -0,0 +1,118 @@ +# E57 measurement protocol (pre-registered) + +**Written 2026-10-03, after the pilot in `results/pilot/`, before the +registered runs.** + +This file must not be edited after the first registered data point. Anything +learned afterwards goes in `AMENDMENT.md`, dated. + +--- + +## The question + +E54 and E56 judged each run by what the trainer saw: its own log, its +weights, or the two sides' records. None of them asks what a faulted run +leaves behind, the policy. Reviewers asked whether the faults matter at all. + +**Evaluated on a clean environment, is a policy trained through a fault +worse than one trained without it? In particular, are the faults that both +curve checks miss?** And does the trainer's log agree with the clean +evaluation? + +--- + +## Declared in advance: what was already known + +1. **The policies.** Every E54 and E56 run saved its final policy + (`model.zip`): 590 runs of E54 (Pendulum, HalfCheetah) and 244 of E56 + (Hopper). Their outcomes under the gain threshold, seed spread, byte + identity, record and ledger are published (`runs.csv`, `ledger.csv`). No + policy had been evaluated outside training before the pilot. +2. **The pilot** (`results/pilot/`), 20 episodes each, on guangzhao: + + | env | `none` seed 0 | `none` seed 10 | `obs:f16` 1.0 seed 0 | + | --- | --- | --- | --- | + | Pendulum | -147.0 | -139.1 | -139.7 | + | HalfCheetah | 647.1 | 779.8 | 851.2 | + | Hopper | 3368.9 | 3651.3 | 1633.5 (mean length 482) | + + Pendulum's episodes vary by about 100 (initial states); HalfCheetah's by + about 130; Hopper's by 10-580. An evaluation took 0.3-2.5 s. + +--- + +## Design + +`evaluate.py RUN_DIR OUT_DIR --episodes 50`. It does the following: +- loads the run's `model.zip` on CPU, with one torch thread; +- makes a fresh `gymnasium.make(id)` at default settings (Pendulum-v1, + HalfCheetah-v5, Hopper-v5), with no bridge, client, wrapper or fault; +- plays 50 episodes with `deterministic=True`, reset with seeds 10000-10049, + which are the same for every policy; +- writes `clean_eval.json`: each episode's return and length, and their mean, + the **clean return**. + +Every run directory of E54 and E56 that holds a `model.zip` is evaluated: +834 runs, 12 at a time on guangzhao (`run.sh`). Then `python summarise.py` +and `python verdicts.py`. + +--- + +## How each run is judged + +- **The fault-free band**, per environment: the clean returns of `none` at + seeds 10-19. The band is its mean and standard deviation. +- **Harmed:** a run's clean return is more than 3 standard deviations below + the band's mean. The rule is the seed-spread check's, applied to the clean + return, and only on the low side. A fault-free seed among 10-19 is judged + against the other nine. +- **Silent on the curve:** the run passed both the gain threshold and seed + spread (`rule` = 0 and `strict` = 0 in E54's or E56's `runs.csv`). +- **The log's error:** the trainer's logged final return (`last` in + `runs.csv`) minus the clean return. + +Boundary faults are E54's 21 client faults (23 on Hopper). Log faults, +controls (`none`, `delay-1ms`, `wire-float64`) and the two loud faults +(`indices:reorder`, `reward:short-array`) are reported separately. The +loud faults raised an error, so they are left out of every count. + +--- + +## Checks that the runs are valid + +- **V1:** all 834 evaluations complete, each with 50 episodes. +- **V2:** two runs with the same weights hash get the same clean return. The + controls and log faults ended on `none`'s weights, so this tests that + evaluation is deterministic. + +--- + +## Predictions, and what falsifies each + +**P1 - The harm rule raises few false alarms.** It flags at most 1 of the +ten fault-free seeds 10-19 in each environment, each judged against the +other nine. Falsified by 2 or more in any environment. + +**P2 - One changed value does no measurable harm.** At dose `one`, at most +10% of boundary-fault runs are harmed, in each environment. Falsified by +more. + +**P3 - A zeroed reward leaves a policy that does not work, behind a log +that looks perfect.** On Pendulum at dose 1.0, `reward:zero` is harmed at +all three seeds, while its log reads 0, the best return Pendulum can give. +Falsified by any seed not harmed. + +> Grounds: P1, E54's and E56's seed-spread false alarms (0, 1, 0 of ten). P2, +> one changed value moved the logged return about as much as a change of +> seed (E54, E56). P3, with every reward zero PPO's advantages are zero +> after normalisation, and nothing is learned. + +**Reported, not predicted:** +- **The main count.** Among boundary-fault runs, at each dose: + - how many are harmed; + - how many are harmed while silent on the curve. +- **The faults behind it.** Every fault and dose that was silent on the curve + at all three seeds and harmed at two or more. +- **The log's error.** For each kind, against the fault-free runs' range. +- **The pilot's lead.** Whether Hopper's `obs:f16` at 1.0 is harmed at seeds 1 + and 2. The pilot saw it at seed 0 only. diff --git a/experiments/e57-clean-eval/evaluate.py b/experiments/e57-clean-eval/evaluate.py new file mode 100644 index 0000000..6d5a9d9 --- /dev/null +++ b/experiments/e57-clean-eval/evaluate.py @@ -0,0 +1,64 @@ +"""E57: evaluate one run's final policy on a clean environment. + + python evaluate.py RUN_DIR OUT_DIR [--episodes N] + +RUN_DIR is an E54 or E56 run directory (model.zip, result.json). The policy +acts deterministically in a fresh gymnasium environment of the same id, at +its default settings, for N episodes reset with seeds 10000, 10001, ... No +bridge, no env client, no fault. Writes OUT_DIR/clean_eval.json. +""" + +import argparse +import json +import pathlib +import time + +import gymnasium as gym +import numpy as np +import torch +from stable_baselines3 import PPO + +IDS = {"pendulum": "Pendulum-v1", "halfcheetah": "HalfCheetah-v5", "hopper": "Hopper-v5"} +SEED0 = 10_000 + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("run", type=pathlib.Path) + ap.add_argument("out", type=pathlib.Path) + ap.add_argument("--episodes", type=int, default=20) + a = ap.parse_args() + torch.set_num_threads(1) + + res = json.loads((a.run / "result.json").read_text()) + env = gym.make(IDS[res["env"]]) + model = PPO.load(a.run / "model.zip", device="cpu") + + returns, lengths = [], [] + t0 = time.time() + for k in range(a.episodes): + obs, _ = env.reset(seed=SEED0 + k) + done, ret, n = False, 0.0, 0 + while not done: + act, _ = model.predict(obs, deterministic=True) + obs, r, term, trunc, _ = env.step(act) + ret += float(r) + n += 1 + done = term or trunc + returns.append(ret) + lengths.append(n) + + out = dict( + env=res["env"], fault=res["fault"], dose=res["dose"], seed=res["seed"], + weights_sha256=res["weights_sha256"], episodes=a.episodes, + returns=returns, lengths=lengths, mean=float(np.mean(returns)), + eval_s=round(time.time() - t0, 3), + ) + a.out.mkdir(parents=True, exist_ok=True) + (a.out / "clean_eval.json").write_text(json.dumps(out)) + print(json.dumps({k: out[k] for k in ("env", "fault", "dose", "seed", "mean", "eval_s")})) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/e57-clean-eval/explore.py b/experiments/e57-clean-eval/explore.py new file mode 100644 index 0000000..5372406 --- /dev/null +++ b/experiments/e57-clean-eval/explore.py @@ -0,0 +1,51 @@ +"""E57, exploratory (not in PROTOCOL.md): are the runs both curve checks miss +shifted as a group, though no single one is harmed? + + python explore.py [results] + +For each env and dose, the clean returns of boundary-fault runs that are +silent on the curve, in units of the fault-free band (none, seeds 10-19): +their mean z, and a two-sided Mann-Whitney U test against the 13 fault-free +runs (seeds 0-2 and 10-19). Also the clean-return gap between each such run +and the fault-free run at its own seed. Writes results/explore.txt. +""" + +import csv +import pathlib +import statistics as st +import sys + +from scipy.stats import mannwhitneyu + +HERE = pathlib.Path(__file__).resolve().parent + + +def main() -> int: + root = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else HERE / "results" + rows = list(csv.DictReader(open(root / "clean.csv"))) + for r in rows: + r["seed"], r["clean"] = int(r["seed"]), float(r["clean"]) + out = ["exploratory: boundary-fault runs silent on the curve, against fault-free runs", + "env dose n mean z median z U-test p twin gap median (clean - none at same seed)"] + for env in sorted({r["env"] for r in rows}): + band = [r["clean"] for r in rows if r["env"] == env and r["fault"] == "none" and r["seed"] >= 10] + mu, sd = st.mean(band), st.stdev(band) + free = [r["clean"] for r in rows if r["env"] == env and r["fault"] == "none"] + twin = {r["seed"]: r["clean"] for r in rows if r["env"] == env and r["fault"] == "none"} + for dose in ("one", "0.001", "0.01", "0.1", "1.0"): + s = [r for r in rows if r["env"] == env and r["dose"] == dose and r["kind"] == "boundary" + and r["seed"] < 10 and r["silent_curve"] == "1"] + if not s: + continue + z = [(r["clean"] - mu) / sd for r in s] + p = mannwhitneyu([r["clean"] for r in s], free, alternative="two-sided").pvalue + gap = st.median(r["clean"] - twin[r["seed"]] for r in s) + out.append(f"{env:12} {dose:6} {len(s):3} {st.mean(z):6.2f} {st.median(z):8.2f} {p:8.3f} {gap:9.1f}") + text = "\n".join(out) + (root / "explore.txt").write_text(text + "\n") + print(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/e57-clean-eval/results/clean.csv b/experiments/e57-clean-eval/results/clean.csv new file mode 100644 index 0000000..9b299c8 --- /dev/null +++ b/experiments/e57-clean-eval/results/clean.csv @@ -0,0 +1,835 @@ +env,fault,dose,seed,kind,episodes,clean,harmed,rule,strict,silent_curve,last,log_error,sha +halfcheetah,action:clip,0.01,0,boundary,50,1131.19,0,0,0,1,706.5,-424.69,ec5a179d +halfcheetah,action:clip,0.01,1,boundary,50,835.7,0,0,0,1,258.8,-576.9,7eceb0cc +halfcheetah,action:clip,0.01,2,boundary,50,971.14,0,0,0,1,578.3,-392.84,35ecae43 +halfcheetah,action:clip,1.0,0,boundary,50,-234.69,1,0,1,0,1118.5,1353.19,8ee6a77d +halfcheetah,action:clip,1.0,1,boundary,50,-269.26,1,0,1,0,1148.4,1417.66,14847897 +halfcheetah,action:clip,1.0,2,boundary,50,-360.08,1,0,1,0,994.4,1354.48,ff882cd9 +halfcheetah,action:clip,one,0,boundary,50,926.03,0,0,0,1,476.1,-449.93,f7759731 +halfcheetah,action:clip,one,1,boundary,50,842.94,0,0,0,1,577.7,-265.24,cf4d7c9a +halfcheetah,action:clip,one,2,boundary,50,849.95,0,0,0,1,710.8,-139.15,9f06912c +halfcheetah,action:f16,0.01,0,boundary,50,920.49,0,0,0,1,526.4,-394.09,a01811f5 +halfcheetah,action:f16,0.01,1,boundary,50,1027.76,0,0,0,1,624.9,-402.86,fbd338e0 +halfcheetah,action:f16,0.01,2,boundary,50,734.64,0,0,0,1,590.0,-144.64,01547561 +halfcheetah,action:f16,1.0,0,boundary,50,941.44,0,0,0,1,575.9,-365.54,bf57cc96 +halfcheetah,action:f16,1.0,1,boundary,50,1050.5,0,0,0,1,594.4,-456.1,ae75b2d6 +halfcheetah,action:f16,1.0,2,boundary,50,842.72,0,0,0,1,507.9,-334.82,00ec2a01 +halfcheetah,action:f16,one,0,boundary,50,1000.49,0,0,0,1,500.7,-499.79,91d85ccb +halfcheetah,action:f16,one,1,boundary,50,782.65,0,0,0,1,488.8,-293.85,4014b47b +halfcheetah,action:f16,one,2,boundary,50,1191.03,0,0,0,1,818.1,-372.93,f3b2bad8 +halfcheetah,action:stale,0.01,0,boundary,50,879.57,0,0,0,1,537.5,-342.07,b0751b3a +halfcheetah,action:stale,0.01,1,boundary,50,1043.57,0,0,0,1,616.0,-427.57,41460c07 +halfcheetah,action:stale,0.01,2,boundary,50,660.89,0,0,0,1,553.2,-107.69,a25b31ba +halfcheetah,action:stale,1.0,0,boundary,50,419.45,0,0,1,0,30.5,-388.95,e3802a5d +halfcheetah,action:stale,1.0,1,boundary,50,370.58,0,0,0,1,254.4,-116.18,dd17a435 +halfcheetah,action:stale,1.0,2,boundary,50,733.52,0,0,0,1,402.8,-330.72,74196379 +halfcheetah,action:stale,one,0,boundary,50,797.6,0,0,0,1,328.4,-469.2,63d03276 +halfcheetah,action:stale,one,1,boundary,50,1160.24,0,0,0,1,605.8,-554.44,7e1a788a +halfcheetah,action:stale,one,2,boundary,50,958.32,0,0,0,1,661.6,-296.72,db4fec84 +halfcheetah,action:swap,0.01,0,boundary,50,641.77,0,0,0,1,356.2,-285.57,21898a3c +halfcheetah,action:swap,0.01,1,boundary,50,981.74,0,0,0,1,595.5,-386.24,90971b4a +halfcheetah,action:swap,0.01,2,boundary,50,841.9,0,0,0,1,530.4,-311.5,50bf08ce +halfcheetah,action:swap,1.0,0,boundary,50,-398.64,1,1,1,0,-360.2,38.44,a46e0958 +halfcheetah,action:swap,1.0,1,boundary,50,-608.15,1,1,1,0,-366.4,241.75,03050f8b +halfcheetah,action:swap,1.0,2,boundary,50,-322.58,1,1,1,0,-334.6,-12.02,33392cdc +halfcheetah,action:swap,one,0,boundary,50,919.89,0,0,0,1,503.2,-416.69,ef171afa +halfcheetah,action:swap,one,1,boundary,50,1518.52,0,0,0,1,616.1,-902.42,5e7659c6 +halfcheetah,action:swap,one,2,boundary,50,1090.05,0,0,0,1,611.7,-478.35,47c3f841 +halfcheetah,action:zero,0.01,0,boundary,50,901.43,0,0,0,1,517.0,-384.43,562a50ad +halfcheetah,action:zero,0.01,1,boundary,50,694.46,0,0,0,1,290.0,-404.46,d1cddc03 +halfcheetah,action:zero,0.01,2,boundary,50,1034.41,0,0,0,1,561.0,-473.41,bf2c4b43 +halfcheetah,action:zero,1.0,0,boundary,50,-463.52,1,1,1,0,-0.1,463.42,71e141b6 +halfcheetah,action:zero,1.0,1,boundary,50,-1521.24,1,1,1,0,-0.1,1521.14,50e349ce +halfcheetah,action:zero,1.0,2,boundary,50,-1213.03,1,1,1,0,-0.1,1212.93,34d1323f +halfcheetah,action:zero,one,0,boundary,50,1085.87,0,0,0,1,593.0,-492.87,7f4cc3d5 +halfcheetah,action:zero,one,1,boundary,50,883.19,0,0,0,1,482.0,-401.19,b5b83f62 +halfcheetah,action:zero,one,2,boundary,50,898.72,0,0,0,1,700.7,-198.02,d2c74325 +halfcheetah,final-obs:f16,0.01,0,boundary,50,849.15,0,0,0,1,444.3,-404.85,9df3b350 +halfcheetah,final-obs:f16,0.01,1,boundary,50,963.82,0,0,0,1,657.4,-306.42,6f562a9b +halfcheetah,final-obs:f16,0.01,2,boundary,50,1114.3,0,0,0,1,742.9,-371.4,4a37d428 +halfcheetah,final-obs:f16,1.0,0,boundary,50,948.65,0,0,0,1,493.1,-455.55,f750ab5d +halfcheetah,final-obs:f16,1.0,1,boundary,50,1637.62,0,0,0,1,736.7,-900.92,73173785 +halfcheetah,final-obs:f16,1.0,2,boundary,50,928.27,0,0,0,1,722.4,-205.87,7b42bf11 +halfcheetah,final-obs:f16,one,0,boundary,50,1057.5,0,0,0,1,558.9,-498.6,23b735c6 +halfcheetah,final-obs:f16,one,1,boundary,50,973.79,0,0,0,1,677.8,-295.99,67aa0867 +halfcheetah,final-obs:f16,one,2,boundary,50,1082.02,0,0,0,1,649.9,-432.12,d57d7e32 +halfcheetah,final-obs:stale,0.01,0,boundary,50,842.08,0,0,0,1,474.0,-368.08,4e63dcef +halfcheetah,final-obs:stale,0.01,1,boundary,50,978.38,0,0,0,1,629.7,-348.68,9049b7a0 +halfcheetah,final-obs:stale,0.01,2,boundary,50,1118.35,0,0,0,1,824.4,-293.95,5ff8d630 +halfcheetah,final-obs:stale,1.0,0,boundary,50,1255.75,0,0,0,1,707.7,-548.05,f3b8ca66 +halfcheetah,final-obs:stale,1.0,1,boundary,50,998.84,0,0,0,1,518.4,-480.44,0d4a77ab +halfcheetah,final-obs:stale,1.0,2,boundary,50,935.51,0,0,0,1,644.3,-291.21,cff4303f +halfcheetah,final-obs:stale,one,0,boundary,50,808.4,0,0,0,1,404.7,-403.7,946e386b +halfcheetah,final-obs:stale,one,1,boundary,50,949.91,0,0,0,1,607.5,-342.41,e7b8784a +halfcheetah,final-obs:stale,one,2,boundary,50,997.93,0,0,0,1,748.8,-249.13,1bec1000 +halfcheetah,final-obs:swap,0.01,0,boundary,50,850.84,0,0,0,1,439.9,-410.94,c8eb7cb5 +halfcheetah,final-obs:swap,0.01,1,boundary,50,853.07,0,0,0,1,586.8,-266.27,9ce1d515 +halfcheetah,final-obs:swap,0.01,2,boundary,50,1131.02,0,0,0,1,620.9,-510.12,64036db1 +halfcheetah,final-obs:swap,1.0,0,boundary,50,981.59,0,0,0,1,583.1,-398.49,f5eaca9e +halfcheetah,final-obs:swap,1.0,1,boundary,50,1538.26,0,0,0,1,503.5,-1034.76,3b5968a4 +halfcheetah,final-obs:swap,1.0,2,boundary,50,1101.74,0,0,0,1,657.2,-444.54,02a9c7bd +halfcheetah,final-obs:swap,one,0,boundary,50,894.17,0,0,0,1,424.4,-469.77,56d822f2 +halfcheetah,final-obs:swap,one,1,boundary,50,1094.01,0,0,0,1,721.1,-372.91,dfd049ad +halfcheetah,final-obs:swap,one,2,boundary,50,1060.68,0,0,0,1,657.7,-402.98,f3452536 +halfcheetah,final-obs:zero,0.01,0,boundary,50,1009.32,0,0,0,1,573.8,-435.52,1b8de501 +halfcheetah,final-obs:zero,0.01,1,boundary,50,912.64,0,0,0,1,652.2,-260.44,56adf1b7 +halfcheetah,final-obs:zero,0.01,2,boundary,50,1050.83,0,0,0,1,645.7,-405.13,17cf743d +halfcheetah,final-obs:zero,1.0,0,boundary,50,975.65,0,0,0,1,568.8,-406.85,894878e1 +halfcheetah,final-obs:zero,1.0,1,boundary,50,815.35,0,0,0,1,291.8,-523.55,49f32b02 +halfcheetah,final-obs:zero,1.0,2,boundary,50,825.11,0,0,0,1,584.1,-241.01,b4f41780 +halfcheetah,final-obs:zero,one,0,boundary,50,1022.02,0,0,0,1,572.4,-449.62,7c7a4867 +halfcheetah,final-obs:zero,one,1,boundary,50,914.18,0,0,0,1,622.3,-291.88,51f6ae06 +halfcheetah,final-obs:zero,one,2,boundary,50,1058.3,0,0,0,1,614.9,-443.4,b0b04148 +halfcheetah,log:drop-first,0.01,0,log,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,log:drop-first,0.01,1,log,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,log:drop-first,0.01,2,log,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,log:drop-first,1.0,0,log,50,647.5,0,0,0,1,206.4,-441.1,8261c03a +halfcheetah,log:drop-first,1.0,1,log,50,935.8,0,0,0,1,604.2,-331.6,86564694 +halfcheetah,log:drop-first,1.0,2,log,50,970.83,0,0,0,1,711.7,-259.13,65cc8843 +halfcheetah,log:drop-first,one,0,log,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,log:drop-first,one,1,log,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,log:drop-first,one,2,log,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,log:short,0.01,0,log,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,log:short,0.01,1,log,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,log:short,0.01,2,log,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,log:short,1.0,0,log,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,log:short,1.0,1,log,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,log:short,1.0,2,log,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,log:short,one,0,log,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,log:short,one,1,log,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,log:short,one,2,log,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,none,1.0,0,control,50,647.5,0,0,0,1,206.6,-440.9,8261c03a +halfcheetah,none,1.0,1,control,50,935.8,0,0,0,1,604.3,-331.5,86564694 +halfcheetah,none,1.0,10,control,50,787.88,0,0,0,1,327.9,-459.98,a14df8cd +halfcheetah,none,1.0,11,control,50,971.44,0,0,0,1,520.5,-450.94,a565e4a1 +halfcheetah,none,1.0,12,control,50,985.35,0,0,0,1,610.0,-375.35,634d7d37 +halfcheetah,none,1.0,13,control,50,1096.71,0,0,0,1,617.6,-479.11,dc8a23a3 +halfcheetah,none,1.0,14,control,50,880.72,0,0,0,1,474.8,-405.92,d31c08d5 +halfcheetah,none,1.0,15,control,50,949.75,0,0,0,1,520.4,-429.35,126daf08 +halfcheetah,none,1.0,16,control,50,505.24,0,0,0,1,152.4,-352.84,783e36d6 +halfcheetah,none,1.0,17,control,50,493.2,0,0,0,1,619.8,126.6,32a0bf84 +halfcheetah,none,1.0,18,control,50,895.41,0,0,0,1,485.0,-410.41,730fcd31 +halfcheetah,none,1.0,19,control,50,960.52,0,0,0,1,523.0,-437.52,8ed877cc +halfcheetah,none,1.0,2,control,50,970.83,0,0,0,1,711.6,-259.23,65cc8843 +halfcheetah,obs:f16,0.01,0,boundary,50,984.72,0,0,0,1,548.1,-436.62,253ad984 +halfcheetah,obs:f16,0.01,1,boundary,50,990.89,0,0,0,1,565.8,-425.09,533b211a +halfcheetah,obs:f16,0.01,2,boundary,50,911.38,0,0,0,1,654.2,-257.18,be11c8be +halfcheetah,obs:f16,1.0,0,boundary,50,872.43,0,0,0,1,477.3,-395.13,bfd9adbe +halfcheetah,obs:f16,1.0,1,boundary,50,849.1,0,0,0,1,529.8,-319.3,69ad858b +halfcheetah,obs:f16,1.0,2,boundary,50,1124.94,0,0,0,1,618.5,-506.44,752df5e1 +halfcheetah,obs:f16,one,0,boundary,50,1025.05,0,0,0,1,557.5,-467.55,eb75b798 +halfcheetah,obs:f16,one,1,boundary,50,856.75,0,0,0,1,584.9,-271.85,e147e00f +halfcheetah,obs:f16,one,2,boundary,50,1076.05,0,0,0,1,758.3,-317.75,245b4242 +halfcheetah,obs:stale,0.01,0,boundary,50,1144.33,0,0,0,1,604.6,-539.73,395d585f +halfcheetah,obs:stale,0.01,1,boundary,50,1113.29,0,0,0,1,465.9,-647.39,983100e4 +halfcheetah,obs:stale,0.01,2,boundary,50,923.47,0,0,0,1,506.2,-417.27,e7983591 +halfcheetah,obs:stale,1.0,0,boundary,50,167.48,1,0,1,0,9.7,-157.78,f6eb4fe1 +halfcheetah,obs:stale,1.0,1,boundary,50,447.55,0,0,0,1,504.1,56.55,1cc21e65 +halfcheetah,obs:stale,1.0,2,boundary,50,309.18,0,0,0,1,181.5,-127.68,7dd93559 +halfcheetah,obs:stale,one,0,boundary,50,846.58,0,0,0,1,551.1,-295.48,871b8995 +halfcheetah,obs:stale,one,1,boundary,50,922.51,0,0,0,1,533.2,-389.31,26dc0746 +halfcheetah,obs:stale,one,2,boundary,50,1005.36,0,0,0,1,600.8,-404.56,d909c6ce +halfcheetah,obs:swap,0.01,0,boundary,50,808.46,0,0,0,1,310.6,-497.86,d51b824a +halfcheetah,obs:swap,0.01,1,boundary,50,1098.38,0,0,0,1,593.5,-504.88,630e98be +halfcheetah,obs:swap,0.01,2,boundary,50,817.13,0,0,0,1,618.5,-198.63,865a3fd0 +halfcheetah,obs:swap,1.0,0,boundary,50,45.77,1,0,1,0,-150.3,-196.07,59cbe2ef +halfcheetah,obs:swap,1.0,1,boundary,50,-91.58,1,1,1,0,-167.8,-76.22,150621e8 +halfcheetah,obs:swap,1.0,2,boundary,50,-44.63,1,0,1,0,-114.3,-69.67,f98c1ebc +halfcheetah,obs:swap,one,0,boundary,50,727.22,0,0,0,1,327.4,-399.82,81776260 +halfcheetah,obs:swap,one,1,boundary,50,839.11,0,0,0,1,486.6,-352.51,370e715d +halfcheetah,obs:swap,one,2,boundary,50,1012.33,0,0,0,1,742.7,-269.63,e06e727f +halfcheetah,obs:zero,0.01,0,boundary,50,969.36,0,0,0,1,610.4,-358.96,f879169b +halfcheetah,obs:zero,0.01,1,boundary,50,970.21,0,0,0,1,593.3,-376.91,dee5d071 +halfcheetah,obs:zero,0.01,2,boundary,50,978.73,0,0,0,1,581.1,-397.63,2d981280 +halfcheetah,obs:zero,1.0,0,boundary,50,-74.66,1,0,1,0,-102.8,-28.14,bda2c4b3 +halfcheetah,obs:zero,1.0,1,boundary,50,-37.26,1,0,1,0,-28.0,9.26,3a4d3e4f +halfcheetah,obs:zero,1.0,2,boundary,50,-39.5,1,0,1,0,-97.3,-57.8,f1005610 +halfcheetah,obs:zero,one,0,boundary,50,769.82,0,0,0,1,450.5,-319.32,99315557 +halfcheetah,obs:zero,one,1,boundary,50,996.39,0,0,0,1,590.6,-405.79,29ff0dd3 +halfcheetah,obs:zero,one,2,boundary,50,1174.65,0,0,0,1,703.3,-471.35,75cd4907 +halfcheetah,reward:f16,0.01,0,boundary,50,869.95,0,0,0,1,455.8,-414.15,bccf9f6d +halfcheetah,reward:f16,0.01,1,boundary,50,1008.84,0,0,0,1,610.4,-398.44,0010ace3 +halfcheetah,reward:f16,0.01,2,boundary,50,1058.88,0,0,0,1,638.3,-420.58,304dca87 +halfcheetah,reward:f16,1.0,0,boundary,50,975.04,0,0,0,1,567.3,-407.74,267455a6 +halfcheetah,reward:f16,1.0,1,boundary,50,1133.53,0,0,0,1,250.7,-882.83,462878df +halfcheetah,reward:f16,1.0,2,boundary,50,683.49,0,0,0,1,641.6,-41.89,0df11866 +halfcheetah,reward:f16,one,0,boundary,50,950.17,0,0,0,1,550.7,-399.47,ab4d1427 +halfcheetah,reward:f16,one,1,boundary,50,818.85,0,0,0,1,541.9,-276.95,cec2abc4 +halfcheetah,reward:f16,one,2,boundary,50,1039.68,0,0,0,1,740.5,-299.18,848926bb +halfcheetah,reward:stale,0.01,0,boundary,50,917.89,0,0,0,1,556.6,-361.29,f5e5c0ef +halfcheetah,reward:stale,0.01,1,boundary,50,1025.57,0,0,0,1,672.3,-353.27,ecd5403d +halfcheetah,reward:stale,0.01,2,boundary,50,918.75,0,0,0,1,527.8,-390.95,24cc43f9 +halfcheetah,reward:stale,1.0,0,boundary,50,1067.42,0,0,0,1,531.9,-535.52,6d2bb921 +halfcheetah,reward:stale,1.0,1,boundary,50,861.11,0,0,0,1,536.7,-324.41,fc88296f +halfcheetah,reward:stale,1.0,2,boundary,50,537.79,0,0,0,1,634.3,96.51,d5c0f68a +halfcheetah,reward:stale,one,0,boundary,50,926.56,0,0,0,1,465.4,-461.16,fce9d70d +halfcheetah,reward:stale,one,1,boundary,50,632.89,0,0,0,1,598.3,-34.59,c8058732 +halfcheetah,reward:stale,one,2,boundary,50,1067.07,0,0,0,1,618.0,-449.07,2edd2c71 +halfcheetah,reward:swap,0.01,0,boundary,50,1048.64,0,0,0,1,380.4,-668.24,3f092318 +halfcheetah,reward:swap,0.01,1,boundary,50,947.95,0,0,0,1,519.5,-428.45,ed625ff5 +halfcheetah,reward:swap,0.01,2,boundary,50,1044.61,0,0,0,1,569.8,-474.81,bb78994b +halfcheetah,reward:swap,1.0,0,boundary,50,-368.8,1,1,1,0,-393.2,-24.4,7ebd034e +halfcheetah,reward:swap,1.0,1,boundary,50,-222.11,1,1,1,0,-347.3,-125.19,7d80b12d +halfcheetah,reward:swap,1.0,2,boundary,50,-345.51,1,1,1,0,-377.2,-31.69,b4f7d9f7 +halfcheetah,reward:swap,one,0,boundary,50,895.85,0,0,0,1,540.8,-355.05,6cdaf68c +halfcheetah,reward:swap,one,1,boundary,50,952.02,0,0,0,1,600.1,-351.92,b62566d1 +halfcheetah,reward:swap,one,2,boundary,50,1211.49,0,0,0,1,613.4,-598.09,597e2c62 +halfcheetah,reward:zero,0.01,0,boundary,50,909.97,0,0,0,1,479.2,-430.77,82efd41f +halfcheetah,reward:zero,0.01,1,boundary,50,933.32,0,0,0,1,582.9,-350.42,2706382b +halfcheetah,reward:zero,0.01,2,boundary,50,464.66,0,0,0,1,556.3,91.64,8bfc3469 +halfcheetah,reward:zero,1.0,0,boundary,50,-482.74,1,1,1,0,0.0,482.74,6b7ff3c6 +halfcheetah,reward:zero,1.0,1,boundary,50,-204.75,1,1,1,0,0.0,204.75,e66e1103 +halfcheetah,reward:zero,1.0,2,boundary,50,-308.79,1,1,1,0,0.0,308.79,7e326e2a +halfcheetah,reward:zero,one,0,boundary,50,951.35,0,0,0,1,523.6,-427.75,4ba64301 +halfcheetah,reward:zero,one,1,boundary,50,844.47,0,0,0,1,636.7,-207.77,c659b6c3 +halfcheetah,reward:zero,one,2,boundary,50,1182.16,0,0,0,1,652.8,-529.36,68ae5d65 +halfcheetah,step:fill,0.01,0,boundary,50,1065.96,0,0,0,1,628.8,-437.16,2f3101b7 +halfcheetah,step:fill,0.01,1,boundary,50,560.74,0,0,0,1,504.9,-55.84,bfb369a5 +halfcheetah,step:fill,0.01,2,boundary,50,983.43,0,0,0,1,556.8,-426.63,8f6334a0 +halfcheetah,step:fill,1.0,0,boundary,50,-1057.3,1,1,1,0,,,e1e8baad +halfcheetah,step:fill,1.0,1,boundary,50,-590.33,1,1,1,0,,,3287709c +halfcheetah,step:fill,1.0,2,boundary,50,-1072.39,1,1,1,0,,,12869326 +halfcheetah,step:fill,one,0,boundary,50,1049.17,0,0,0,1,574.1,-475.07,53c93857 +halfcheetah,step:fill,one,1,boundary,50,964.93,0,0,0,1,579.2,-385.73,1e17bf66 +halfcheetah,step:fill,one,2,boundary,50,1005.25,0,0,0,1,687.7,-317.55,c8819de7 +halfcheetah,terminated:spurious,0.01,0,boundary,50,722.81,0,1,1,0,-29.9,-752.71,e2343ceb +halfcheetah,terminated:spurious,0.01,1,boundary,50,651.42,0,1,1,0,-25.2,-676.62,f4ee2288 +halfcheetah,terminated:spurious,0.01,2,boundary,50,605.59,0,1,1,0,-26.5,-632.09,455961b2 +halfcheetah,terminated:spurious,1.0,0,boundary,50,-126.65,1,1,1,0,-0.2,126.45,74906e7c +halfcheetah,terminated:spurious,1.0,1,boundary,50,-212.53,1,1,1,0,-0.2,212.33,b0fe3376 +halfcheetah,terminated:spurious,1.0,2,boundary,50,-190.64,1,1,1,0,-0.3,190.34,dfc955f6 +halfcheetah,terminated:spurious,one,0,boundary,50,553.71,0,0,0,1,153.3,-400.41,c73bee27 +halfcheetah,terminated:spurious,one,1,boundary,50,675.33,0,0,0,1,402.7,-272.63,fc151425 +halfcheetah,terminated:spurious,one,2,boundary,50,1174.3,0,0,0,1,757.0,-417.3,17ee96dd +halfcheetah,truncated:as-term,0.01,0,boundary,50,923.15,0,0,0,1,556.2,-366.95,a135d640 +halfcheetah,truncated:as-term,0.01,1,boundary,50,962.73,0,0,0,1,585.0,-377.73,4dd488a0 +halfcheetah,truncated:as-term,0.01,2,boundary,50,1189.14,0,0,0,1,707.7,-481.44,4142bf18 +halfcheetah,truncated:as-term,1.0,0,boundary,50,916.95,0,0,0,1,561.7,-355.25,44569672 +halfcheetah,truncated:as-term,1.0,1,boundary,50,979.92,0,0,0,1,503.4,-476.52,31fe8d84 +halfcheetah,truncated:as-term,1.0,2,boundary,50,556.98,0,0,0,1,555.0,-1.98,4dd63686 +halfcheetah,truncated:as-term,one,0,boundary,50,923.77,0,0,0,1,510.6,-413.17,2cedd064 +halfcheetah,truncated:as-term,one,1,boundary,50,879.61,0,0,0,1,527.9,-351.71,911a4ab6 +halfcheetah,truncated:as-term,one,2,boundary,50,1225.0,0,0,0,1,730.3,-494.7,1e586fe2 +halfcheetah,truncated:drop,0.01,0,boundary,50,828.27,0,1,1,0,,,c04e6e0f +halfcheetah,truncated:drop,0.01,1,boundary,50,1065.37,0,1,1,0,,,89cbf6ff +halfcheetah,truncated:drop,0.01,2,boundary,50,1013.67,0,1,1,0,,,e13f7427 +halfcheetah,truncated:drop,1.0,0,boundary,50,979.3,0,1,1,0,,,61eb52a9 +halfcheetah,truncated:drop,1.0,1,boundary,50,894.5,0,1,1,0,,,3b93a0aa +halfcheetah,truncated:drop,1.0,2,boundary,50,572.13,0,1,1,0,,,1f5772fb +halfcheetah,truncated:drop,one,0,boundary,50,1022.17,0,1,1,0,,,112dda46 +halfcheetah,truncated:drop,one,1,boundary,50,992.04,0,1,1,0,,,0db88f4e +halfcheetah,truncated:drop,one,2,boundary,50,1056.68,0,1,1,0,,,f9390fe1 +hopper,action:clip,0.01,0,boundary,50,3456.6,0,0,0,1,1978.6,-1478.0,6d8a059d +hopper,action:clip,0.01,1,boundary,50,1707.66,0,0,0,1,1794.9,87.24,bdd175c3 +hopper,action:clip,0.01,2,boundary,50,3564.8,0,0,0,1,1896.9,-1667.9,e7929304 +hopper,action:clip,1.0,0,boundary,50,248.45,0,1,1,0,787.6,539.15,38a5ded5 +hopper,action:clip,1.0,1,boundary,50,235.77,0,0,1,0,1201.3,965.53,cfad8be0 +hopper,action:clip,1.0,2,boundary,50,698.01,0,1,1,0,372.3,-325.71,24f64c24 +hopper,action:clip,one,0,boundary,50,2556.9,0,0,0,1,2344.5,-212.4,8be60679 +hopper,action:clip,one,1,boundary,50,3451.47,0,0,0,1,1797.0,-1654.47,52489707 +hopper,action:clip,one,2,boundary,50,3524.4,0,0,0,1,1620.8,-1903.6,fc696b76 +hopper,action:f16,0.01,0,boundary,50,2798.11,0,0,0,1,1816.2,-981.91,2e0158cf +hopper,action:f16,0.01,1,boundary,50,3581.65,0,0,0,1,2096.8,-1484.85,81ec6cfb +hopper,action:f16,0.01,2,boundary,50,3591.43,0,0,0,1,2197.9,-1393.53,d35e7771 +hopper,action:f16,1.0,0,boundary,50,3503.92,0,0,0,1,2368.1,-1135.82,d44914f5 +hopper,action:f16,1.0,1,boundary,50,3596.8,0,0,0,1,2014.9,-1581.9,16ef799d +hopper,action:f16,1.0,2,boundary,50,3588.96,0,0,0,1,2304.0,-1284.96,ce3db36d +hopper,action:f16,one,0,boundary,50,2521.57,0,0,0,1,1783.1,-738.47,179aeb27 +hopper,action:f16,one,1,boundary,50,3049.9,0,0,0,1,1834.4,-1215.5,4635140f +hopper,action:f16,one,2,boundary,50,3536.54,0,0,0,1,2192.5,-1344.04,f4266978 +hopper,action:stale,0.01,0,boundary,50,1829.59,0,0,0,1,1737.5,-92.09,44aa1d08 +hopper,action:stale,0.01,1,boundary,50,3556.76,0,0,0,1,1947.7,-1609.06,279da55f +hopper,action:stale,0.01,2,boundary,50,3577.45,0,0,0,1,2262.4,-1315.05,2e0a1dbe +hopper,action:stale,1.0,0,boundary,50,3303.25,0,0,0,1,1981.9,-1321.35,25538fca +hopper,action:stale,1.0,1,boundary,50,3480.7,0,0,0,1,1877.2,-1603.5,2bff1555 +hopper,action:stale,1.0,2,boundary,50,1659.95,0,0,0,1,1944.8,284.85,d144972a +hopper,action:stale,one,0,boundary,50,3390.55,0,0,0,1,1858.9,-1531.65,16b63912 +hopper,action:stale,one,1,boundary,50,3557.98,0,0,0,1,2067.3,-1490.68,fbe29342 +hopper,action:stale,one,2,boundary,50,2776.87,0,0,0,1,1895.5,-881.37,2b7e62ad +hopper,action:swap,0.01,0,boundary,50,3531.94,0,0,0,1,1612.8,-1919.14,b116e227 +hopper,action:swap,0.01,1,boundary,50,3582.29,0,0,0,1,2076.6,-1505.69,0a61ea98 +hopper,action:swap,0.01,2,boundary,50,3461.62,0,0,0,1,2035.2,-1426.42,e045239b +hopper,action:swap,1.0,0,boundary,50,5.06,1,1,1,0,8.1,3.04,1f07144f +hopper,action:swap,1.0,1,boundary,50,18.65,1,1,1,0,7.3,-11.35,20dec241 +hopper,action:swap,1.0,2,boundary,50,61.63,1,1,1,0,5.6,-56.03,52a8e45b +hopper,action:swap,one,0,boundary,50,2670.46,0,0,0,1,2131.2,-539.26,8a23ae42 +hopper,action:swap,one,1,boundary,50,3625.8,0,0,0,1,2167.7,-1458.1,1c793580 +hopper,action:swap,one,2,boundary,50,3557.04,0,0,0,1,2307.1,-1249.94,0a6e64d7 +hopper,action:zero,0.01,0,boundary,50,3349.21,0,0,0,1,1991.8,-1357.41,b3bedcf3 +hopper,action:zero,0.01,1,boundary,50,2655.58,0,0,0,1,1552.0,-1103.58,df18c84b +hopper,action:zero,0.01,2,boundary,50,1669.71,0,0,0,1,1770.9,101.19,cfa28245 +hopper,action:zero,1.0,0,boundary,50,132.22,1,1,1,0,147.0,14.78,a83be1a7 +hopper,action:zero,1.0,1,boundary,50,1.2,1,1,1,0,147.3,146.1,fe7fb370 +hopper,action:zero,1.0,2,boundary,50,11.87,1,1,1,0,146.5,134.63,1ae6351d +hopper,action:zero,one,0,boundary,50,3604.62,0,0,0,1,2017.1,-1587.52,233ff646 +hopper,action:zero,one,1,boundary,50,1269.65,0,0,0,1,2222.6,952.95,c65411f9 +hopper,action:zero,one,2,boundary,50,2686.33,0,0,0,1,1757.2,-929.13,182c7eab +hopper,delay-1ms,1.0,0,control,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,delay-1ms,1.0,1,control,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,delay-1ms,1.0,2,control,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,final-obs:f16,0.01,0,boundary,50,1249.49,0,0,0,1,1948.7,699.21,dab6bfa5 +hopper,final-obs:f16,0.01,1,boundary,50,3586.41,0,0,0,1,2066.4,-1520.01,32b4a88c +hopper,final-obs:f16,0.01,2,boundary,50,1292.2,0,0,0,1,1970.0,677.8,a4c1c920 +hopper,final-obs:f16,1.0,0,boundary,50,3006.91,0,0,0,1,1945.6,-1061.31,0822d8cd +hopper,final-obs:f16,1.0,1,boundary,50,3567.33,0,0,0,1,1787.0,-1780.33,f79ba434 +hopper,final-obs:f16,1.0,2,boundary,50,1197.67,0,0,0,1,1989.2,791.53,81b212a4 +hopper,final-obs:f16,one,0,boundary,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,final-obs:f16,one,1,boundary,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,final-obs:f16,one,2,boundary,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,final-obs:stale,0.01,0,boundary,50,3494.07,0,0,0,1,1779.5,-1714.57,d9789c49 +hopper,final-obs:stale,0.01,1,boundary,50,3594.96,0,0,0,1,1975.4,-1619.56,ab9b4b7b +hopper,final-obs:stale,0.01,2,boundary,50,2886.18,0,0,0,1,2090.4,-795.78,d20a7273 +hopper,final-obs:stale,1.0,0,boundary,50,3538.06,0,0,0,1,1960.8,-1577.26,c16e62f3 +hopper,final-obs:stale,1.0,1,boundary,50,3520.35,0,0,0,1,1955.7,-1564.65,94fe0250 +hopper,final-obs:stale,1.0,2,boundary,50,3604.21,0,0,0,1,1994.6,-1609.61,66dc8455 +hopper,final-obs:stale,one,0,boundary,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,final-obs:stale,one,1,boundary,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,final-obs:stale,one,2,boundary,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,final-obs:swap,0.01,0,boundary,50,2584.87,0,0,0,1,1829.5,-755.37,b3cb8a2d +hopper,final-obs:swap,0.01,1,boundary,50,2065.61,0,0,0,1,1914.3,-151.31,33911c22 +hopper,final-obs:swap,0.01,2,boundary,50,2446.75,0,0,0,1,1838.9,-607.85,f51c5465 +hopper,final-obs:swap,1.0,0,boundary,50,3558.69,0,0,0,1,1855.0,-1703.69,88e663e0 +hopper,final-obs:swap,1.0,1,boundary,50,3561.6,0,0,0,1,1925.4,-1636.2,1184a831 +hopper,final-obs:swap,1.0,2,boundary,50,3355.54,0,0,0,1,2025.1,-1330.44,3fdf94e1 +hopper,final-obs:swap,one,0,boundary,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,final-obs:swap,one,1,boundary,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,final-obs:swap,one,2,boundary,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,final-obs:zero,0.01,0,boundary,50,1353.25,0,0,0,1,1836.1,482.85,53b50822 +hopper,final-obs:zero,0.01,1,boundary,50,3586.05,0,0,0,1,1890.2,-1695.85,4fda6f17 +hopper,final-obs:zero,0.01,2,boundary,50,3210.49,0,0,0,1,2053.1,-1157.39,853b75a0 +hopper,final-obs:zero,1.0,0,boundary,50,3191.61,0,0,0,1,1858.2,-1333.41,5443fa72 +hopper,final-obs:zero,1.0,1,boundary,50,1387.33,0,0,0,1,1922.8,535.47,fc096467 +hopper,final-obs:zero,1.0,2,boundary,50,1329.53,0,0,0,1,2034.1,704.57,8d9a34a3 +hopper,final-obs:zero,one,0,boundary,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,final-obs:zero,one,1,boundary,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,final-obs:zero,one,2,boundary,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,log:drop-first,0.01,0,log,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,log:drop-first,0.01,1,log,50,2602.3,0,0,0,1,2268.8,-333.5,da9f3114 +hopper,log:drop-first,0.01,2,log,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,log:drop-first,1.0,0,log,50,3421.52,0,0,0,1,1753.8,-1667.72,13daa014 +hopper,log:drop-first,1.0,1,log,50,2602.3,0,0,0,1,2267.8,-334.5,da9f3114 +hopper,log:drop-first,1.0,2,log,50,1288.72,0,0,0,1,1776.8,488.08,dc7c18bd +hopper,log:drop-first,one,0,log,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,log:drop-first,one,1,log,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,log:drop-first,one,2,log,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,log:short,0.01,0,log,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,log:short,0.01,1,log,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,log:short,0.01,2,log,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,log:short,1.0,0,log,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,log:short,1.0,1,log,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,log:short,1.0,2,log,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,log:short,one,0,log,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,log:short,one,1,log,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,log:short,one,2,log,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,none,1.0,0,control,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,none,1.0,1,control,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,none,1.0,10,control,50,3651.22,0,0,0,1,2190.9,-1460.32,d5a3ae29 +hopper,none,1.0,11,control,50,3558.53,0,0,0,1,2267.8,-1290.73,2308827d +hopper,none,1.0,12,control,50,2739.89,0,0,0,1,1930.3,-809.59,9ee81f6e +hopper,none,1.0,13,control,50,1412.84,0,0,0,1,2218.7,805.86,f48b7b6b +hopper,none,1.0,14,control,50,1706.07,0,0,0,1,1612.8,-93.27,e75667e3 +hopper,none,1.0,15,control,50,2672.23,0,0,0,1,2190.5,-481.73,6e5886cd +hopper,none,1.0,16,control,50,2711.51,0,0,0,1,2271.7,-439.81,af5afed3 +hopper,none,1.0,17,control,50,1703.03,0,0,0,1,1693.8,-9.23,5aff75d6 +hopper,none,1.0,18,control,50,3607.0,0,0,0,1,2190.1,-1416.9,03cbf434 +hopper,none,1.0,19,control,50,3412.04,0,0,0,1,1747.9,-1664.14,53d2bb4d +hopper,none,1.0,2,control,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +hopper,obs:f16,0.01,0,boundary,50,3596.94,0,0,0,1,1790.1,-1806.84,bc35ca61 +hopper,obs:f16,0.01,1,boundary,50,3490.92,0,0,0,1,1636.8,-1854.12,e860d985 +hopper,obs:f16,0.01,2,boundary,50,3437.37,0,0,0,1,1936.8,-1500.57,e25b790c +hopper,obs:f16,1.0,0,boundary,50,1505.25,0,0,0,1,1838.1,332.85,0b88c98a +hopper,obs:f16,1.0,1,boundary,50,3588.14,0,0,0,1,1759.7,-1828.44,a39c8fdc +hopper,obs:f16,1.0,2,boundary,50,3592.49,0,0,0,1,2215.4,-1377.09,992371df +hopper,obs:f16,one,0,boundary,50,3544.22,0,0,0,1,2016.1,-1528.12,53f4d25d +hopper,obs:f16,one,1,boundary,50,1527.57,0,0,0,1,2144.1,616.53,453a803a +hopper,obs:f16,one,2,boundary,50,3590.79,0,0,0,1,1972.1,-1618.69,3a996d59 +hopper,obs:stale,0.01,0,boundary,50,3493.91,0,0,0,1,1789.6,-1704.31,b50d63b7 +hopper,obs:stale,0.01,1,boundary,50,2608.45,0,0,0,1,2312.3,-296.15,b5b8c46a +hopper,obs:stale,0.01,2,boundary,50,3023.5,0,0,0,1,1748.2,-1275.3,b7db0b90 +hopper,obs:stale,1.0,0,boundary,50,2692.35,0,0,0,1,1328.3,-1364.05,5c7c9824 +hopper,obs:stale,1.0,1,boundary,50,942.02,0,1,1,0,991.0,48.98,96c4ca77 +hopper,obs:stale,1.0,2,boundary,50,3365.72,0,0,0,1,1829.8,-1535.92,9543c655 +hopper,obs:stale,one,0,boundary,50,3217.58,0,0,0,1,2036.6,-1180.98,c74f370d +hopper,obs:stale,one,1,boundary,50,3149.61,0,0,0,1,2070.6,-1079.01,50ca29ec +hopper,obs:stale,one,2,boundary,50,3566.91,0,0,0,1,2086.7,-1480.21,41e61933 +hopper,obs:swap,0.01,0,boundary,50,3571.37,0,0,0,1,1632.0,-1939.37,7e429cff +hopper,obs:swap,0.01,1,boundary,50,3240.32,0,0,0,1,1631.3,-1609.02,9bf6d868 +hopper,obs:swap,0.01,2,boundary,50,2433.84,0,0,0,1,1513.4,-920.44,16a1a6b5 +hopper,obs:swap,1.0,0,boundary,50,91.93,1,1,1,0,101.9,9.97,9f5758c0 +hopper,obs:swap,1.0,1,boundary,50,165.63,0,1,1,0,104.7,-60.93,96d218df +hopper,obs:swap,1.0,2,boundary,50,139.06,1,1,1,0,108.6,-30.46,1056980e +hopper,obs:swap,one,0,boundary,50,1473.43,0,0,0,1,1710.4,236.97,d066684e +hopper,obs:swap,one,1,boundary,50,2471.48,0,0,0,1,2184.7,-286.78,ab2dff3d +hopper,obs:swap,one,2,boundary,50,2103.88,0,0,0,1,2227.0,123.12,b8dceb9c +hopper,obs:zero,0.01,0,boundary,50,3582.65,0,0,0,1,1671.7,-1910.95,134d754d +hopper,obs:zero,0.01,1,boundary,50,3251.86,0,0,0,1,1594.3,-1657.56,76045eee +hopper,obs:zero,0.01,2,boundary,50,3512.84,0,0,0,1,1897.6,-1615.24,b905d051 +hopper,obs:zero,1.0,0,boundary,50,76.72,1,1,1,0,102.3,25.58,9299661a +hopper,obs:zero,1.0,1,boundary,50,66.32,1,1,1,0,88.7,22.38,15147da6 +hopper,obs:zero,1.0,2,boundary,50,100.8,1,1,1,0,109.8,9.0,ace7d78f +hopper,obs:zero,one,0,boundary,50,3556.27,0,0,0,1,2073.6,-1482.67,8344511e +hopper,obs:zero,one,1,boundary,50,3549.81,0,0,0,1,1589.9,-1959.91,d7a8bcef +hopper,obs:zero,one,2,boundary,50,3552.54,0,0,0,1,2048.2,-1504.34,1582d021 +hopper,reward:f16,0.01,0,boundary,50,3301.16,0,0,0,1,1749.0,-1552.16,9db6abe8 +hopper,reward:f16,0.01,1,boundary,50,1288.65,0,0,0,1,2003.4,714.75,abe9c4c1 +hopper,reward:f16,0.01,2,boundary,50,3587.59,0,0,0,1,1940.2,-1647.39,71f23cdd +hopper,reward:f16,1.0,0,boundary,50,3489.48,0,0,0,1,1992.5,-1496.98,ff75b00a +hopper,reward:f16,1.0,1,boundary,50,3409.9,0,0,0,1,1905.4,-1504.5,a8b32d4e +hopper,reward:f16,1.0,2,boundary,50,2760.34,0,0,0,1,1737.5,-1022.84,906dcd4e +hopper,reward:f16,one,0,boundary,50,3580.24,0,0,0,1,1951.9,-1628.34,fafbc73d +hopper,reward:f16,one,1,boundary,50,1294.95,0,0,0,1,1948.0,653.05,e06d8d8c +hopper,reward:f16,one,2,boundary,50,3247.54,0,0,0,1,2502.9,-744.64,17ba85af +hopper,reward:stale,0.01,0,boundary,50,1472.94,0,0,0,1,2208.9,735.96,66bd30a5 +hopper,reward:stale,0.01,1,boundary,50,3615.88,0,0,0,1,2220.7,-1395.18,21dbf1f4 +hopper,reward:stale,0.01,2,boundary,50,3240.26,0,0,0,1,2069.5,-1170.76,165d0c7e +hopper,reward:stale,1.0,0,boundary,50,3553.87,0,0,0,1,1985.1,-1568.77,e92ff4da +hopper,reward:stale,1.0,1,boundary,50,2821.33,0,0,0,1,2083.4,-737.93,a01b5fe5 +hopper,reward:stale,1.0,2,boundary,50,3555.24,0,0,0,1,1901.1,-1654.14,625d58d1 +hopper,reward:stale,one,0,boundary,50,3577.29,0,0,0,1,1882.7,-1694.59,c05ce218 +hopper,reward:stale,one,1,boundary,50,3508.21,0,0,0,1,1650.2,-1858.01,7a049739 +hopper,reward:stale,one,2,boundary,50,2847.42,0,0,0,1,1973.7,-873.72,f962b077 +hopper,reward:swap,0.01,0,boundary,50,3464.08,0,0,0,1,1927.1,-1536.98,4d889602 +hopper,reward:swap,0.01,1,boundary,50,1352.15,0,0,0,1,1880.3,528.15,6052ed84 +hopper,reward:swap,0.01,2,boundary,50,3462.04,0,0,0,1,2266.3,-1195.74,38d84a8b +hopper,reward:swap,1.0,0,boundary,50,1010.82,0,1,1,0,1006.1,-4.72,83d234b3 +hopper,reward:swap,1.0,1,boundary,50,1014.82,0,1,1,0,1005.8,-9.02,af362146 +hopper,reward:swap,1.0,2,boundary,50,991.16,0,1,1,0,986.5,-4.66,9d7614b4 +hopper,reward:swap,one,0,boundary,50,3598.38,0,0,0,1,2173.5,-1424.88,40b97b3b +hopper,reward:swap,one,1,boundary,50,1560.18,0,0,0,1,2168.5,608.32,91bd93bb +hopper,reward:swap,one,2,boundary,50,1416.82,0,0,0,1,2075.0,658.18,db70a417 +hopper,reward:zero,0.01,0,boundary,50,1586.35,0,0,0,1,1839.2,252.85,725bef31 +hopper,reward:zero,0.01,1,boundary,50,3516.98,0,0,0,1,1873.7,-1643.28,1827574c +hopper,reward:zero,0.01,2,boundary,50,3412.84,0,0,0,1,1915.9,-1496.94,add9fb50 +hopper,reward:zero,1.0,0,boundary,50,59.64,1,1,1,0,0.0,-59.64,1c1f9d61 +hopper,reward:zero,1.0,1,boundary,50,11.58,1,1,1,0,0.0,-11.58,7b1220df +hopper,reward:zero,1.0,2,boundary,50,2.83,1,1,1,0,0.0,-2.83,60f405bc +hopper,reward:zero,one,0,boundary,50,2497.95,0,0,0,1,1878.4,-619.55,062be833 +hopper,reward:zero,one,1,boundary,50,1817.13,0,0,0,1,1942.9,125.77,a8a2cceb +hopper,reward:zero,one,2,boundary,50,3494.61,0,0,0,1,2202.6,-1292.01,032d0fdb +hopper,step:fill,0.01,0,boundary,50,3367.99,0,0,0,1,1637.7,-1730.29,344f13fc +hopper,step:fill,0.01,1,boundary,50,2505.64,0,0,0,1,2240.5,-265.14,e29fba07 +hopper,step:fill,0.01,2,boundary,50,3283.72,0,0,0,1,1891.4,-1392.32,a93f1571 +hopper,step:fill,1.0,0,boundary,50,5.74,1,1,1,0,,,54dd6b3a +hopper,step:fill,1.0,1,boundary,50,6.13,1,1,1,0,,,f0d9a30d +hopper,step:fill,1.0,2,boundary,50,7.73,1,1,1,0,,,1f3f0052 +hopper,step:fill,one,0,boundary,50,3549.82,0,0,0,1,1800.2,-1749.62,678f362f +hopper,step:fill,one,1,boundary,50,3611.29,0,0,0,1,2080.1,-1531.19,0ad39777 +hopper,step:fill,one,2,boundary,50,3460.54,0,0,0,1,2180.7,-1279.84,dc482495 +hopper,terminated:as-trunc,0.01,0,boundary,50,3519.3,0,0,0,1,1936.0,-1583.3,ed3079b7 +hopper,terminated:as-trunc,0.01,1,boundary,50,2751.81,0,0,0,1,1882.8,-869.01,2370a04f +hopper,terminated:as-trunc,0.01,2,boundary,50,3472.93,0,0,0,1,1903.7,-1569.23,dfd89e43 +hopper,terminated:as-trunc,1.0,0,boundary,50,764.94,0,1,1,0,459.8,-305.14,5ec55c9b +hopper,terminated:as-trunc,1.0,1,boundary,50,786.16,0,1,1,0,632.5,-153.66,7854334f +hopper,terminated:as-trunc,1.0,2,boundary,50,833.02,0,1,1,0,618.6,-214.42,e6715198 +hopper,terminated:as-trunc,one,0,boundary,50,2259.45,0,0,0,1,2099.4,-160.05,cd28dc41 +hopper,terminated:as-trunc,one,1,boundary,50,3586.09,0,0,0,1,2245.5,-1340.59,df3b7dea +hopper,terminated:as-trunc,one,2,boundary,50,3338.22,0,0,0,1,1981.1,-1357.12,0301697f +hopper,terminated:drop,0.01,0,boundary,50,2113.56,0,0,0,1,1673.5,-440.06,a7f1ba6f +hopper,terminated:drop,0.01,1,boundary,50,688.58,0,0,0,1,1557.3,868.72,c0b1970d +hopper,terminated:drop,0.01,2,boundary,50,3321.4,0,0,0,1,1717.6,-1603.8,c77d5ed8 +hopper,terminated:drop,1.0,0,boundary,50,678.15,0,1,1,0,,,8589d941 +hopper,terminated:drop,1.0,1,boundary,50,730.15,0,1,1,0,,,74024aca +hopper,terminated:drop,1.0,2,boundary,50,679.19,0,1,1,0,,,af6d0f6d +hopper,terminated:drop,one,0,boundary,50,3553.62,0,0,0,1,2057.9,-1495.72,abff6921 +hopper,terminated:drop,one,1,boundary,50,2874.39,0,0,0,1,1874.8,-999.59,fd6db757 +hopper,terminated:drop,one,2,boundary,50,2493.64,0,0,0,1,2017.5,-476.14,e1d130e1 +hopper,terminated:spurious,0.01,0,boundary,50,838.11,0,1,1,0,223.1,-615.01,f2aaa548 +hopper,terminated:spurious,0.01,1,boundary,50,552.69,0,1,1,0,197.4,-355.29,beffce16 +hopper,terminated:spurious,0.01,2,boundary,50,554.4,0,1,1,0,214.6,-339.8,7ffeb15d +hopper,terminated:spurious,1.0,0,boundary,50,39.83,1,1,1,0,2.0,-37.83,19c835d8 +hopper,terminated:spurious,1.0,1,boundary,50,178.43,0,1,1,0,2.2,-176.23,bc17908b +hopper,terminated:spurious,1.0,2,boundary,50,91.65,1,1,1,0,2.0,-89.65,bc723cdc +hopper,terminated:spurious,one,0,boundary,50,3563.88,0,0,0,1,2258.9,-1304.98,f89379d8 +hopper,terminated:spurious,one,1,boundary,50,1476.21,0,0,0,1,1999.2,522.99,9d4e2b0a +hopper,terminated:spurious,one,2,boundary,50,3521.63,0,0,0,1,2063.2,-1458.43,550d977f +hopper,truncated:as-term,0.01,0,boundary,50,1170.23,0,0,0,1,2332.7,1162.47,fc2b4c5a +hopper,truncated:as-term,0.01,1,boundary,50,1748.97,0,0,0,1,1633.4,-115.57,d658d08b +hopper,truncated:as-term,0.01,2,boundary,50,3109.6,0,0,0,1,2065.3,-1044.3,7ea8931e +hopper,truncated:as-term,1.0,0,boundary,50,3443.72,0,0,0,1,1817.8,-1625.92,6cc70f32 +hopper,truncated:as-term,1.0,1,boundary,50,1744.75,0,0,0,1,2014.9,270.15,6165442f +hopper,truncated:as-term,1.0,2,boundary,50,3259.94,0,0,0,1,2006.0,-1253.94,2d622c59 +hopper,truncated:as-term,one,0,boundary,50,3611.32,0,0,0,1,1972.5,-1638.82,c0c310c6 +hopper,truncated:as-term,one,1,boundary,50,3232.66,0,0,0,1,2080.9,-1151.76,a3ef8fa9 +hopper,truncated:as-term,one,2,boundary,50,3033.63,0,0,0,1,1904.1,-1129.53,4044fb86 +hopper,truncated:drop,0.01,0,boundary,50,3384.78,0,0,0,1,1866.3,-1518.48,339bf434 +hopper,truncated:drop,0.01,1,boundary,50,2692.73,0,0,0,1,2063.4,-629.33,abe45244 +hopper,truncated:drop,0.01,2,boundary,50,1529.53,0,0,0,1,1784.8,255.27,61c5468c +hopper,truncated:drop,1.0,0,boundary,50,1334.83,0,0,0,1,2167.4,832.57,43393fb8 +hopper,truncated:drop,1.0,1,boundary,50,3449.78,0,0,0,1,1864.0,-1585.78,1497fee0 +hopper,truncated:drop,1.0,2,boundary,50,1656.91,0,0,0,1,1599.4,-57.51,00272617 +hopper,truncated:drop,one,0,boundary,50,3552.05,0,0,0,1,1722.5,-1829.55,dcfe43fe +hopper,truncated:drop,one,1,boundary,50,1985.88,0,0,0,1,1852.9,-132.98,dc04fce5 +hopper,truncated:drop,one,2,boundary,50,3546.73,0,0,0,1,1973.7,-1573.03,655e7475 +hopper,wire-float64,1.0,0,control,50,3421.52,0,0,0,1,1754.8,-1666.72,13daa014 +hopper,wire-float64,1.0,1,control,50,2602.3,0,0,0,1,2268.9,-333.4,da9f3114 +hopper,wire-float64,1.0,2,control,50,1288.72,0,0,0,1,1777.8,489.08,dc7c18bd +pendulum,action:clip,0.001,0,boundary,50,-169.32,0,0,1,0,-179.5,-10.18,c96a0e3a +pendulum,action:clip,0.001,1,boundary,50,-161.02,0,0,0,1,-197.6,-36.58,5f35479a +pendulum,action:clip,0.001,2,boundary,50,-169.42,0,0,0,1,-210.5,-41.08,5693bb59 +pendulum,action:clip,0.01,0,boundary,50,-175.08,0,0,0,1,-227.8,-52.72,96d41f53 +pendulum,action:clip,0.01,1,boundary,50,-170.54,0,0,1,0,-183.0,-12.46,f6c883c2 +pendulum,action:clip,0.01,2,boundary,50,-171.79,0,0,0,1,-229.0,-57.21,d6eebc9c +pendulum,action:clip,0.1,0,boundary,50,-171.67,0,0,0,1,-190.3,-18.63,02b0b664 +pendulum,action:clip,0.1,1,boundary,50,-165.21,0,0,0,1,-190.8,-25.59,7c24b984 +pendulum,action:clip,0.1,2,boundary,50,-169.16,0,0,0,1,-218.7,-49.54,bfdc084c +pendulum,action:clip,1.0,0,boundary,50,-169.23,0,0,1,0,-369.2,-199.97,9c89fd2e +pendulum,action:clip,1.0,1,boundary,50,-169.34,0,0,1,0,-362.0,-192.66,319f69be +pendulum,action:clip,1.0,2,boundary,50,-165.68,0,0,1,0,-361.2,-195.52,f19dd117 +pendulum,action:clip,one,0,boundary,50,-169.04,0,0,0,1,-187.9,-18.86,46222c2a +pendulum,action:clip,one,1,boundary,50,-169.76,0,0,0,1,-213.9,-44.14,ba4ba4cc +pendulum,action:clip,one,2,boundary,50,-176.32,0,0,0,1,-196.7,-20.38,6ccbd988 +pendulum,action:f16,0.001,0,boundary,50,-175.21,0,0,0,1,-194.1,-18.89,be8c6c8d +pendulum,action:f16,0.001,1,boundary,50,-169.9,0,0,0,1,-192.2,-22.3,1afff977 +pendulum,action:f16,0.001,2,boundary,50,-168.72,0,0,0,1,-208.5,-39.78,fc41ad45 +pendulum,action:f16,0.01,0,boundary,50,-178.0,0,0,0,1,-197.7,-19.7,0b05a50a +pendulum,action:f16,0.01,1,boundary,50,-182.94,0,0,0,1,-192.7,-9.76,b4f02fd1 +pendulum,action:f16,0.01,2,boundary,50,-165.45,0,0,0,1,-195.5,-30.05,d0384b0f +pendulum,action:f16,0.1,0,boundary,50,-172.61,0,0,0,1,-211.0,-38.39,ef66636a +pendulum,action:f16,0.1,1,boundary,50,-175.67,0,0,0,1,-185.4,-9.73,140e788c +pendulum,action:f16,0.1,2,boundary,50,-170.21,0,0,0,1,-193.6,-23.39,c36f4bc1 +pendulum,action:f16,1.0,0,boundary,50,-166.49,0,0,0,1,-194.0,-27.51,9f5f03a4 +pendulum,action:f16,1.0,1,boundary,50,-163.98,0,0,1,0,-177.8,-13.82,490d8a3f +pendulum,action:f16,1.0,2,boundary,50,-173.55,0,0,0,1,-192.1,-18.55,42e1ffc3 +pendulum,action:f16,one,0,boundary,50,-197.05,0,0,0,1,-211.6,-14.55,c86def24 +pendulum,action:f16,one,1,boundary,50,-173.99,0,0,0,1,-187.6,-13.61,93384ed4 +pendulum,action:f16,one,2,boundary,50,-173.49,0,0,0,1,-201.6,-28.11,32901a88 +pendulum,action:stale,0.001,0,boundary,50,-167.1,0,0,0,1,-199.9,-32.8,e1ae03c1 +pendulum,action:stale,0.001,1,boundary,50,-312.31,1,0,1,0,-353.4,-41.09,610b5cb5 +pendulum,action:stale,0.001,2,boundary,50,-166.9,0,0,0,1,-193.2,-26.3,c1adab4b +pendulum,action:stale,0.01,0,boundary,50,-168.83,0,0,0,1,-187.1,-18.27,523f9fdb +pendulum,action:stale,0.01,1,boundary,50,-171.12,0,0,0,1,-214.2,-43.08,a5ebc8a2 +pendulum,action:stale,0.01,2,boundary,50,-165.96,0,0,0,1,-195.0,-29.04,02583893 +pendulum,action:stale,0.1,0,boundary,50,-160.83,0,0,0,1,-197.7,-36.87,d6ae8045 +pendulum,action:stale,0.1,1,boundary,50,-176.11,0,0,0,1,-226.8,-50.69,fefa8cca +pendulum,action:stale,0.1,2,boundary,50,-171.65,0,0,0,1,-199.6,-27.95,5e58606e +pendulum,action:stale,1.0,0,boundary,50,-167.03,0,0,0,1,-193.6,-26.57,bcbbc6fc +pendulum,action:stale,1.0,1,boundary,50,-157.9,0,0,1,0,-175.8,-17.9,86d9270e +pendulum,action:stale,1.0,2,boundary,50,-159.02,0,0,1,0,-181.2,-22.18,fb489c27 +pendulum,action:stale,one,0,boundary,50,-171.76,0,0,0,1,-205.2,-33.44,aa1a75e2 +pendulum,action:stale,one,1,boundary,50,-177.72,0,0,0,1,-187.9,-10.18,fd625f6e +pendulum,action:stale,one,2,boundary,50,-160.92,0,0,0,1,-204.1,-43.18,d00464b8 +pendulum,action:swap,0.001,0,boundary,50,-177.36,0,0,0,1,-198.8,-21.44,963db528 +pendulum,action:swap,0.001,1,boundary,50,-165.98,0,0,0,1,-186.1,-20.12,74b5aa39 +pendulum,action:swap,0.001,2,boundary,50,-212.85,0,0,0,1,-196.8,16.05,fb249d01 +pendulum,action:swap,0.01,0,boundary,50,-163.78,0,0,0,1,-196.3,-32.52,e5ad4cab +pendulum,action:swap,0.01,1,boundary,50,-172.17,0,0,0,1,-214.4,-42.23,27dd148a +pendulum,action:swap,0.01,2,boundary,50,-172.13,0,0,0,1,-208.3,-36.17,a87eaf38 +pendulum,action:swap,0.1,0,boundary,50,-165.44,0,0,0,1,-241.7,-76.26,0bd3b36f +pendulum,action:swap,0.1,1,boundary,50,-170.91,0,0,1,0,-263.2,-92.29,6a01345d +pendulum,action:swap,0.1,2,boundary,50,-166.85,0,0,0,1,-242.2,-75.35,25533d12 +pendulum,action:swap,1.0,0,boundary,50,-1171.18,1,1,1,0,-1244.2,-73.02,889766d5 +pendulum,action:swap,1.0,1,boundary,50,-1609.28,1,1,1,0,-1223.7,385.58,fe95f321 +pendulum,action:swap,1.0,2,boundary,50,-1351.25,1,1,1,0,-1231.5,119.75,93086fa0 +pendulum,action:swap,one,0,boundary,50,-198.43,0,0,0,1,-192.6,5.83,9b3b2cd8 +pendulum,action:swap,one,1,boundary,50,-169.41,0,0,0,1,-194.8,-25.39,c5240226 +pendulum,action:swap,one,2,boundary,50,-161.11,0,0,0,1,-193.6,-32.49,7a44221a +pendulum,action:zero,0.001,0,boundary,50,-197.11,0,0,0,1,-223.2,-26.09,f026739b +pendulum,action:zero,0.001,1,boundary,50,-159.01,0,0,0,1,-190.3,-31.29,fb252878 +pendulum,action:zero,0.001,2,boundary,50,-169.76,0,0,0,1,-187.7,-17.94,bd98a512 +pendulum,action:zero,0.01,0,boundary,50,-175.92,0,0,0,1,-209.8,-33.88,ce2f2d84 +pendulum,action:zero,0.01,1,boundary,50,-166.1,0,0,0,1,-195.1,-29.0,414ad5c5 +pendulum,action:zero,0.01,2,boundary,50,-193.72,0,0,0,1,-195.7,-1.98,32a19fe7 +pendulum,action:zero,0.1,0,boundary,50,-164.86,0,0,0,1,-225.3,-60.44,e7d84007 +pendulum,action:zero,0.1,1,boundary,50,-168.55,0,0,0,1,-212.8,-44.25,f1c7df11 +pendulum,action:zero,0.1,2,boundary,50,-155.82,0,0,0,1,-212.5,-56.68,d6feb38a +pendulum,action:zero,1.0,0,boundary,50,-1559.43,1,1,1,0,-1214.6,344.83,fc69c04c +pendulum,action:zero,1.0,1,boundary,50,-1368.29,1,1,1,0,-1219.7,148.59,b2feb45b +pendulum,action:zero,1.0,2,boundary,50,-1344.58,1,1,1,0,-1232.7,111.88,9fd2c154 +pendulum,action:zero,one,0,boundary,50,-178.23,0,0,0,1,-214.8,-36.57,f4f801a9 +pendulum,action:zero,one,1,boundary,50,-167.85,0,0,0,1,-199.0,-31.15,476aadb0 +pendulum,action:zero,one,2,boundary,50,-165.27,0,0,0,1,-203.0,-37.73,2d5728a9 +pendulum,delay-1ms,1.0,0,control,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,delay-1ms,1.0,1,control,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,delay-1ms,1.0,2,control,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,final-obs:f16,0.001,0,boundary,50,-196.59,0,0,0,1,-200.7,-4.11,e34029d3 +pendulum,final-obs:f16,0.001,1,boundary,50,-169.35,0,0,0,1,-205.7,-36.35,d81de110 +pendulum,final-obs:f16,0.001,2,boundary,50,-170.96,0,0,0,1,-210.7,-39.74,21f35a0f +pendulum,final-obs:f16,0.01,0,boundary,50,-169.87,0,0,0,1,-234.5,-64.63,8b050884 +pendulum,final-obs:f16,0.01,1,boundary,50,-170.04,0,0,0,1,-189.3,-19.26,d1fea845 +pendulum,final-obs:f16,0.01,2,boundary,50,-243.8,0,0,0,1,-208.8,35.0,8d4f5d81 +pendulum,final-obs:f16,0.1,0,boundary,50,-332.87,1,0,0,1,-211.3,121.57,b890ca5d +pendulum,final-obs:f16,0.1,1,boundary,50,-169.71,0,0,0,1,-191.8,-22.09,98d60d01 +pendulum,final-obs:f16,0.1,2,boundary,50,-164.7,0,0,0,1,-193.0,-28.3,927524ca +pendulum,final-obs:f16,1.0,0,boundary,50,-172.55,0,0,0,1,-191.5,-18.95,3aaf60a0 +pendulum,final-obs:f16,1.0,1,boundary,50,-170.06,0,0,0,1,-186.7,-16.64,209d2f10 +pendulum,final-obs:f16,1.0,2,boundary,50,-162.68,0,0,0,1,-185.3,-22.62,3b34c6f2 +pendulum,final-obs:f16,one,0,boundary,50,-170.23,0,0,0,1,-199.3,-29.07,bf6e41a3 +pendulum,final-obs:f16,one,1,boundary,50,-168.31,0,0,0,1,-196.6,-28.29,18d8ddf9 +pendulum,final-obs:f16,one,2,boundary,50,-201.36,0,0,0,1,-202.5,-1.14,ab939388 +pendulum,final-obs:stale,0.001,0,boundary,50,-197.61,0,0,0,1,-203.3,-5.69,b2025182 +pendulum,final-obs:stale,0.001,1,boundary,50,-170.26,0,0,0,1,-204.9,-34.64,43e6e91f +pendulum,final-obs:stale,0.001,2,boundary,50,-166.36,0,0,0,1,-194.3,-27.94,02a0ebc8 +pendulum,final-obs:stale,0.01,0,boundary,50,-174.9,0,0,0,1,-214.0,-39.1,a059213f +pendulum,final-obs:stale,0.01,1,boundary,50,-194.51,0,0,0,1,-194.7,-0.19,f99f790f +pendulum,final-obs:stale,0.01,2,boundary,50,-180.21,0,0,0,1,-228.3,-48.09,e3e423de +pendulum,final-obs:stale,0.1,0,boundary,50,-170.67,0,0,1,0,-182.6,-11.93,4e044c07 +pendulum,final-obs:stale,0.1,1,boundary,50,-166.36,0,0,0,1,-201.1,-34.74,4611c4b7 +pendulum,final-obs:stale,0.1,2,boundary,50,-169.17,0,0,0,1,-205.8,-36.63,8a5819d6 +pendulum,final-obs:stale,1.0,0,boundary,50,-176.08,0,0,1,0,-260.8,-84.72,114f86a0 +pendulum,final-obs:stale,1.0,1,boundary,50,-172.84,0,0,0,1,-206.3,-33.46,2a99cc39 +pendulum,final-obs:stale,1.0,2,boundary,50,-168.28,0,0,0,1,-201.2,-32.92,64ddd2f4 +pendulum,final-obs:stale,one,0,boundary,50,-182.41,0,0,0,1,-203.0,-20.59,2cd3a46f +pendulum,final-obs:stale,one,1,boundary,50,-169.44,0,0,0,1,-190.0,-20.56,3d2b49b0 +pendulum,final-obs:stale,one,2,boundary,50,-171.66,0,0,0,1,-193.4,-21.74,9080a4bf +pendulum,final-obs:swap,0.001,0,boundary,50,-169.88,0,0,0,1,-198.6,-28.72,813800a8 +pendulum,final-obs:swap,0.001,1,boundary,50,-170.32,0,0,0,1,-205.8,-35.48,e3cb3e06 +pendulum,final-obs:swap,0.001,2,boundary,50,-165.98,0,0,0,1,-203.4,-37.42,50029099 +pendulum,final-obs:swap,0.01,0,boundary,50,-171.23,0,0,0,1,-198.7,-27.47,5bd0a6b2 +pendulum,final-obs:swap,0.01,1,boundary,50,-243.35,0,0,0,1,-199.0,44.35,c97adf0d +pendulum,final-obs:swap,0.01,2,boundary,50,-164.04,0,0,0,1,-200.6,-36.56,56c13906 +pendulum,final-obs:swap,0.1,0,boundary,50,-170.24,0,0,0,1,-209.1,-38.86,6b23376c +pendulum,final-obs:swap,0.1,1,boundary,50,-160.97,0,0,0,1,-193.7,-32.73,95344a14 +pendulum,final-obs:swap,0.1,2,boundary,50,-171.24,0,0,0,1,-205.2,-33.96,9645d7b2 +pendulum,final-obs:swap,1.0,0,boundary,50,-174.2,0,0,0,1,-212.6,-38.4,3f8956ed +pendulum,final-obs:swap,1.0,1,boundary,50,-175.43,0,0,0,1,-203.2,-27.77,c81ec51a +pendulum,final-obs:swap,1.0,2,boundary,50,-183.57,0,0,0,1,-202.0,-18.43,1b6b79fd +pendulum,final-obs:swap,one,0,boundary,50,-170.05,0,0,0,1,-189.5,-19.45,4e8492f7 +pendulum,final-obs:swap,one,1,boundary,50,-182.45,0,0,0,1,-198.7,-16.25,a85f2e22 +pendulum,final-obs:swap,one,2,boundary,50,-171.92,0,0,0,1,-191.0,-19.08,8f41be4c +pendulum,final-obs:zero,0.001,0,boundary,50,-171.5,0,0,0,1,-207.9,-36.4,52f5a966 +pendulum,final-obs:zero,0.001,1,boundary,50,-168.12,0,0,0,1,-204.8,-36.68,da4ee014 +pendulum,final-obs:zero,0.001,2,boundary,50,-167.45,0,0,0,1,-193.9,-26.45,97d053b1 +pendulum,final-obs:zero,0.01,0,boundary,50,-168.97,0,0,0,1,-188.6,-19.63,010adf87 +pendulum,final-obs:zero,0.01,1,boundary,50,-166.83,0,0,0,1,-217.2,-50.37,fcdec6da +pendulum,final-obs:zero,0.01,2,boundary,50,-239.71,0,0,0,1,-239.8,-0.09,4e34a328 +pendulum,final-obs:zero,0.1,0,boundary,50,-171.77,0,0,0,1,-216.6,-44.83,352a0dfc +pendulum,final-obs:zero,0.1,1,boundary,50,-296.76,1,0,1,0,-337.4,-40.64,b6bda6f1 +pendulum,final-obs:zero,0.1,2,boundary,50,-176.85,0,0,0,1,-218.2,-41.35,72354b6f +pendulum,final-obs:zero,1.0,0,boundary,50,-242.44,0,0,1,0,-313.6,-71.16,efd619dc +pendulum,final-obs:zero,1.0,1,boundary,50,-169.96,0,0,1,0,-253.0,-83.04,b56b40ae +pendulum,final-obs:zero,1.0,2,boundary,50,-173.29,0,0,1,0,-258.0,-84.71,226ab780 +pendulum,final-obs:zero,one,0,boundary,50,-198.23,0,0,0,1,-207.0,-8.77,bf033c56 +pendulum,final-obs:zero,one,1,boundary,50,-175.9,0,0,0,1,-202.5,-26.6,d0314318 +pendulum,final-obs:zero,one,2,boundary,50,-176.83,0,0,0,1,-216.0,-39.17,81a24d76 +pendulum,indices:reorder,one,0,loud,50,-172.59,0,0,,0,,,5333d4e3 +pendulum,indices:reorder,one,1,loud,50,-174.52,0,0,,0,,,afaa8757 +pendulum,indices:reorder,one,2,loud,50,-170.24,0,0,,0,,,fde67d33 +pendulum,log:drop-first,0.001,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:drop-first,0.001,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:drop-first,0.001,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:drop-first,0.01,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:drop-first,0.01,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:drop-first,0.01,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:drop-first,0.1,0,log,50,-200.52,0,0,0,1,-199.5,1.02,ec5b8226 +pendulum,log:drop-first,0.1,1,log,50,-174.6,0,0,0,1,-208.4,-33.8,062e92e3 +pendulum,log:drop-first,0.1,2,log,50,-176.0,0,0,0,1,-245.7,-69.7,3a8d968f +pendulum,log:drop-first,1.0,0,log,50,-200.52,0,0,0,1,-196.5,4.02,ec5b8226 +pendulum,log:drop-first,1.0,1,log,50,-174.6,0,0,0,1,-205.4,-30.8,062e92e3 +pendulum,log:drop-first,1.0,2,log,50,-176.0,0,0,0,1,-242.6,-66.6,3a8d968f +pendulum,log:drop-first,one,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:drop-first,one,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:drop-first,one,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:short,0.001,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:short,0.001,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:short,0.001,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:short,0.01,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:short,0.01,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:short,0.01,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:short,0.1,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:short,0.1,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:short,0.1,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:short,1.0,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:short,1.0,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:short,1.0,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,log:short,one,0,log,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,log:short,one,1,log,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,log:short,one,2,log,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,none,1.0,0,control,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,none,1.0,1,control,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,none,1.0,10,control,50,-170.02,0,0,0,1,-226.2,-56.18,4b5c8f2d +pendulum,none,1.0,11,control,50,-167.96,0,0,0,1,-210.7,-42.74,648cc27f +pendulum,none,1.0,12,control,50,-171.22,0,0,0,1,-215.4,-44.18,f60c2bd7 +pendulum,none,1.0,13,control,50,-238.31,0,0,0,1,-205.0,33.31,c11211ed +pendulum,none,1.0,14,control,50,-169.36,0,0,0,1,-225.4,-56.04,c69e3e42 +pendulum,none,1.0,15,control,50,-173.97,0,0,0,1,-221.9,-47.93,9e7baf1a +pendulum,none,1.0,16,control,50,-169.78,0,0,0,1,-205.0,-35.22,b4dafc53 +pendulum,none,1.0,17,control,50,-170.5,0,0,0,1,-234.7,-64.2,ae3c8039 +pendulum,none,1.0,18,control,50,-221.1,0,0,0,1,-200.5,20.6,af2a0223 +pendulum,none,1.0,19,control,50,-206.5,0,0,0,1,-222.0,-15.5,e1b7ed5c +pendulum,none,1.0,2,control,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f +pendulum,obs:f16,0.001,0,boundary,50,-177.27,0,0,0,1,-213.1,-35.83,99628584 +pendulum,obs:f16,0.001,1,boundary,50,-196.82,0,0,0,1,-193.9,2.92,ae53323b +pendulum,obs:f16,0.001,2,boundary,50,-162.12,0,0,0,1,-191.9,-29.78,f66f3be5 +pendulum,obs:f16,0.01,0,boundary,50,-177.81,0,0,0,1,-216.6,-38.79,8cdce0a6 +pendulum,obs:f16,0.01,1,boundary,50,-167.26,0,0,0,1,-191.9,-24.64,9efc34bd +pendulum,obs:f16,0.01,2,boundary,50,-166.67,0,0,0,1,-184.8,-18.13,a291b401 +pendulum,obs:f16,0.1,0,boundary,50,-175.38,0,0,0,1,-198.6,-23.22,e705e552 +pendulum,obs:f16,0.1,1,boundary,50,-176.53,0,0,0,1,-195.5,-18.97,458c24ce +pendulum,obs:f16,0.1,2,boundary,50,-163.82,0,0,1,0,-259.5,-95.68,67315f13 +pendulum,obs:f16,1.0,0,boundary,50,-169.95,0,0,0,1,-200.7,-30.75,6bd93edb +pendulum,obs:f16,1.0,1,boundary,50,-168.3,0,0,0,1,-194.5,-26.2,db006462 +pendulum,obs:f16,1.0,2,boundary,50,-167.1,0,0,0,1,-214.1,-47.0,0655bd23 +pendulum,obs:f16,one,0,boundary,50,-173.34,0,0,0,1,-220.9,-47.56,d1484e30 +pendulum,obs:f16,one,1,boundary,50,-167.63,0,0,0,1,-191.3,-23.67,7809451b +pendulum,obs:f16,one,2,boundary,50,-165.24,0,0,0,1,-193.1,-27.86,3026d8df +pendulum,obs:stale,0.001,0,boundary,50,-236.06,0,0,0,1,-196.4,39.66,229c165b +pendulum,obs:stale,0.001,1,boundary,50,-168.2,0,0,0,1,-189.4,-21.2,f75dec8d +pendulum,obs:stale,0.001,2,boundary,50,-174.27,0,0,0,1,-207.8,-33.53,17980b93 +pendulum,obs:stale,0.01,0,boundary,50,-161.93,0,0,0,1,-201.1,-39.17,0cbb3296 +pendulum,obs:stale,0.01,1,boundary,50,-172.37,0,0,0,1,-208.3,-35.93,2a0550ff +pendulum,obs:stale,0.01,2,boundary,50,-162.07,0,0,0,1,-191.2,-29.13,08b18278 +pendulum,obs:stale,0.1,0,boundary,50,-277.48,1,0,0,1,-207.2,70.28,cf301c7d +pendulum,obs:stale,0.1,1,boundary,50,-175.62,0,0,0,1,-218.8,-43.18,c99edca5 +pendulum,obs:stale,0.1,2,boundary,50,-187.83,0,0,0,1,-197.2,-9.37,9a8442c4 +pendulum,obs:stale,1.0,0,boundary,50,-249.92,0,0,1,0,-270.1,-20.18,bd9261b4 +pendulum,obs:stale,1.0,1,boundary,50,-162.11,0,0,0,1,-187.7,-25.59,3b8e0f67 +pendulum,obs:stale,1.0,2,boundary,50,-163.33,0,0,0,1,-186.1,-22.77,b2634e34 +pendulum,obs:stale,one,0,boundary,50,-172.45,0,0,0,1,-232.6,-60.15,765a1ee0 +pendulum,obs:stale,one,1,boundary,50,-429.93,1,0,0,1,-238.2,191.73,12613a2a +pendulum,obs:stale,one,2,boundary,50,-242.29,0,0,0,1,-200.6,41.69,978b430d +pendulum,obs:swap,0.001,0,boundary,50,-170.95,0,0,0,1,-210.6,-39.65,9ca3c02d +pendulum,obs:swap,0.001,1,boundary,50,-170.57,0,0,0,1,-185.3,-14.73,df38b2d7 +pendulum,obs:swap,0.001,2,boundary,50,-165.61,0,0,0,1,-192.2,-26.59,8126a1aa +pendulum,obs:swap,0.01,0,boundary,50,-238.74,0,0,0,1,-200.0,38.74,f6cb32fa +pendulum,obs:swap,0.01,1,boundary,50,-505.75,1,0,0,1,-212.2,293.55,deeb971c +pendulum,obs:swap,0.01,2,boundary,50,-175.66,0,0,0,1,-244.6,-68.94,407d9729 +pendulum,obs:swap,0.1,0,boundary,50,-247.51,0,0,1,0,-433.1,-185.59,0c30d1d9 +pendulum,obs:swap,0.1,1,boundary,50,-168.74,0,0,1,0,-325.5,-156.76,23e149f6 +pendulum,obs:swap,0.1,2,boundary,50,-165.27,0,0,1,0,-435.5,-270.23,94257467 +pendulum,obs:swap,1.0,0,boundary,50,-1172.18,1,1,1,0,-1217.5,-45.32,516c7ce6 +pendulum,obs:swap,1.0,1,boundary,50,-1613.51,1,1,1,0,-1219.5,394.01,ee7dcd35 +pendulum,obs:swap,1.0,2,boundary,50,-1343.89,1,1,1,0,-1214.5,129.39,220ba53a +pendulum,obs:swap,one,0,boundary,50,-173.7,0,0,0,1,-208.8,-35.1,3fb58b16 +pendulum,obs:swap,one,1,boundary,50,-165.06,0,0,0,1,-186.8,-21.74,9e1f81cd +pendulum,obs:swap,one,2,boundary,50,-169.21,0,0,0,1,-206.9,-37.69,bfb37b1a +pendulum,obs:zero,0.001,0,boundary,50,-172.31,0,0,0,1,-232.8,-60.49,aa6bbfc1 +pendulum,obs:zero,0.001,1,boundary,50,-169.75,0,0,0,1,-233.2,-63.45,dc412791 +pendulum,obs:zero,0.001,2,boundary,50,-175.37,0,0,0,1,-216.1,-40.73,e50671f5 +pendulum,obs:zero,0.01,0,boundary,50,-175.99,0,0,0,1,-230.2,-54.21,ed34c167 +pendulum,obs:zero,0.01,1,boundary,50,-166.28,0,0,0,1,-216.9,-50.62,3f052f94 +pendulum,obs:zero,0.01,2,boundary,50,-175.74,0,0,0,1,-210.3,-34.56,7b02c343 +pendulum,obs:zero,0.1,0,boundary,50,-221.4,0,0,1,0,-495.1,-273.7,d1791e56 +pendulum,obs:zero,0.1,1,boundary,50,-171.92,0,0,1,0,-331.5,-159.58,fc9a9d60 +pendulum,obs:zero,0.1,2,boundary,50,-229.19,0,0,1,0,-679.8,-450.61,085b1477 +pendulum,obs:zero,1.0,0,boundary,50,-1183.23,1,1,1,0,-1214.6,-31.37,6f23c454 +pendulum,obs:zero,1.0,1,boundary,50,-1181.59,1,1,1,0,-1219.7,-38.11,bfbe3209 +pendulum,obs:zero,1.0,2,boundary,50,-1176.68,1,1,1,0,-1232.7,-56.02,e1dbe226 +pendulum,obs:zero,one,0,boundary,50,-176.8,0,0,0,1,-193.3,-16.5,e2f06a8f +pendulum,obs:zero,one,1,boundary,50,-164.23,0,0,0,1,-197.4,-33.17,9be2e057 +pendulum,obs:zero,one,2,boundary,50,-170.49,0,0,0,1,-190.7,-20.21,2f7f6acc +pendulum,reward:f16,0.001,0,boundary,50,-170.95,0,0,0,1,-229.2,-58.25,52cf7ecc +pendulum,reward:f16,0.001,1,boundary,50,-167.84,0,0,0,1,-189.5,-21.66,91c2a04e +pendulum,reward:f16,0.001,2,boundary,50,-744.3,1,0,0,1,-222.1,522.2,20a63eaf +pendulum,reward:f16,0.01,0,boundary,50,-168.5,0,0,0,1,-200.8,-32.3,957547ea +pendulum,reward:f16,0.01,1,boundary,50,-165.19,0,0,0,1,-188.9,-23.71,ff01fa2a +pendulum,reward:f16,0.01,2,boundary,50,-172.65,0,0,0,1,-207.3,-34.65,1404abe1 +pendulum,reward:f16,0.1,0,boundary,50,-174.49,0,0,0,1,-208.7,-34.21,761d3243 +pendulum,reward:f16,0.1,1,boundary,50,-196.39,0,0,0,1,-184.4,11.99,b5be10fa +pendulum,reward:f16,0.1,2,boundary,50,-186.66,0,0,0,1,-190.4,-3.74,6c4077d0 +pendulum,reward:f16,1.0,0,boundary,50,-165.98,0,0,0,1,-205.8,-39.82,1d482f24 +pendulum,reward:f16,1.0,1,boundary,50,-170.76,0,0,0,1,-246.8,-76.04,add3c078 +pendulum,reward:f16,1.0,2,boundary,50,-167.96,0,0,0,1,-205.1,-37.14,c6599770 +pendulum,reward:f16,one,0,boundary,50,-171.98,0,0,0,1,-212.2,-40.22,e95cad4d +pendulum,reward:f16,one,1,boundary,50,-182.0,0,0,0,1,-205.3,-23.3,833fe81c +pendulum,reward:f16,one,2,boundary,50,-169.31,0,0,0,1,-198.1,-28.79,9bbbf32e +pendulum,reward:short-array,one,0,loud,50,-172.59,0,0,,0,,,5333d4e3 +pendulum,reward:short-array,one,1,loud,50,-174.52,0,0,,0,,,afaa8757 +pendulum,reward:short-array,one,2,loud,50,-170.24,0,0,,0,,,fde67d33 +pendulum,reward:stale,0.001,0,boundary,50,-171.96,0,0,0,1,-212.3,-40.34,bdaacf4d +pendulum,reward:stale,0.001,1,boundary,50,-170.8,0,0,0,1,-183.8,-13.0,39c8b7d4 +pendulum,reward:stale,0.001,2,boundary,50,-169.73,0,0,0,1,-205.8,-36.07,55e6f6cb +pendulum,reward:stale,0.01,0,boundary,50,-166.21,0,0,0,1,-201.5,-35.29,c3ecfd3a +pendulum,reward:stale,0.01,1,boundary,50,-195.35,0,0,0,1,-186.4,8.95,3d6114f7 +pendulum,reward:stale,0.01,2,boundary,50,-169.82,0,0,0,1,-202.1,-32.28,70133a54 +pendulum,reward:stale,0.1,0,boundary,50,-166.57,0,0,0,1,-200.3,-33.73,54a9cdf1 +pendulum,reward:stale,0.1,1,boundary,50,-167.87,0,0,0,1,-207.0,-39.13,98883dd9 +pendulum,reward:stale,0.1,2,boundary,50,-166.39,0,0,0,1,-203.3,-36.91,d1846c72 +pendulum,reward:stale,1.0,0,boundary,50,-171.52,0,0,0,1,-201.7,-30.18,982f942a +pendulum,reward:stale,1.0,1,boundary,50,-170.05,0,0,0,1,-187.4,-17.35,0cddc6e3 +pendulum,reward:stale,1.0,2,boundary,50,-166.97,0,0,0,1,-191.7,-24.73,87c6042c +pendulum,reward:stale,one,0,boundary,50,-199.7,0,0,0,1,-207.9,-8.2,755acd6f +pendulum,reward:stale,one,1,boundary,50,-170.0,0,0,0,1,-192.6,-22.6,29a4a30e +pendulum,reward:stale,one,2,boundary,50,-167.22,0,0,0,1,-199.5,-32.28,8b012764 +pendulum,reward:swap,0.001,0,boundary,50,-176.36,0,0,0,1,-193.1,-16.74,5be1abca +pendulum,reward:swap,0.001,1,boundary,50,-164.66,0,0,0,1,-206.5,-41.84,83e44dcc +pendulum,reward:swap,0.001,2,boundary,50,-252.39,0,0,0,1,-199.1,53.29,a6b6f114 +pendulum,reward:swap,0.01,0,boundary,50,-171.88,0,0,0,1,-193.7,-21.82,c5bef941 +pendulum,reward:swap,0.01,1,boundary,50,-170.05,0,0,0,1,-185.5,-15.45,49df93d6 +pendulum,reward:swap,0.01,2,boundary,50,-168.89,0,0,0,1,-220.7,-51.81,1f2b8986 +pendulum,reward:swap,0.1,0,boundary,50,-178.36,0,0,0,1,-243.8,-65.44,483edb81 +pendulum,reward:swap,0.1,1,boundary,50,-164.84,0,0,0,1,-241.7,-76.86,3396496c +pendulum,reward:swap,0.1,2,boundary,50,-167.38,0,0,0,1,-224.4,-57.02,998e1fa2 +pendulum,reward:swap,1.0,0,boundary,50,-1415.51,1,1,1,0,-1213.8,201.71,f15ee7af +pendulum,reward:swap,1.0,1,boundary,50,-1127.24,1,1,1,0,-1173.4,-46.16,333cd4cd +pendulum,reward:swap,1.0,2,boundary,50,-1569.02,1,1,1,0,-1353.2,215.82,5a1c7763 +pendulum,reward:swap,one,0,boundary,50,-175.07,0,0,0,1,-210.5,-35.43,67f6552c +pendulum,reward:swap,one,1,boundary,50,-200.42,0,0,0,1,-211.7,-11.28,38b7ad55 +pendulum,reward:swap,one,2,boundary,50,-164.66,0,0,0,1,-188.9,-24.24,dd11b659 +pendulum,reward:zero,0.001,0,boundary,50,-169.53,0,0,0,1,-188.2,-18.67,62b73298 +pendulum,reward:zero,0.001,1,boundary,50,-169.03,0,0,0,1,-190.5,-21.47,451c93c7 +pendulum,reward:zero,0.001,2,boundary,50,-175.53,0,0,0,1,-201.6,-26.07,e2a3a84a +pendulum,reward:zero,0.01,0,boundary,50,-169.98,0,0,0,1,-196.9,-26.92,11d1d7dc +pendulum,reward:zero,0.01,1,boundary,50,-168.73,0,0,0,1,-196.3,-27.57,e90c654b +pendulum,reward:zero,0.01,2,boundary,50,-173.11,0,0,0,1,-216.0,-42.89,57614cea +pendulum,reward:zero,0.1,0,boundary,50,-166.07,0,0,0,1,-194.0,-27.93,6230d345 +pendulum,reward:zero,0.1,1,boundary,50,-180.64,0,0,0,1,-221.8,-41.16,dc7fc3b4 +pendulum,reward:zero,0.1,2,boundary,50,-182.39,0,0,0,1,-211.9,-29.51,13b5a9ee +pendulum,reward:zero,1.0,0,boundary,50,-1195.55,1,1,1,0,0.0,1195.55,f3c1c4c3 +pendulum,reward:zero,1.0,1,boundary,50,-1549.63,1,1,1,0,0.0,1549.63,d483503e +pendulum,reward:zero,1.0,2,boundary,50,-1639.38,1,1,1,0,0.0,1639.38,092cb09b +pendulum,reward:zero,one,0,boundary,50,-172.09,0,0,0,1,-203.5,-31.41,e42c5fa8 +pendulum,reward:zero,one,1,boundary,50,-171.3,0,0,0,1,-209.0,-37.7,b0cdde97 +pendulum,reward:zero,one,2,boundary,50,-225.03,0,0,0,1,-204.8,20.23,77163c4c +pendulum,step:fill,0.001,0,boundary,50,-171.17,0,0,0,1,-190.8,-19.63,84594ba6 +pendulum,step:fill,0.001,1,boundary,50,-173.48,0,0,0,1,-190.2,-16.72,aaef822c +pendulum,step:fill,0.001,2,boundary,50,-169.14,0,0,0,1,-193.3,-24.16,9d28430e +pendulum,step:fill,0.01,0,boundary,50,-183.71,0,0,0,1,-188.7,-4.99,32ba03dd +pendulum,step:fill,0.01,1,boundary,50,-172.08,0,0,0,1,-199.4,-27.32,c201aca0 +pendulum,step:fill,0.01,2,boundary,50,-166.2,0,0,0,1,-206.3,-40.1,b14b76d2 +pendulum,step:fill,0.1,0,boundary,50,-179.04,0,0,0,1,-209.3,-30.26,7cc67c68 +pendulum,step:fill,0.1,1,boundary,50,-170.44,0,0,0,1,-221.1,-50.66,6a8df066 +pendulum,step:fill,0.1,2,boundary,50,-158.12,0,0,0,1,-239.3,-81.18,e53add0e +pendulum,step:fill,1.0,0,boundary,50,-1521.44,1,1,1,0,,,7c353631 +pendulum,step:fill,1.0,1,boundary,50,-1598.55,1,1,1,0,,,d303871d +pendulum,step:fill,1.0,2,boundary,50,-1451.93,1,1,1,0,,,36f6d91e +pendulum,step:fill,one,0,boundary,50,-172.22,0,0,0,1,-208.6,-36.38,1426c402 +pendulum,step:fill,one,1,boundary,50,-168.88,0,0,0,1,-212.3,-43.42,1b86e0d9 +pendulum,step:fill,one,2,boundary,50,-168.62,0,0,0,1,-194.4,-25.78,e2d1cee4 +pendulum,terminated:spurious,0.001,0,boundary,50,-170.31,0,0,1,0,-168.1,2.21,d6f284ed +pendulum,terminated:spurious,0.001,1,boundary,50,-174.73,0,0,1,0,-163.0,11.73,1e3f87cc +pendulum,terminated:spurious,0.001,2,boundary,50,-255.89,0,0,1,0,-178.6,77.29,eb7e82e2 +pendulum,terminated:spurious,0.01,0,boundary,50,-174.93,0,1,1,0,-178.6,-3.67,bfaf6961 +pendulum,terminated:spurious,0.01,1,boundary,50,-161.27,0,1,1,0,-142.4,18.87,d166081a +pendulum,terminated:spurious,0.01,2,boundary,50,-180.37,0,1,1,0,-152.5,27.87,66d62461 +pendulum,terminated:spurious,0.1,0,boundary,50,-623.94,1,1,1,0,-48.8,575.14,c336fac8 +pendulum,terminated:spurious,0.1,1,boundary,50,-667.9,1,1,1,0,-44.9,623.0,f1cfa982 +pendulum,terminated:spurious,0.1,2,boundary,50,-865.41,1,1,1,0,-50.6,814.81,711b8f59 +pendulum,terminated:spurious,1.0,0,boundary,50,-1388.18,1,1,1,0,-6.5,1381.68,f885d396 +pendulum,terminated:spurious,1.0,1,boundary,50,-1356.0,1,1,1,0,-6.3,1349.7,ef551e57 +pendulum,terminated:spurious,1.0,2,boundary,50,-1174.6,1,1,1,0,-6.0,1168.6,1611c255 +pendulum,terminated:spurious,one,0,boundary,50,-170.7,0,0,0,1,-210.5,-39.8,44d6e670 +pendulum,terminated:spurious,one,1,boundary,50,-167.63,0,0,0,1,-193.6,-25.97,ad8c101a +pendulum,terminated:spurious,one,2,boundary,50,-157.55,0,0,0,1,-193.1,-35.55,b9f30a62 +pendulum,truncated:as-term,0.001,0,boundary,50,-169.78,0,0,0,1,-200.8,-31.02,891f5fc9 +pendulum,truncated:as-term,0.001,1,boundary,50,-168.51,0,0,0,1,-202.5,-33.99,c098bfe9 +pendulum,truncated:as-term,0.001,2,boundary,50,-162.81,0,0,0,1,-196.6,-33.79,486d154e +pendulum,truncated:as-term,0.01,0,boundary,50,-194.48,0,0,0,1,-224.4,-29.92,9ea56785 +pendulum,truncated:as-term,0.01,1,boundary,50,-164.11,0,0,0,1,-195.1,-30.99,5e225449 +pendulum,truncated:as-term,0.01,2,boundary,50,-217.04,0,0,0,1,-206.6,10.44,8ffa7e42 +pendulum,truncated:as-term,0.1,0,boundary,50,-173.93,0,0,0,1,-211.8,-37.87,24da6230 +pendulum,truncated:as-term,0.1,1,boundary,50,-168.04,0,0,0,1,-191.1,-23.06,c6e6a98a +pendulum,truncated:as-term,0.1,2,boundary,50,-170.46,0,0,0,1,-195.9,-25.44,0f716951 +pendulum,truncated:as-term,1.0,0,boundary,50,-166.25,0,0,0,1,-204.6,-38.35,57eb5601 +pendulum,truncated:as-term,1.0,1,boundary,50,-165.36,0,0,0,1,-237.9,-72.54,2c8a0d7a +pendulum,truncated:as-term,1.0,2,boundary,50,-163.57,0,0,0,1,-208.8,-45.23,3292a6ce +pendulum,truncated:as-term,one,0,boundary,50,-171.16,0,0,0,1,-187.0,-15.84,1778bb25 +pendulum,truncated:as-term,one,1,boundary,50,-171.35,0,0,0,1,-194.8,-23.45,31459006 +pendulum,truncated:as-term,one,2,boundary,50,-173.56,0,0,0,1,-200.0,-26.44,bb2c9d55 +pendulum,truncated:drop,0.001,0,boundary,50,-195.56,0,0,0,1,-197.0,-1.44,e075c57a +pendulum,truncated:drop,0.001,1,boundary,50,-170.25,0,0,0,1,-205.3,-35.05,22cb06b9 +pendulum,truncated:drop,0.001,2,boundary,50,-169.98,0,0,0,1,-201.5,-31.52,496ca350 +pendulum,truncated:drop,0.01,0,boundary,50,-168.3,0,0,0,1,-194.0,-25.7,bfafb0fb +pendulum,truncated:drop,0.01,1,boundary,50,-168.15,0,0,0,1,-197.2,-29.05,db17509e +pendulum,truncated:drop,0.01,2,boundary,50,-176.03,0,0,0,1,-222.3,-46.27,8c5d0450 +pendulum,truncated:drop,0.1,0,boundary,50,-170.96,0,0,0,1,-242.9,-71.94,d4b1ea6c +pendulum,truncated:drop,0.1,1,boundary,50,-170.25,0,0,0,1,-241.8,-71.55,2f1eea58 +pendulum,truncated:drop,0.1,2,boundary,50,-167.43,0,0,0,1,-236.6,-69.17,48109f2c +pendulum,truncated:drop,1.0,0,boundary,50,-203.96,0,1,1,0,,,1a833295 +pendulum,truncated:drop,1.0,1,boundary,50,-161.45,0,1,1,0,,,c077fca8 +pendulum,truncated:drop,1.0,2,boundary,50,-177.85,0,1,1,0,,,85691232 +pendulum,truncated:drop,one,0,boundary,50,-171.57,0,0,0,1,-192.0,-20.43,754e49b9 +pendulum,truncated:drop,one,1,boundary,50,-173.81,0,0,0,1,-225.5,-51.69,b21d8718 +pendulum,truncated:drop,one,2,boundary,50,-165.34,0,0,0,1,-201.6,-36.26,4bc2ef7b +pendulum,wire-float64,1.0,0,control,50,-200.52,0,0,0,1,-199.8,0.72,ec5b8226 +pendulum,wire-float64,1.0,1,control,50,-174.6,0,0,0,1,-208.8,-34.2,062e92e3 +pendulum,wire-float64,1.0,2,control,50,-176.0,0,0,0,1,-246.0,-70.0,3a8d968f diff --git a/experiments/e57-clean-eval/results/e57-src.sha256 b/experiments/e57-clean-eval/results/e57-src.sha256 new file mode 100644 index 0000000..4fb0195 --- /dev/null +++ b/experiments/e57-clean-eval/results/e57-src.sha256 @@ -0,0 +1,2 @@ +277daf21ebd4d4b066dad5a800fa4b0f2af43e7d352d5a5af1655e3480272e0b evaluate.py +9cbe443184dc3b89fb5f77bfe06373a201590f51859b8c92e6569f0435e20501 summarise.py diff --git a/experiments/e57-clean-eval/results/explore.txt b/experiments/e57-clean-eval/results/explore.txt new file mode 100644 index 0000000..458df73 --- /dev/null +++ b/experiments/e57-clean-eval/results/explore.txt @@ -0,0 +1,13 @@ +exploratory: boundary-fault runs silent on the curve, against fault-free runs +env dose n mean z median z U-test p twin gap median (clean - none at same seed) +halfcheetah one 60 0.56 0.51 0.096 100.7 +halfcheetah 0.01 57 0.43 0.55 0.139 80.0 +halfcheetah 1.0 31 0.28 0.41 0.571 63.0 +hopper one 69 0.19 0.62 0.534 142.4 +hopper 0.01 66 0.16 0.67 0.539 168.3 +hopper 1.0 35 0.27 0.81 0.342 368.2 +pendulum one 63 0.24 0.55 0.195 6.8 +pendulum 0.001 58 0.03 0.60 0.053 6.5 +pendulum 0.01 59 0.09 0.54 0.208 7.3 +pendulum 0.1 50 0.36 0.60 0.061 8.4 +pendulum 1.0 25 0.65 0.68 0.003 9.2 diff --git a/experiments/e57-clean-eval/results/jobs.txt b/experiments/e57-clean-eval/results/jobs.txt new file mode 100644 index 0000000..295da0e --- /dev/null +++ b/experiments/e57-clean-eval/results/jobs.txt @@ -0,0 +1,834 @@ +../e54-boundary-faults/results/halfcheetah/action__clip/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/action__clip/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/action__clip/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/action__clip/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/action__clip/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/action__clip/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/action__clip/one/seed0 +../e54-boundary-faults/results/halfcheetah/action__clip/one/seed1 +../e54-boundary-faults/results/halfcheetah/action__clip/one/seed2 +../e54-boundary-faults/results/halfcheetah/action__f16/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/action__f16/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/action__f16/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/action__f16/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/action__f16/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/action__f16/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/action__f16/one/seed0 +../e54-boundary-faults/results/halfcheetah/action__f16/one/seed1 +../e54-boundary-faults/results/halfcheetah/action__f16/one/seed2 +../e54-boundary-faults/results/halfcheetah/action__stale/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/action__stale/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/action__stale/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/action__stale/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/action__stale/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/action__stale/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/action__stale/one/seed0 +../e54-boundary-faults/results/halfcheetah/action__stale/one/seed1 +../e54-boundary-faults/results/halfcheetah/action__stale/one/seed2 +../e54-boundary-faults/results/halfcheetah/action__swap/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/action__swap/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/action__swap/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/action__swap/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/action__swap/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/action__swap/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/action__swap/one/seed0 +../e54-boundary-faults/results/halfcheetah/action__swap/one/seed1 +../e54-boundary-faults/results/halfcheetah/action__swap/one/seed2 +../e54-boundary-faults/results/halfcheetah/action__zero/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/action__zero/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/action__zero/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/action__zero/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/action__zero/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/action__zero/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/action__zero/one/seed0 +../e54-boundary-faults/results/halfcheetah/action__zero/one/seed1 +../e54-boundary-faults/results/halfcheetah/action__zero/one/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/one/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/one/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__f16/one/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/one/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/one/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__stale/one/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/one/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/one/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__swap/one/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/one/seed0 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/one/seed1 +../e54-boundary-faults/results/halfcheetah/final-obs__zero/one/seed2 +../e54-boundary-faults/results/halfcheetah/log__drop-first/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/log__drop-first/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/log__drop-first/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/log__drop-first/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/log__drop-first/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/log__drop-first/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/log__drop-first/one/seed0 +../e54-boundary-faults/results/halfcheetah/log__drop-first/one/seed1 +../e54-boundary-faults/results/halfcheetah/log__drop-first/one/seed2 +../e54-boundary-faults/results/halfcheetah/log__short/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/log__short/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/log__short/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/log__short/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/log__short/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/log__short/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/log__short/one/seed0 +../e54-boundary-faults/results/halfcheetah/log__short/one/seed1 +../e54-boundary-faults/results/halfcheetah/log__short/one/seed2 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed10 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed11 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed12 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed13 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed14 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed15 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed16 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed17 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed18 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed19 +../e54-boundary-faults/results/halfcheetah/none/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/obs__f16/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/obs__f16/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/obs__f16/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/obs__f16/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/obs__f16/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/obs__f16/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/obs__f16/one/seed0 +../e54-boundary-faults/results/halfcheetah/obs__f16/one/seed1 +../e54-boundary-faults/results/halfcheetah/obs__f16/one/seed2 +../e54-boundary-faults/results/halfcheetah/obs__stale/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/obs__stale/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/obs__stale/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/obs__stale/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/obs__stale/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/obs__stale/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/obs__stale/one/seed0 +../e54-boundary-faults/results/halfcheetah/obs__stale/one/seed1 +../e54-boundary-faults/results/halfcheetah/obs__stale/one/seed2 +../e54-boundary-faults/results/halfcheetah/obs__swap/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/obs__swap/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/obs__swap/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/obs__swap/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/obs__swap/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/obs__swap/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/obs__swap/one/seed0 +../e54-boundary-faults/results/halfcheetah/obs__swap/one/seed1 +../e54-boundary-faults/results/halfcheetah/obs__swap/one/seed2 +../e54-boundary-faults/results/halfcheetah/obs__zero/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/obs__zero/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/obs__zero/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/obs__zero/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/obs__zero/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/obs__zero/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/obs__zero/one/seed0 +../e54-boundary-faults/results/halfcheetah/obs__zero/one/seed1 +../e54-boundary-faults/results/halfcheetah/obs__zero/one/seed2 +../e54-boundary-faults/results/halfcheetah/reward__f16/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/reward__f16/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/reward__f16/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/reward__f16/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/reward__f16/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/reward__f16/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/reward__f16/one/seed0 +../e54-boundary-faults/results/halfcheetah/reward__f16/one/seed1 +../e54-boundary-faults/results/halfcheetah/reward__f16/one/seed2 +../e54-boundary-faults/results/halfcheetah/reward__stale/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/reward__stale/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/reward__stale/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/reward__stale/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/reward__stale/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/reward__stale/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/reward__stale/one/seed0 +../e54-boundary-faults/results/halfcheetah/reward__stale/one/seed1 +../e54-boundary-faults/results/halfcheetah/reward__stale/one/seed2 +../e54-boundary-faults/results/halfcheetah/reward__swap/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/reward__swap/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/reward__swap/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/reward__swap/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/reward__swap/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/reward__swap/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/reward__swap/one/seed0 +../e54-boundary-faults/results/halfcheetah/reward__swap/one/seed1 +../e54-boundary-faults/results/halfcheetah/reward__swap/one/seed2 +../e54-boundary-faults/results/halfcheetah/reward__zero/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/reward__zero/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/reward__zero/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/reward__zero/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/reward__zero/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/reward__zero/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/reward__zero/one/seed0 +../e54-boundary-faults/results/halfcheetah/reward__zero/one/seed1 +../e54-boundary-faults/results/halfcheetah/reward__zero/one/seed2 +../e54-boundary-faults/results/halfcheetah/step__fill/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/step__fill/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/step__fill/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/step__fill/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/step__fill/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/step__fill/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/step__fill/one/seed0 +../e54-boundary-faults/results/halfcheetah/step__fill/one/seed1 +../e54-boundary-faults/results/halfcheetah/step__fill/one/seed2 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/one/seed0 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/one/seed1 +../e54-boundary-faults/results/halfcheetah/terminated__spurious/one/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/one/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/one/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__as-term/one/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__drop/0.01/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__drop/0.01/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__drop/0.01/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__drop/1.0/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__drop/1.0/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__drop/1.0/seed2 +../e54-boundary-faults/results/halfcheetah/truncated__drop/one/seed0 +../e54-boundary-faults/results/halfcheetah/truncated__drop/one/seed1 +../e54-boundary-faults/results/halfcheetah/truncated__drop/one/seed2 +../e54-boundary-faults/results/pendulum/action__clip/0.001/seed0 +../e54-boundary-faults/results/pendulum/action__clip/0.001/seed1 +../e54-boundary-faults/results/pendulum/action__clip/0.001/seed2 +../e54-boundary-faults/results/pendulum/action__clip/0.01/seed0 +../e54-boundary-faults/results/pendulum/action__clip/0.01/seed1 +../e54-boundary-faults/results/pendulum/action__clip/0.01/seed2 +../e54-boundary-faults/results/pendulum/action__clip/0.1/seed0 +../e54-boundary-faults/results/pendulum/action__clip/0.1/seed1 +../e54-boundary-faults/results/pendulum/action__clip/0.1/seed2 +../e54-boundary-faults/results/pendulum/action__clip/1.0/seed0 +../e54-boundary-faults/results/pendulum/action__clip/1.0/seed1 +../e54-boundary-faults/results/pendulum/action__clip/1.0/seed2 +../e54-boundary-faults/results/pendulum/action__clip/one/seed0 +../e54-boundary-faults/results/pendulum/action__clip/one/seed1 +../e54-boundary-faults/results/pendulum/action__clip/one/seed2 +../e54-boundary-faults/results/pendulum/action__f16/0.001/seed0 +../e54-boundary-faults/results/pendulum/action__f16/0.001/seed1 +../e54-boundary-faults/results/pendulum/action__f16/0.001/seed2 +../e54-boundary-faults/results/pendulum/action__f16/0.01/seed0 +../e54-boundary-faults/results/pendulum/action__f16/0.01/seed1 +../e54-boundary-faults/results/pendulum/action__f16/0.01/seed2 +../e54-boundary-faults/results/pendulum/action__f16/0.1/seed0 +../e54-boundary-faults/results/pendulum/action__f16/0.1/seed1 +../e54-boundary-faults/results/pendulum/action__f16/0.1/seed2 +../e54-boundary-faults/results/pendulum/action__f16/1.0/seed0 +../e54-boundary-faults/results/pendulum/action__f16/1.0/seed1 +../e54-boundary-faults/results/pendulum/action__f16/1.0/seed2 +../e54-boundary-faults/results/pendulum/action__f16/one/seed0 +../e54-boundary-faults/results/pendulum/action__f16/one/seed1 +../e54-boundary-faults/results/pendulum/action__f16/one/seed2 +../e54-boundary-faults/results/pendulum/action__stale/0.001/seed0 +../e54-boundary-faults/results/pendulum/action__stale/0.001/seed1 +../e54-boundary-faults/results/pendulum/action__stale/0.001/seed2 +../e54-boundary-faults/results/pendulum/action__stale/0.01/seed0 +../e54-boundary-faults/results/pendulum/action__stale/0.01/seed1 +../e54-boundary-faults/results/pendulum/action__stale/0.01/seed2 +../e54-boundary-faults/results/pendulum/action__stale/0.1/seed0 +../e54-boundary-faults/results/pendulum/action__stale/0.1/seed1 +../e54-boundary-faults/results/pendulum/action__stale/0.1/seed2 +../e54-boundary-faults/results/pendulum/action__stale/1.0/seed0 +../e54-boundary-faults/results/pendulum/action__stale/1.0/seed1 +../e54-boundary-faults/results/pendulum/action__stale/1.0/seed2 +../e54-boundary-faults/results/pendulum/action__stale/one/seed0 +../e54-boundary-faults/results/pendulum/action__stale/one/seed1 +../e54-boundary-faults/results/pendulum/action__stale/one/seed2 +../e54-boundary-faults/results/pendulum/action__swap/0.001/seed0 +../e54-boundary-faults/results/pendulum/action__swap/0.001/seed1 +../e54-boundary-faults/results/pendulum/action__swap/0.001/seed2 +../e54-boundary-faults/results/pendulum/action__swap/0.01/seed0 +../e54-boundary-faults/results/pendulum/action__swap/0.01/seed1 +../e54-boundary-faults/results/pendulum/action__swap/0.01/seed2 +../e54-boundary-faults/results/pendulum/action__swap/0.1/seed0 +../e54-boundary-faults/results/pendulum/action__swap/0.1/seed1 +../e54-boundary-faults/results/pendulum/action__swap/0.1/seed2 +../e54-boundary-faults/results/pendulum/action__swap/1.0/seed0 +../e54-boundary-faults/results/pendulum/action__swap/1.0/seed1 +../e54-boundary-faults/results/pendulum/action__swap/1.0/seed2 +../e54-boundary-faults/results/pendulum/action__swap/one/seed0 +../e54-boundary-faults/results/pendulum/action__swap/one/seed1 +../e54-boundary-faults/results/pendulum/action__swap/one/seed2 +../e54-boundary-faults/results/pendulum/action__zero/0.001/seed0 +../e54-boundary-faults/results/pendulum/action__zero/0.001/seed1 +../e54-boundary-faults/results/pendulum/action__zero/0.001/seed2 +../e54-boundary-faults/results/pendulum/action__zero/0.01/seed0 +../e54-boundary-faults/results/pendulum/action__zero/0.01/seed1 +../e54-boundary-faults/results/pendulum/action__zero/0.01/seed2 +../e54-boundary-faults/results/pendulum/action__zero/0.1/seed0 +../e54-boundary-faults/results/pendulum/action__zero/0.1/seed1 +../e54-boundary-faults/results/pendulum/action__zero/0.1/seed2 +../e54-boundary-faults/results/pendulum/action__zero/1.0/seed0 +../e54-boundary-faults/results/pendulum/action__zero/1.0/seed1 +../e54-boundary-faults/results/pendulum/action__zero/1.0/seed2 +../e54-boundary-faults/results/pendulum/action__zero/one/seed0 +../e54-boundary-faults/results/pendulum/action__zero/one/seed1 +../e54-boundary-faults/results/pendulum/action__zero/one/seed2 +../e54-boundary-faults/results/pendulum/delay-1ms/1.0/seed0 +../e54-boundary-faults/results/pendulum/delay-1ms/1.0/seed1 +../e54-boundary-faults/results/pendulum/delay-1ms/1.0/seed2 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.001/seed0 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.001/seed1 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.001/seed2 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.01/seed0 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.01/seed1 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.01/seed2 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.1/seed0 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.1/seed1 +../e54-boundary-faults/results/pendulum/final-obs__f16/0.1/seed2 +../e54-boundary-faults/results/pendulum/final-obs__f16/1.0/seed0 +../e54-boundary-faults/results/pendulum/final-obs__f16/1.0/seed1 +../e54-boundary-faults/results/pendulum/final-obs__f16/1.0/seed2 +../e54-boundary-faults/results/pendulum/final-obs__f16/one/seed0 +../e54-boundary-faults/results/pendulum/final-obs__f16/one/seed1 +../e54-boundary-faults/results/pendulum/final-obs__f16/one/seed2 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.001/seed0 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.001/seed1 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.001/seed2 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.01/seed0 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.01/seed1 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.01/seed2 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.1/seed0 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.1/seed1 +../e54-boundary-faults/results/pendulum/final-obs__stale/0.1/seed2 +../e54-boundary-faults/results/pendulum/final-obs__stale/1.0/seed0 +../e54-boundary-faults/results/pendulum/final-obs__stale/1.0/seed1 +../e54-boundary-faults/results/pendulum/final-obs__stale/1.0/seed2 +../e54-boundary-faults/results/pendulum/final-obs__stale/one/seed0 +../e54-boundary-faults/results/pendulum/final-obs__stale/one/seed1 +../e54-boundary-faults/results/pendulum/final-obs__stale/one/seed2 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.001/seed0 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.001/seed1 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.001/seed2 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.01/seed0 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.01/seed1 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.01/seed2 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.1/seed0 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.1/seed1 +../e54-boundary-faults/results/pendulum/final-obs__swap/0.1/seed2 +../e54-boundary-faults/results/pendulum/final-obs__swap/1.0/seed0 +../e54-boundary-faults/results/pendulum/final-obs__swap/1.0/seed1 +../e54-boundary-faults/results/pendulum/final-obs__swap/1.0/seed2 +../e54-boundary-faults/results/pendulum/final-obs__swap/one/seed0 +../e54-boundary-faults/results/pendulum/final-obs__swap/one/seed1 +../e54-boundary-faults/results/pendulum/final-obs__swap/one/seed2 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.001/seed0 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.001/seed1 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.001/seed2 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.01/seed0 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.01/seed1 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.01/seed2 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.1/seed0 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.1/seed1 +../e54-boundary-faults/results/pendulum/final-obs__zero/0.1/seed2 +../e54-boundary-faults/results/pendulum/final-obs__zero/1.0/seed0 +../e54-boundary-faults/results/pendulum/final-obs__zero/1.0/seed1 +../e54-boundary-faults/results/pendulum/final-obs__zero/1.0/seed2 +../e54-boundary-faults/results/pendulum/final-obs__zero/one/seed0 +../e54-boundary-faults/results/pendulum/final-obs__zero/one/seed1 +../e54-boundary-faults/results/pendulum/final-obs__zero/one/seed2 +../e54-boundary-faults/results/pendulum/indices__reorder/one/seed0 +../e54-boundary-faults/results/pendulum/indices__reorder/one/seed1 +../e54-boundary-faults/results/pendulum/indices__reorder/one/seed2 +../e54-boundary-faults/results/pendulum/log__drop-first/0.001/seed0 +../e54-boundary-faults/results/pendulum/log__drop-first/0.001/seed1 +../e54-boundary-faults/results/pendulum/log__drop-first/0.001/seed2 +../e54-boundary-faults/results/pendulum/log__drop-first/0.01/seed0 +../e54-boundary-faults/results/pendulum/log__drop-first/0.01/seed1 +../e54-boundary-faults/results/pendulum/log__drop-first/0.01/seed2 +../e54-boundary-faults/results/pendulum/log__drop-first/0.1/seed0 +../e54-boundary-faults/results/pendulum/log__drop-first/0.1/seed1 +../e54-boundary-faults/results/pendulum/log__drop-first/0.1/seed2 +../e54-boundary-faults/results/pendulum/log__drop-first/1.0/seed0 +../e54-boundary-faults/results/pendulum/log__drop-first/1.0/seed1 +../e54-boundary-faults/results/pendulum/log__drop-first/1.0/seed2 +../e54-boundary-faults/results/pendulum/log__drop-first/one/seed0 +../e54-boundary-faults/results/pendulum/log__drop-first/one/seed1 +../e54-boundary-faults/results/pendulum/log__drop-first/one/seed2 +../e54-boundary-faults/results/pendulum/log__short/0.001/seed0 +../e54-boundary-faults/results/pendulum/log__short/0.001/seed1 +../e54-boundary-faults/results/pendulum/log__short/0.001/seed2 +../e54-boundary-faults/results/pendulum/log__short/0.01/seed0 +../e54-boundary-faults/results/pendulum/log__short/0.01/seed1 +../e54-boundary-faults/results/pendulum/log__short/0.01/seed2 +../e54-boundary-faults/results/pendulum/log__short/0.1/seed0 +../e54-boundary-faults/results/pendulum/log__short/0.1/seed1 +../e54-boundary-faults/results/pendulum/log__short/0.1/seed2 +../e54-boundary-faults/results/pendulum/log__short/1.0/seed0 +../e54-boundary-faults/results/pendulum/log__short/1.0/seed1 +../e54-boundary-faults/results/pendulum/log__short/1.0/seed2 +../e54-boundary-faults/results/pendulum/log__short/one/seed0 +../e54-boundary-faults/results/pendulum/log__short/one/seed1 +../e54-boundary-faults/results/pendulum/log__short/one/seed2 +../e54-boundary-faults/results/pendulum/none/1.0/seed0 +../e54-boundary-faults/results/pendulum/none/1.0/seed1 +../e54-boundary-faults/results/pendulum/none/1.0/seed10 +../e54-boundary-faults/results/pendulum/none/1.0/seed11 +../e54-boundary-faults/results/pendulum/none/1.0/seed12 +../e54-boundary-faults/results/pendulum/none/1.0/seed13 +../e54-boundary-faults/results/pendulum/none/1.0/seed14 +../e54-boundary-faults/results/pendulum/none/1.0/seed15 +../e54-boundary-faults/results/pendulum/none/1.0/seed16 +../e54-boundary-faults/results/pendulum/none/1.0/seed17 +../e54-boundary-faults/results/pendulum/none/1.0/seed18 +../e54-boundary-faults/results/pendulum/none/1.0/seed19 +../e54-boundary-faults/results/pendulum/none/1.0/seed2 +../e54-boundary-faults/results/pendulum/obs__f16/0.001/seed0 +../e54-boundary-faults/results/pendulum/obs__f16/0.001/seed1 +../e54-boundary-faults/results/pendulum/obs__f16/0.001/seed2 +../e54-boundary-faults/results/pendulum/obs__f16/0.01/seed0 +../e54-boundary-faults/results/pendulum/obs__f16/0.01/seed1 +../e54-boundary-faults/results/pendulum/obs__f16/0.01/seed2 +../e54-boundary-faults/results/pendulum/obs__f16/0.1/seed0 +../e54-boundary-faults/results/pendulum/obs__f16/0.1/seed1 +../e54-boundary-faults/results/pendulum/obs__f16/0.1/seed2 +../e54-boundary-faults/results/pendulum/obs__f16/1.0/seed0 +../e54-boundary-faults/results/pendulum/obs__f16/1.0/seed1 +../e54-boundary-faults/results/pendulum/obs__f16/1.0/seed2 +../e54-boundary-faults/results/pendulum/obs__f16/one/seed0 +../e54-boundary-faults/results/pendulum/obs__f16/one/seed1 +../e54-boundary-faults/results/pendulum/obs__f16/one/seed2 +../e54-boundary-faults/results/pendulum/obs__stale/0.001/seed0 +../e54-boundary-faults/results/pendulum/obs__stale/0.001/seed1 +../e54-boundary-faults/results/pendulum/obs__stale/0.001/seed2 +../e54-boundary-faults/results/pendulum/obs__stale/0.01/seed0 +../e54-boundary-faults/results/pendulum/obs__stale/0.01/seed1 +../e54-boundary-faults/results/pendulum/obs__stale/0.01/seed2 +../e54-boundary-faults/results/pendulum/obs__stale/0.1/seed0 +../e54-boundary-faults/results/pendulum/obs__stale/0.1/seed1 +../e54-boundary-faults/results/pendulum/obs__stale/0.1/seed2 +../e54-boundary-faults/results/pendulum/obs__stale/1.0/seed0 +../e54-boundary-faults/results/pendulum/obs__stale/1.0/seed1 +../e54-boundary-faults/results/pendulum/obs__stale/1.0/seed2 +../e54-boundary-faults/results/pendulum/obs__stale/one/seed0 +../e54-boundary-faults/results/pendulum/obs__stale/one/seed1 +../e54-boundary-faults/results/pendulum/obs__stale/one/seed2 +../e54-boundary-faults/results/pendulum/obs__swap/0.001/seed0 +../e54-boundary-faults/results/pendulum/obs__swap/0.001/seed1 +../e54-boundary-faults/results/pendulum/obs__swap/0.001/seed2 +../e54-boundary-faults/results/pendulum/obs__swap/0.01/seed0 +../e54-boundary-faults/results/pendulum/obs__swap/0.01/seed1 +../e54-boundary-faults/results/pendulum/obs__swap/0.01/seed2 +../e54-boundary-faults/results/pendulum/obs__swap/0.1/seed0 +../e54-boundary-faults/results/pendulum/obs__swap/0.1/seed1 +../e54-boundary-faults/results/pendulum/obs__swap/0.1/seed2 +../e54-boundary-faults/results/pendulum/obs__swap/1.0/seed0 +../e54-boundary-faults/results/pendulum/obs__swap/1.0/seed1 +../e54-boundary-faults/results/pendulum/obs__swap/1.0/seed2 +../e54-boundary-faults/results/pendulum/obs__swap/one/seed0 +../e54-boundary-faults/results/pendulum/obs__swap/one/seed1 +../e54-boundary-faults/results/pendulum/obs__swap/one/seed2 +../e54-boundary-faults/results/pendulum/obs__zero/0.001/seed0 +../e54-boundary-faults/results/pendulum/obs__zero/0.001/seed1 +../e54-boundary-faults/results/pendulum/obs__zero/0.001/seed2 +../e54-boundary-faults/results/pendulum/obs__zero/0.01/seed0 +../e54-boundary-faults/results/pendulum/obs__zero/0.01/seed1 +../e54-boundary-faults/results/pendulum/obs__zero/0.01/seed2 +../e54-boundary-faults/results/pendulum/obs__zero/0.1/seed0 +../e54-boundary-faults/results/pendulum/obs__zero/0.1/seed1 +../e54-boundary-faults/results/pendulum/obs__zero/0.1/seed2 +../e54-boundary-faults/results/pendulum/obs__zero/1.0/seed0 +../e54-boundary-faults/results/pendulum/obs__zero/1.0/seed1 +../e54-boundary-faults/results/pendulum/obs__zero/1.0/seed2 +../e54-boundary-faults/results/pendulum/obs__zero/one/seed0 +../e54-boundary-faults/results/pendulum/obs__zero/one/seed1 +../e54-boundary-faults/results/pendulum/obs__zero/one/seed2 +../e54-boundary-faults/results/pendulum/reward__f16/0.001/seed0 +../e54-boundary-faults/results/pendulum/reward__f16/0.001/seed1 +../e54-boundary-faults/results/pendulum/reward__f16/0.001/seed2 +../e54-boundary-faults/results/pendulum/reward__f16/0.01/seed0 +../e54-boundary-faults/results/pendulum/reward__f16/0.01/seed1 +../e54-boundary-faults/results/pendulum/reward__f16/0.01/seed2 +../e54-boundary-faults/results/pendulum/reward__f16/0.1/seed0 +../e54-boundary-faults/results/pendulum/reward__f16/0.1/seed1 +../e54-boundary-faults/results/pendulum/reward__f16/0.1/seed2 +../e54-boundary-faults/results/pendulum/reward__f16/1.0/seed0 +../e54-boundary-faults/results/pendulum/reward__f16/1.0/seed1 +../e54-boundary-faults/results/pendulum/reward__f16/1.0/seed2 +../e54-boundary-faults/results/pendulum/reward__f16/one/seed0 +../e54-boundary-faults/results/pendulum/reward__f16/one/seed1 +../e54-boundary-faults/results/pendulum/reward__f16/one/seed2 +../e54-boundary-faults/results/pendulum/reward__short-array/one/seed0 +../e54-boundary-faults/results/pendulum/reward__short-array/one/seed1 +../e54-boundary-faults/results/pendulum/reward__short-array/one/seed2 +../e54-boundary-faults/results/pendulum/reward__stale/0.001/seed0 +../e54-boundary-faults/results/pendulum/reward__stale/0.001/seed1 +../e54-boundary-faults/results/pendulum/reward__stale/0.001/seed2 +../e54-boundary-faults/results/pendulum/reward__stale/0.01/seed0 +../e54-boundary-faults/results/pendulum/reward__stale/0.01/seed1 +../e54-boundary-faults/results/pendulum/reward__stale/0.01/seed2 +../e54-boundary-faults/results/pendulum/reward__stale/0.1/seed0 +../e54-boundary-faults/results/pendulum/reward__stale/0.1/seed1 +../e54-boundary-faults/results/pendulum/reward__stale/0.1/seed2 +../e54-boundary-faults/results/pendulum/reward__stale/1.0/seed0 +../e54-boundary-faults/results/pendulum/reward__stale/1.0/seed1 +../e54-boundary-faults/results/pendulum/reward__stale/1.0/seed2 +../e54-boundary-faults/results/pendulum/reward__stale/one/seed0 +../e54-boundary-faults/results/pendulum/reward__stale/one/seed1 +../e54-boundary-faults/results/pendulum/reward__stale/one/seed2 +../e54-boundary-faults/results/pendulum/reward__swap/0.001/seed0 +../e54-boundary-faults/results/pendulum/reward__swap/0.001/seed1 +../e54-boundary-faults/results/pendulum/reward__swap/0.001/seed2 +../e54-boundary-faults/results/pendulum/reward__swap/0.01/seed0 +../e54-boundary-faults/results/pendulum/reward__swap/0.01/seed1 +../e54-boundary-faults/results/pendulum/reward__swap/0.01/seed2 +../e54-boundary-faults/results/pendulum/reward__swap/0.1/seed0 +../e54-boundary-faults/results/pendulum/reward__swap/0.1/seed1 +../e54-boundary-faults/results/pendulum/reward__swap/0.1/seed2 +../e54-boundary-faults/results/pendulum/reward__swap/1.0/seed0 +../e54-boundary-faults/results/pendulum/reward__swap/1.0/seed1 +../e54-boundary-faults/results/pendulum/reward__swap/1.0/seed2 +../e54-boundary-faults/results/pendulum/reward__swap/one/seed0 +../e54-boundary-faults/results/pendulum/reward__swap/one/seed1 +../e54-boundary-faults/results/pendulum/reward__swap/one/seed2 +../e54-boundary-faults/results/pendulum/reward__zero/0.001/seed0 +../e54-boundary-faults/results/pendulum/reward__zero/0.001/seed1 +../e54-boundary-faults/results/pendulum/reward__zero/0.001/seed2 +../e54-boundary-faults/results/pendulum/reward__zero/0.01/seed0 +../e54-boundary-faults/results/pendulum/reward__zero/0.01/seed1 +../e54-boundary-faults/results/pendulum/reward__zero/0.01/seed2 +../e54-boundary-faults/results/pendulum/reward__zero/0.1/seed0 +../e54-boundary-faults/results/pendulum/reward__zero/0.1/seed1 +../e54-boundary-faults/results/pendulum/reward__zero/0.1/seed2 +../e54-boundary-faults/results/pendulum/reward__zero/1.0/seed0 +../e54-boundary-faults/results/pendulum/reward__zero/1.0/seed1 +../e54-boundary-faults/results/pendulum/reward__zero/1.0/seed2 +../e54-boundary-faults/results/pendulum/reward__zero/one/seed0 +../e54-boundary-faults/results/pendulum/reward__zero/one/seed1 +../e54-boundary-faults/results/pendulum/reward__zero/one/seed2 +../e54-boundary-faults/results/pendulum/step__fill/0.001/seed0 +../e54-boundary-faults/results/pendulum/step__fill/0.001/seed1 +../e54-boundary-faults/results/pendulum/step__fill/0.001/seed2 +../e54-boundary-faults/results/pendulum/step__fill/0.01/seed0 +../e54-boundary-faults/results/pendulum/step__fill/0.01/seed1 +../e54-boundary-faults/results/pendulum/step__fill/0.01/seed2 +../e54-boundary-faults/results/pendulum/step__fill/0.1/seed0 +../e54-boundary-faults/results/pendulum/step__fill/0.1/seed1 +../e54-boundary-faults/results/pendulum/step__fill/0.1/seed2 +../e54-boundary-faults/results/pendulum/step__fill/1.0/seed0 +../e54-boundary-faults/results/pendulum/step__fill/1.0/seed1 +../e54-boundary-faults/results/pendulum/step__fill/1.0/seed2 +../e54-boundary-faults/results/pendulum/step__fill/one/seed0 +../e54-boundary-faults/results/pendulum/step__fill/one/seed1 +../e54-boundary-faults/results/pendulum/step__fill/one/seed2 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.001/seed0 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.001/seed1 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.001/seed2 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.01/seed0 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.01/seed1 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.01/seed2 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.1/seed0 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.1/seed1 +../e54-boundary-faults/results/pendulum/terminated__spurious/0.1/seed2 +../e54-boundary-faults/results/pendulum/terminated__spurious/1.0/seed0 +../e54-boundary-faults/results/pendulum/terminated__spurious/1.0/seed1 +../e54-boundary-faults/results/pendulum/terminated__spurious/1.0/seed2 +../e54-boundary-faults/results/pendulum/terminated__spurious/one/seed0 +../e54-boundary-faults/results/pendulum/terminated__spurious/one/seed1 +../e54-boundary-faults/results/pendulum/terminated__spurious/one/seed2 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.001/seed0 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.001/seed1 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.001/seed2 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.01/seed0 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.01/seed1 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.01/seed2 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.1/seed0 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.1/seed1 +../e54-boundary-faults/results/pendulum/truncated__as-term/0.1/seed2 +../e54-boundary-faults/results/pendulum/truncated__as-term/1.0/seed0 +../e54-boundary-faults/results/pendulum/truncated__as-term/1.0/seed1 +../e54-boundary-faults/results/pendulum/truncated__as-term/1.0/seed2 +../e54-boundary-faults/results/pendulum/truncated__as-term/one/seed0 +../e54-boundary-faults/results/pendulum/truncated__as-term/one/seed1 +../e54-boundary-faults/results/pendulum/truncated__as-term/one/seed2 +../e54-boundary-faults/results/pendulum/truncated__drop/0.001/seed0 +../e54-boundary-faults/results/pendulum/truncated__drop/0.001/seed1 +../e54-boundary-faults/results/pendulum/truncated__drop/0.001/seed2 +../e54-boundary-faults/results/pendulum/truncated__drop/0.01/seed0 +../e54-boundary-faults/results/pendulum/truncated__drop/0.01/seed1 +../e54-boundary-faults/results/pendulum/truncated__drop/0.01/seed2 +../e54-boundary-faults/results/pendulum/truncated__drop/0.1/seed0 +../e54-boundary-faults/results/pendulum/truncated__drop/0.1/seed1 +../e54-boundary-faults/results/pendulum/truncated__drop/0.1/seed2 +../e54-boundary-faults/results/pendulum/truncated__drop/1.0/seed0 +../e54-boundary-faults/results/pendulum/truncated__drop/1.0/seed1 +../e54-boundary-faults/results/pendulum/truncated__drop/1.0/seed2 +../e54-boundary-faults/results/pendulum/truncated__drop/one/seed0 +../e54-boundary-faults/results/pendulum/truncated__drop/one/seed1 +../e54-boundary-faults/results/pendulum/truncated__drop/one/seed2 +../e54-boundary-faults/results/pendulum/wire-float64/1.0/seed0 +../e54-boundary-faults/results/pendulum/wire-float64/1.0/seed1 +../e54-boundary-faults/results/pendulum/wire-float64/1.0/seed2 +../e56-hopper-faults/results/hopper/action__clip/0.01/seed0 +../e56-hopper-faults/results/hopper/action__clip/0.01/seed1 +../e56-hopper-faults/results/hopper/action__clip/0.01/seed2 +../e56-hopper-faults/results/hopper/action__clip/1.0/seed0 +../e56-hopper-faults/results/hopper/action__clip/1.0/seed1 +../e56-hopper-faults/results/hopper/action__clip/1.0/seed2 +../e56-hopper-faults/results/hopper/action__clip/one/seed0 +../e56-hopper-faults/results/hopper/action__clip/one/seed1 +../e56-hopper-faults/results/hopper/action__clip/one/seed2 +../e56-hopper-faults/results/hopper/action__f16/0.01/seed0 +../e56-hopper-faults/results/hopper/action__f16/0.01/seed1 +../e56-hopper-faults/results/hopper/action__f16/0.01/seed2 +../e56-hopper-faults/results/hopper/action__f16/1.0/seed0 +../e56-hopper-faults/results/hopper/action__f16/1.0/seed1 +../e56-hopper-faults/results/hopper/action__f16/1.0/seed2 +../e56-hopper-faults/results/hopper/action__f16/one/seed0 +../e56-hopper-faults/results/hopper/action__f16/one/seed1 +../e56-hopper-faults/results/hopper/action__f16/one/seed2 +../e56-hopper-faults/results/hopper/action__stale/0.01/seed0 +../e56-hopper-faults/results/hopper/action__stale/0.01/seed1 +../e56-hopper-faults/results/hopper/action__stale/0.01/seed2 +../e56-hopper-faults/results/hopper/action__stale/1.0/seed0 +../e56-hopper-faults/results/hopper/action__stale/1.0/seed1 +../e56-hopper-faults/results/hopper/action__stale/1.0/seed2 +../e56-hopper-faults/results/hopper/action__stale/one/seed0 +../e56-hopper-faults/results/hopper/action__stale/one/seed1 +../e56-hopper-faults/results/hopper/action__stale/one/seed2 +../e56-hopper-faults/results/hopper/action__swap/0.01/seed0 +../e56-hopper-faults/results/hopper/action__swap/0.01/seed1 +../e56-hopper-faults/results/hopper/action__swap/0.01/seed2 +../e56-hopper-faults/results/hopper/action__swap/1.0/seed0 +../e56-hopper-faults/results/hopper/action__swap/1.0/seed1 +../e56-hopper-faults/results/hopper/action__swap/1.0/seed2 +../e56-hopper-faults/results/hopper/action__swap/one/seed0 +../e56-hopper-faults/results/hopper/action__swap/one/seed1 +../e56-hopper-faults/results/hopper/action__swap/one/seed2 +../e56-hopper-faults/results/hopper/action__zero/0.01/seed0 +../e56-hopper-faults/results/hopper/action__zero/0.01/seed1 +../e56-hopper-faults/results/hopper/action__zero/0.01/seed2 +../e56-hopper-faults/results/hopper/action__zero/1.0/seed0 +../e56-hopper-faults/results/hopper/action__zero/1.0/seed1 +../e56-hopper-faults/results/hopper/action__zero/1.0/seed2 +../e56-hopper-faults/results/hopper/action__zero/one/seed0 +../e56-hopper-faults/results/hopper/action__zero/one/seed1 +../e56-hopper-faults/results/hopper/action__zero/one/seed2 +../e56-hopper-faults/results/hopper/delay-1ms/1.0/seed0 +../e56-hopper-faults/results/hopper/delay-1ms/1.0/seed1 +../e56-hopper-faults/results/hopper/delay-1ms/1.0/seed2 +../e56-hopper-faults/results/hopper/final-obs__f16/0.01/seed0 +../e56-hopper-faults/results/hopper/final-obs__f16/0.01/seed1 +../e56-hopper-faults/results/hopper/final-obs__f16/0.01/seed2 +../e56-hopper-faults/results/hopper/final-obs__f16/1.0/seed0 +../e56-hopper-faults/results/hopper/final-obs__f16/1.0/seed1 +../e56-hopper-faults/results/hopper/final-obs__f16/1.0/seed2 +../e56-hopper-faults/results/hopper/final-obs__f16/one/seed0 +../e56-hopper-faults/results/hopper/final-obs__f16/one/seed1 +../e56-hopper-faults/results/hopper/final-obs__f16/one/seed2 +../e56-hopper-faults/results/hopper/final-obs__stale/0.01/seed0 +../e56-hopper-faults/results/hopper/final-obs__stale/0.01/seed1 +../e56-hopper-faults/results/hopper/final-obs__stale/0.01/seed2 +../e56-hopper-faults/results/hopper/final-obs__stale/1.0/seed0 +../e56-hopper-faults/results/hopper/final-obs__stale/1.0/seed1 +../e56-hopper-faults/results/hopper/final-obs__stale/1.0/seed2 +../e56-hopper-faults/results/hopper/final-obs__stale/one/seed0 +../e56-hopper-faults/results/hopper/final-obs__stale/one/seed1 +../e56-hopper-faults/results/hopper/final-obs__stale/one/seed2 +../e56-hopper-faults/results/hopper/final-obs__swap/0.01/seed0 +../e56-hopper-faults/results/hopper/final-obs__swap/0.01/seed1 +../e56-hopper-faults/results/hopper/final-obs__swap/0.01/seed2 +../e56-hopper-faults/results/hopper/final-obs__swap/1.0/seed0 +../e56-hopper-faults/results/hopper/final-obs__swap/1.0/seed1 +../e56-hopper-faults/results/hopper/final-obs__swap/1.0/seed2 +../e56-hopper-faults/results/hopper/final-obs__swap/one/seed0 +../e56-hopper-faults/results/hopper/final-obs__swap/one/seed1 +../e56-hopper-faults/results/hopper/final-obs__swap/one/seed2 +../e56-hopper-faults/results/hopper/final-obs__zero/0.01/seed0 +../e56-hopper-faults/results/hopper/final-obs__zero/0.01/seed1 +../e56-hopper-faults/results/hopper/final-obs__zero/0.01/seed2 +../e56-hopper-faults/results/hopper/final-obs__zero/1.0/seed0 +../e56-hopper-faults/results/hopper/final-obs__zero/1.0/seed1 +../e56-hopper-faults/results/hopper/final-obs__zero/1.0/seed2 +../e56-hopper-faults/results/hopper/final-obs__zero/one/seed0 +../e56-hopper-faults/results/hopper/final-obs__zero/one/seed1 +../e56-hopper-faults/results/hopper/final-obs__zero/one/seed2 +../e56-hopper-faults/results/hopper/log__drop-first/0.01/seed0 +../e56-hopper-faults/results/hopper/log__drop-first/0.01/seed1 +../e56-hopper-faults/results/hopper/log__drop-first/0.01/seed2 +../e56-hopper-faults/results/hopper/log__drop-first/1.0/seed0 +../e56-hopper-faults/results/hopper/log__drop-first/1.0/seed1 +../e56-hopper-faults/results/hopper/log__drop-first/1.0/seed2 +../e56-hopper-faults/results/hopper/log__drop-first/one/seed0 +../e56-hopper-faults/results/hopper/log__drop-first/one/seed1 +../e56-hopper-faults/results/hopper/log__drop-first/one/seed2 +../e56-hopper-faults/results/hopper/log__short/0.01/seed0 +../e56-hopper-faults/results/hopper/log__short/0.01/seed1 +../e56-hopper-faults/results/hopper/log__short/0.01/seed2 +../e56-hopper-faults/results/hopper/log__short/1.0/seed0 +../e56-hopper-faults/results/hopper/log__short/1.0/seed1 +../e56-hopper-faults/results/hopper/log__short/1.0/seed2 +../e56-hopper-faults/results/hopper/log__short/one/seed0 +../e56-hopper-faults/results/hopper/log__short/one/seed1 +../e56-hopper-faults/results/hopper/log__short/one/seed2 +../e56-hopper-faults/results/hopper/none/1.0/seed0 +../e56-hopper-faults/results/hopper/none/1.0/seed1 +../e56-hopper-faults/results/hopper/none/1.0/seed10 +../e56-hopper-faults/results/hopper/none/1.0/seed11 +../e56-hopper-faults/results/hopper/none/1.0/seed12 +../e56-hopper-faults/results/hopper/none/1.0/seed13 +../e56-hopper-faults/results/hopper/none/1.0/seed14 +../e56-hopper-faults/results/hopper/none/1.0/seed15 +../e56-hopper-faults/results/hopper/none/1.0/seed16 +../e56-hopper-faults/results/hopper/none/1.0/seed17 +../e56-hopper-faults/results/hopper/none/1.0/seed18 +../e56-hopper-faults/results/hopper/none/1.0/seed19 +../e56-hopper-faults/results/hopper/none/1.0/seed2 +../e56-hopper-faults/results/hopper/obs__f16/0.01/seed0 +../e56-hopper-faults/results/hopper/obs__f16/0.01/seed1 +../e56-hopper-faults/results/hopper/obs__f16/0.01/seed2 +../e56-hopper-faults/results/hopper/obs__f16/1.0/seed0 +../e56-hopper-faults/results/hopper/obs__f16/1.0/seed1 +../e56-hopper-faults/results/hopper/obs__f16/1.0/seed2 +../e56-hopper-faults/results/hopper/obs__f16/one/seed0 +../e56-hopper-faults/results/hopper/obs__f16/one/seed1 +../e56-hopper-faults/results/hopper/obs__f16/one/seed2 +../e56-hopper-faults/results/hopper/obs__stale/0.01/seed0 +../e56-hopper-faults/results/hopper/obs__stale/0.01/seed1 +../e56-hopper-faults/results/hopper/obs__stale/0.01/seed2 +../e56-hopper-faults/results/hopper/obs__stale/1.0/seed0 +../e56-hopper-faults/results/hopper/obs__stale/1.0/seed1 +../e56-hopper-faults/results/hopper/obs__stale/1.0/seed2 +../e56-hopper-faults/results/hopper/obs__stale/one/seed0 +../e56-hopper-faults/results/hopper/obs__stale/one/seed1 +../e56-hopper-faults/results/hopper/obs__stale/one/seed2 +../e56-hopper-faults/results/hopper/obs__swap/0.01/seed0 +../e56-hopper-faults/results/hopper/obs__swap/0.01/seed1 +../e56-hopper-faults/results/hopper/obs__swap/0.01/seed2 +../e56-hopper-faults/results/hopper/obs__swap/1.0/seed0 +../e56-hopper-faults/results/hopper/obs__swap/1.0/seed1 +../e56-hopper-faults/results/hopper/obs__swap/1.0/seed2 +../e56-hopper-faults/results/hopper/obs__swap/one/seed0 +../e56-hopper-faults/results/hopper/obs__swap/one/seed1 +../e56-hopper-faults/results/hopper/obs__swap/one/seed2 +../e56-hopper-faults/results/hopper/obs__zero/0.01/seed0 +../e56-hopper-faults/results/hopper/obs__zero/0.01/seed1 +../e56-hopper-faults/results/hopper/obs__zero/0.01/seed2 +../e56-hopper-faults/results/hopper/obs__zero/1.0/seed0 +../e56-hopper-faults/results/hopper/obs__zero/1.0/seed1 +../e56-hopper-faults/results/hopper/obs__zero/1.0/seed2 +../e56-hopper-faults/results/hopper/obs__zero/one/seed0 +../e56-hopper-faults/results/hopper/obs__zero/one/seed1 +../e56-hopper-faults/results/hopper/obs__zero/one/seed2 +../e56-hopper-faults/results/hopper/reward__f16/0.01/seed0 +../e56-hopper-faults/results/hopper/reward__f16/0.01/seed1 +../e56-hopper-faults/results/hopper/reward__f16/0.01/seed2 +../e56-hopper-faults/results/hopper/reward__f16/1.0/seed0 +../e56-hopper-faults/results/hopper/reward__f16/1.0/seed1 +../e56-hopper-faults/results/hopper/reward__f16/1.0/seed2 +../e56-hopper-faults/results/hopper/reward__f16/one/seed0 +../e56-hopper-faults/results/hopper/reward__f16/one/seed1 +../e56-hopper-faults/results/hopper/reward__f16/one/seed2 +../e56-hopper-faults/results/hopper/reward__stale/0.01/seed0 +../e56-hopper-faults/results/hopper/reward__stale/0.01/seed1 +../e56-hopper-faults/results/hopper/reward__stale/0.01/seed2 +../e56-hopper-faults/results/hopper/reward__stale/1.0/seed0 +../e56-hopper-faults/results/hopper/reward__stale/1.0/seed1 +../e56-hopper-faults/results/hopper/reward__stale/1.0/seed2 +../e56-hopper-faults/results/hopper/reward__stale/one/seed0 +../e56-hopper-faults/results/hopper/reward__stale/one/seed1 +../e56-hopper-faults/results/hopper/reward__stale/one/seed2 +../e56-hopper-faults/results/hopper/reward__swap/0.01/seed0 +../e56-hopper-faults/results/hopper/reward__swap/0.01/seed1 +../e56-hopper-faults/results/hopper/reward__swap/0.01/seed2 +../e56-hopper-faults/results/hopper/reward__swap/1.0/seed0 +../e56-hopper-faults/results/hopper/reward__swap/1.0/seed1 +../e56-hopper-faults/results/hopper/reward__swap/1.0/seed2 +../e56-hopper-faults/results/hopper/reward__swap/one/seed0 +../e56-hopper-faults/results/hopper/reward__swap/one/seed1 +../e56-hopper-faults/results/hopper/reward__swap/one/seed2 +../e56-hopper-faults/results/hopper/reward__zero/0.01/seed0 +../e56-hopper-faults/results/hopper/reward__zero/0.01/seed1 +../e56-hopper-faults/results/hopper/reward__zero/0.01/seed2 +../e56-hopper-faults/results/hopper/reward__zero/1.0/seed0 +../e56-hopper-faults/results/hopper/reward__zero/1.0/seed1 +../e56-hopper-faults/results/hopper/reward__zero/1.0/seed2 +../e56-hopper-faults/results/hopper/reward__zero/one/seed0 +../e56-hopper-faults/results/hopper/reward__zero/one/seed1 +../e56-hopper-faults/results/hopper/reward__zero/one/seed2 +../e56-hopper-faults/results/hopper/step__fill/0.01/seed0 +../e56-hopper-faults/results/hopper/step__fill/0.01/seed1 +../e56-hopper-faults/results/hopper/step__fill/0.01/seed2 +../e56-hopper-faults/results/hopper/step__fill/1.0/seed0 +../e56-hopper-faults/results/hopper/step__fill/1.0/seed1 +../e56-hopper-faults/results/hopper/step__fill/1.0/seed2 +../e56-hopper-faults/results/hopper/step__fill/one/seed0 +../e56-hopper-faults/results/hopper/step__fill/one/seed1 +../e56-hopper-faults/results/hopper/step__fill/one/seed2 +../e56-hopper-faults/results/hopper/terminated__as-trunc/0.01/seed0 +../e56-hopper-faults/results/hopper/terminated__as-trunc/0.01/seed1 +../e56-hopper-faults/results/hopper/terminated__as-trunc/0.01/seed2 +../e56-hopper-faults/results/hopper/terminated__as-trunc/1.0/seed0 +../e56-hopper-faults/results/hopper/terminated__as-trunc/1.0/seed1 +../e56-hopper-faults/results/hopper/terminated__as-trunc/1.0/seed2 +../e56-hopper-faults/results/hopper/terminated__as-trunc/one/seed0 +../e56-hopper-faults/results/hopper/terminated__as-trunc/one/seed1 +../e56-hopper-faults/results/hopper/terminated__as-trunc/one/seed2 +../e56-hopper-faults/results/hopper/terminated__drop/0.01/seed0 +../e56-hopper-faults/results/hopper/terminated__drop/0.01/seed1 +../e56-hopper-faults/results/hopper/terminated__drop/0.01/seed2 +../e56-hopper-faults/results/hopper/terminated__drop/1.0/seed0 +../e56-hopper-faults/results/hopper/terminated__drop/1.0/seed1 +../e56-hopper-faults/results/hopper/terminated__drop/1.0/seed2 +../e56-hopper-faults/results/hopper/terminated__drop/one/seed0 +../e56-hopper-faults/results/hopper/terminated__drop/one/seed1 +../e56-hopper-faults/results/hopper/terminated__drop/one/seed2 +../e56-hopper-faults/results/hopper/terminated__spurious/0.01/seed0 +../e56-hopper-faults/results/hopper/terminated__spurious/0.01/seed1 +../e56-hopper-faults/results/hopper/terminated__spurious/0.01/seed2 +../e56-hopper-faults/results/hopper/terminated__spurious/1.0/seed0 +../e56-hopper-faults/results/hopper/terminated__spurious/1.0/seed1 +../e56-hopper-faults/results/hopper/terminated__spurious/1.0/seed2 +../e56-hopper-faults/results/hopper/terminated__spurious/one/seed0 +../e56-hopper-faults/results/hopper/terminated__spurious/one/seed1 +../e56-hopper-faults/results/hopper/terminated__spurious/one/seed2 +../e56-hopper-faults/results/hopper/truncated__as-term/0.01/seed0 +../e56-hopper-faults/results/hopper/truncated__as-term/0.01/seed1 +../e56-hopper-faults/results/hopper/truncated__as-term/0.01/seed2 +../e56-hopper-faults/results/hopper/truncated__as-term/1.0/seed0 +../e56-hopper-faults/results/hopper/truncated__as-term/1.0/seed1 +../e56-hopper-faults/results/hopper/truncated__as-term/1.0/seed2 +../e56-hopper-faults/results/hopper/truncated__as-term/one/seed0 +../e56-hopper-faults/results/hopper/truncated__as-term/one/seed1 +../e56-hopper-faults/results/hopper/truncated__as-term/one/seed2 +../e56-hopper-faults/results/hopper/truncated__drop/0.01/seed0 +../e56-hopper-faults/results/hopper/truncated__drop/0.01/seed1 +../e56-hopper-faults/results/hopper/truncated__drop/0.01/seed2 +../e56-hopper-faults/results/hopper/truncated__drop/1.0/seed0 +../e56-hopper-faults/results/hopper/truncated__drop/1.0/seed1 +../e56-hopper-faults/results/hopper/truncated__drop/1.0/seed2 +../e56-hopper-faults/results/hopper/truncated__drop/one/seed0 +../e56-hopper-faults/results/hopper/truncated__drop/one/seed1 +../e56-hopper-faults/results/hopper/truncated__drop/one/seed2 +../e56-hopper-faults/results/hopper/wire-float64/1.0/seed0 +../e56-hopper-faults/results/hopper/wire-float64/1.0/seed1 +../e56-hopper-faults/results/hopper/wire-float64/1.0/seed2 diff --git a/experiments/e57-clean-eval/results/load.txt b/experiments/e57-clean-eval/results/load.txt new file mode 100644 index 0000000..9aebb6d --- /dev/null +++ b/experiments/e57-clean-eval/results/load.txt @@ -0,0 +1,2 @@ + 00:16:58 up 28 days, 13:18, 1 user, load average: 0.37, 0.25, 0.31 + 00:22:43 up 28 days, 13:24, 1 user, load average: 12.06, 8.48, 4.03 diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..f955b4d --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "halfcheetah", "fault": "none", "dose": "1.0", "seed": 0, "weights_sha256": "8261c03a5b604bf51d281f80b5a1b7388b9ecb845f7dea1f4ac8e141146886e1", "episodes": 20, "returns": [651.3234362808489, 696.2781656870862, 457.64691140055936, 719.6892388681085, 770.8939605452848, 692.4743877630106, 690.9157875583197, 664.4332815274836, 426.51696380134877, 750.943370696577, 615.6042047909582, 469.3785269595863, 662.618710156006, 724.2538544019615, 677.2488278432409, 792.2352990503962, 351.3828695739779, 638.1224243751633, 830.7964619192353, 659.4809622459012], "lengths": [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000], "mean": 647.1118822722527, "eval_s": 1.853} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed10/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed10/clean_eval.json new file mode 100644 index 0000000..82d118f --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_none_1.0_seed10/clean_eval.json @@ -0,0 +1 @@ +{"env": "halfcheetah", "fault": "none", "dose": "1.0", "seed": 10, "weights_sha256": "a14df8cd72aeddb785c16f675d6214b4c0e9b0288cb48bade6ef9832f0636e72", "episodes": 20, "returns": [908.992906528071, 798.0622444063571, 834.8269480516399, 861.7588342301656, 671.6906998393595, 690.2348618102058, 823.04533078838, 952.5620003200672, 819.7638177500404, 961.9808441514579, 315.8738334752362, 893.4219227618573, 831.871036891478, 777.5199051167675, 632.6554938342715, 819.5329361523134, 850.6075704599257, 478.26193617321405, 943.5968694686496, 730.537552264455], "lengths": [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000], "mean": 779.8398772236957, "eval_s": 1.812} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_obs__f16_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_obs__f16_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..36ec06c --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_halfcheetah_obs__f16_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "halfcheetah", "fault": "obs:f16", "dose": "1.0", "seed": 0, "weights_sha256": "bfd9adbece1d1fc237193ed37a3755a697175a1b0414fe9b25375a17252f11fa", "episodes": 20, "returns": [902.9361746388151, 895.0753539798012, 867.532404387183, 898.5785154221388, 877.250657326663, 813.8040834600864, 499.73164316170073, 942.3308807016163, 832.7886597122447, 917.1675000351388, 888.4551547697322, 879.0018400901978, 754.9256621603416, 862.072819509723, 900.6832470258539, 822.6763016953425, 881.1377377803522, 832.6380501092463, 889.7950516756217, 865.1509525223811], "lengths": [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000], "mean": 851.1866345082092, "eval_s": 1.994} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..c3e7439 --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "pendulum", "fault": "none", "dose": "1.0", "seed": 0, "weights_sha256": "ec5b8226b227f3fa58bd771b411e3673b751ec2c062b010cf0dcef60baed42bc", "episodes": 20, "returns": [-0.606069784632254, -127.87956452531162, -130.55602317528292, -126.99531136451037, -268.0517554576323, -129.28876254287457, -276.22865412728487, -274.345028433382, -131.55061270241927, -0.7529306006592864, -250.57787816515716, -0.657350151429643, -263.12665856648846, -1.6163179230989386, -288.50654566692236, -0.5924826923941123, -274.2252146814768, -131.8219735024185, -133.55651415663112, -128.6710668345104], "lengths": [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200], "mean": -146.98033575272586, "eval_s": 0.298} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed10/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed10/clean_eval.json new file mode 100644 index 0000000..b9127d9 --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_none_1.0_seed10/clean_eval.json @@ -0,0 +1 @@ +{"env": "pendulum", "fault": "none", "dose": "1.0", "seed": 10, "weights_sha256": "4b5c8f2df2310767eca9864efacb3bb23b70ebe2816d644eba3047da0f75d1fd", "episodes": 20, "returns": [-1.345055564716433, -128.99849595329638, -132.2679291750097, -128.91724704544876, -269.0226065271199, -127.58905137510965, -271.8635722322068, -142.41458330422932, -133.13320410482996, -1.4739765433139627, -246.55385814143477, -1.4626804061421903, -250.48236399838922, -2.3633394163632193, -282.21824532147986, -1.3508471159041537, -263.7392933453349, -132.77984837073848, -134.13439727987054, -130.05515861379908], "lengths": [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200], "mean": -139.10828769173685, "eval_s": 0.305} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_obs__f16_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_obs__f16_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..4748b18 --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e54-boundary-faults_results_pendulum_obs__f16_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "pendulum", "fault": "obs:f16", "dose": "1.0", "seed": 0, "weights_sha256": "6bd93edb7230cacb85550b44354f730ce295ba935b79400d7b0a0eee27feb54c", "episodes": 20, "returns": [-0.6148502847222538, -132.41961828831072, -135.25942367129457, -131.76644471085925, -266.83810574873667, -127.97829042128303, -273.76599189561944, -139.17173076058464, -136.09559530010796, -0.8128044903542682, -249.30802416340077, -0.6422430478390052, -251.32060725199315, -1.5906682832198593, -282.7034127005091, -0.5700696343421382, -261.2770329685088, -135.59073490063162, -132.7067860869225, -133.57138538569802], "lengths": [200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200, 200], "mean": -139.7001909997469, "eval_s": 0.292} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..1491ad7 --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "hopper", "fault": "none", "dose": "1.0", "seed": 0, "weights_sha256": "13daa0142e18b9b947cc1f947014d6e46222902b7e92751a277bd271f745a0ef", "episodes": 20, "returns": [3535.660787262251, 3553.1947064922574, 2317.175320043081, 3540.1566842578504, 3543.777768352139, 3551.4380178924816, 3554.7992593801555, 2450.7444275937023, 3557.471994373874, 3555.7826748504667, 3566.675281385641, 3561.5672199772544, 3165.5111868620806, 3563.062375718484, 3559.8602567082185, 3564.547527405282, 3550.415193343448, 2585.3148668086537, 3551.056190629265, 3549.79485945411], "lengths": [1000, 1000, 612, 1000, 1000, 1000, 1000, 654, 1000, 1000, 1000, 1000, 842, 1000, 1000, 1000, 1000, 684, 1000, 1000], "mean": 3368.9003299395345, "eval_s": 2.388} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed10/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed10/clean_eval.json new file mode 100644 index 0000000..de2e412 --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_none_1.0_seed10/clean_eval.json @@ -0,0 +1 @@ +{"env": "hopper", "fault": "none", "dose": "1.0", "seed": 10, "weights_sha256": "d5a3ae295d1210de734820e1c0749cceb8b96dac1c772c88decf2f1087070059", "episodes": 20, "returns": [3655.3590510916865, 3644.4871137657647, 3655.728741264313, 3661.9561155117226, 3624.861410819209, 3635.8586701972695, 3631.335030254979, 3650.8436597907303, 3652.747050004136, 3654.6272943409763, 3660.238657064603, 3656.8643740073194, 3655.8339245776524, 3660.677531334535, 3656.7611321078543, 3642.555571671143, 3660.6933156673626, 3657.475136328713, 3647.7388874930575, 3658.4614451449943], "lengths": [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000], "mean": 3651.2552056219006, "eval_s": 2.546} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_obs__f16_1.0_seed0/clean_eval.json b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_obs__f16_1.0_seed0/clean_eval.json new file mode 100644 index 0000000..0a7b7da --- /dev/null +++ b/experiments/e57-clean-eval/results/pilot/e56-hopper-faults_results_hopper_obs__f16_1.0_seed0/clean_eval.json @@ -0,0 +1 @@ +{"env": "hopper", "fault": "obs:f16", "dose": "1.0", "seed": 0, "weights_sha256": "0b88c98ace320426c458bd2e94e8459b82fedb6e663014b44a748d617b1c2cc5", "episodes": 20, "returns": [1185.3575695364227, 1449.1056958741244, 1453.5602127413536, 1252.0261061462352, 1468.015830852532, 1290.881145498214, 1285.2192677014775, 3356.463267298476, 1220.4963692535464, 1309.4698936381449, 1490.3647389104456, 1453.3022586372335, 1701.0866609159627, 1201.7808588855692, 1541.3838549545537, 2692.9326962402884, 1410.631657584418, 2586.1661345431985, 1840.5422086839878, 1480.9880026293386], "lengths": [362, 419, 420, 377, 424, 385, 384, 1000, 371, 391, 432, 419, 495, 366, 447, 805, 410, 743, 553, 430], "mean": 1633.488721526276, "eval_s": 1.221} \ No newline at end of file diff --git a/experiments/e57-clean-eval/results/registered.out b/experiments/e57-clean-eval/results/registered.out new file mode 100644 index 0000000..9c35597 --- /dev/null +++ b/experiments/e57-clean-eval/results/registered.out @@ -0,0 +1,3 @@ + 00:16:58 up 28 days, 13:18, 1 user, load average: 0.37, 0.25, 0.31 +834 runs, 12 at a time +E57_DONE diff --git a/experiments/e57-clean-eval/results/summary.txt b/experiments/e57-clean-eval/results/summary.txt new file mode 100644 index 0000000..50a52b0 --- /dev/null +++ b/experiments/e57-clean-eval/results/summary.txt @@ -0,0 +1,29 @@ +halfcheetah: band (none, seeds 10-19) mean 852.6 sd 202.5 n 10; harm bar 245.1; band seeds flagged against the other nine: 0 of 10 +hopper: band (none, seeds 10-19) mean 2717.4 sd 857.0 n 10; harm bar 146.5; band seeds flagged against the other nine: 0 of 10 +pendulum: band (none, seeds 10-19) mean -185.9 sd 26.1 n 10; harm bar -264.1; band seeds flagged against the other nine: 0 of 10 + +boundary-fault runs, seeds 0-2: harmed / runs; harmed among those silent on the curve + halfcheetah one harmed 0/63 silent on the curve 60, of them harmed 0 + halfcheetah 0.01 harmed 0/63 silent on the curve 57, of them harmed 0 + halfcheetah 1.0 harmed 28/63 silent on the curve 31, of them harmed 0 + hopper one harmed 0/69 silent on the curve 69, of them harmed 0 + hopper 0.01 harmed 0/69 silent on the curve 66, of them harmed 0 + hopper 1.0 harmed 19/69 silent on the curve 35, of them harmed 0 + pendulum one harmed 1/63 silent on the curve 63, of them harmed 1 + pendulum 0.001 harmed 2/63 silent on the curve 58, of them harmed 1 + pendulum 0.01 harmed 1/63 silent on the curve 59, of them harmed 1 + pendulum 0.1 harmed 6/63 silent on the curve 50, of them harmed 2 + pendulum 1.0 harmed 24/63 silent on the curve 25, of them harmed 0 + +faults silent on the curve at all three seeds and harmed at two or more (fault, dose: clean returns) + +log error (logged final return - clean return): min / median / max by kind + halfcheetah control n 13 -479.1 -410.4 126.6 + halfcheetah boundary n 177 -1034.8 -377.7 1521.1 + halfcheetah log n 18 -441.1 -331.5 -259.1 + hopper control n 19 -1666.7 -439.8 805.9 + hopper boundary n 201 -1959.9 -1129.5 1162.5 + hopper log n 18 -1667.7 -333.4 489.1 + pendulum control n 19 -70.0 -35.2 33.3 + pendulum boundary n 309 -450.6 -29.0 1639.4 + pendulum log n 30 -70.0 -34.2 4.0 diff --git a/experiments/e57-clean-eval/results/verdicts.txt b/experiments/e57-clean-eval/results/verdicts.txt new file mode 100644 index 0000000..48b04ee --- /dev/null +++ b/experiments/e57-clean-eval/results/verdicts.txt @@ -0,0 +1,6 @@ +V1: holds - 834 evaluations of 834 runs, 834 with 50 episodes +V2: holds - 12 weights shared by two or more runs; 0 with differing clean returns [] +P1: holds - band seeds flagged: {'halfcheetah': 0, 'hopper': 0, 'pendulum': 0} +P2: holds - harmed at one value (harmed, runs): {'halfcheetah': (0, 63), 'hopper': (0, 69), 'pendulum': (1, 63)} +P3: holds - pendulum reward:zero 1.0: [(0, -1195.55, '0.0', 1), (1, -1549.63, '0.0', 1), (2, -1639.38, '0.0', 1)] +reported: hopper obs:f16 1.0 (seed, clean, harmed, silent on the curve): [(0, 1505.25, 0, '1'), (1, 3588.14, 0, '1'), (2, 3592.49, 0, '1')] diff --git a/experiments/e57-clean-eval/run.sh b/experiments/e57-clean-eval/run.sh new file mode 100644 index 0000000..839d323 --- /dev/null +++ b/experiments/e57-clean-eval/run.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# E57 on guangzhao: evaluate every E54 and E56 final policy on a clean +# environment, 12 at a time. Writes results////seed/clean_eval.json. +# bash run.sh +set -uo pipefail +export OMP_NUM_THREADS=1 +PY="$HOME/zuogou/plugrl/bridges-venv/bin/python" +cd "$(dirname "$0")" +mkdir -p results +find ../e54-boundary-faults/results ../e56-hopper-faults/results -name model.zip \ + -not -path '*/pilot/*' -printf '%h\n' | sort > results/jobs.txt +sha256sum evaluate.py summarise.py > results/e57-src.sha256 +uptime | tee results/load.txt +echo "$(wc -l < results/jobs.txt) runs, 12 at a time" +xargs -P 12 -I{} sh -c ' + r="$1"; rel="${r#*/results/}" + "$0" evaluate.py "$r" "results/$rel" --episodes 50 > /dev/null 2> "results/$(echo "$rel" | tr / _).err" \ + && rm -f "results/$(echo "$rel" | tr / _).err" || echo "failed: $r" +' "$PY" {} < results/jobs.txt +uptime >> results/load.txt +echo E57_DONE diff --git a/experiments/e57-clean-eval/summarise.py b/experiments/e57-clean-eval/summarise.py new file mode 100644 index 0000000..8df63c3 --- /dev/null +++ b/experiments/e57-clean-eval/summarise.py @@ -0,0 +1,120 @@ +"""E57: join every clean evaluation with E54's and E56's runs.csv, and count. + + python summarise.py [results] + +Writes results/clean.csv (one row per run) and results/summary.txt. +""" + +import csv +import json +import pathlib +import statistics as st +import sys + +HERE = pathlib.Path(__file__).resolve().parent +RUNS = [HERE.parent / "e54-boundary-faults/results/runs.csv", + HERE.parent / "e56-hopper-faults/results/runs.csv"] +CONTROLS = {"none", "delay-1ms", "wire-float64"} +LOUD = {"indices:reorder", "reward:short-array"} + + +def kind(fault: str) -> str: + if fault in CONTROLS: + return "control" + if fault in LOUD: + return "loud" + if fault.startswith("log:"): + return "log" + return "boundary" + + +def load(root: pathlib.Path) -> list[dict]: + trained = {} + for path in RUNS: + for r in csv.DictReader(open(path)): + trained[(r["env"], r["fault"], r["dose"], int(r["seed"]))] = r + rows = [] + for f in sorted(root.glob("*/*/*/seed*/clean_eval.json")): + d = json.loads(f.read_text()) + t = trained[(d["env"], d["fault"], d["dose"], int(d["seed"]))] + rows.append(dict( + env=d["env"], fault=d["fault"], dose=d["dose"], seed=int(d["seed"]), kind=kind(d["fault"]), + episodes=d["episodes"], clean=round(d["mean"], 2), sha=d["weights_sha256"][:8], + # The loud faults' runs have no `strict` (no return was logged); see AMENDMENT.md. + rule=int(t["rule"]) if t["rule"] else None, strict=int(t["strict"]) if t["strict"] else None, + last=float(t["last"]) if t["last"] else None, + )) + return rows + + +def judge(rows: list[dict]) -> dict: + """Band per env (none, seeds 10-19); harmed = more than 3 s.d. below its mean.""" + bands = {} + for env in sorted({r["env"] for r in rows}): + band = [r for r in rows if r["env"] == env and r["fault"] == "none" and r["seed"] >= 10] + vals = [r["clean"] for r in band] + bands[env] = dict(mean=st.mean(vals), sd=st.stdev(vals), n=len(vals)) + # Each band seed judged against the other nine. + for r in band: + rest = [x["clean"] for x in band if x is not r] + r["harmed"] = int(r["clean"] < st.mean(rest) - 3 * st.stdev(rest)) + for r in rows: + if "harmed" not in r: + b = bands[r["env"]] + r["harmed"] = int(r["clean"] < b["mean"] - 3 * b["sd"]) + r["silent_curve"] = int(r["rule"] == 0 and r["strict"] == 0) + r["log_error"] = None if r["last"] is None else round(r["last"] - r["clean"], 2) + return bands + + +def main() -> int: + root = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else HERE / "results" + rows = load(root) + bands = judge(rows) + cols = ["env", "fault", "dose", "seed", "kind", "episodes", "clean", "harmed", "rule", "strict", + "silent_curve", "last", "log_error", "sha"] + with open(root / "clean.csv", "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=cols, lineterminator="\n") + w.writeheader() + for r in rows: + w.writerow({k: r[k] for k in cols}) + + out = [] + for env, b in bands.items(): + flagged = sum(r["harmed"] for r in rows if r["env"] == env and r["fault"] == "none" and r["seed"] >= 10) + out.append(f"{env}: band (none, seeds 10-19) mean {b['mean']:.1f} sd {b['sd']:.1f} n {b['n']}; " + f"harm bar {b['mean'] - 3 * b['sd']:.1f}; band seeds flagged against the other nine: {flagged} of 10") + out.append("") + out.append("boundary-fault runs, seeds 0-2: harmed / runs; harmed among those silent on the curve") + for env in bands: + for dose in ("one", "0.001", "0.01", "0.1", "1.0"): + d = [r for r in rows if r["env"] == env and r["dose"] == dose and r["kind"] == "boundary" and r["seed"] < 10] + if not d: + continue + s = [r for r in d if r["silent_curve"]] + out.append(f" {env:12} {dose:6} harmed {sum(r['harmed'] for r in d):3}/{len(d):<3} " + f"silent on the curve {len(s):3}, of them harmed {sum(r['harmed'] for r in s):3}") + out.append("") + out.append("faults silent on the curve at all three seeds and harmed at two or more (fault, dose: clean returns)") + for env in bands: + b = bands[env] + for (fault, dose) in sorted({(r["fault"], r["dose"]) for r in rows if r["env"] == env and r["kind"] == "boundary"}): + d = [r for r in rows if r["env"] == env and r["fault"] == fault and r["dose"] == dose and r["seed"] < 10] + if len(d) == 3 and all(r["silent_curve"] for r in d) and sum(r["harmed"] for r in d) >= 2: + z = [round((r["clean"] - b["mean"]) / b["sd"], 1) for r in d] + out.append(f" {env:12} {fault:20} {dose:6} clean {[r['clean'] for r in d]} in band s.d. {z}") + out.append("") + out.append("log error (logged final return - clean return): min / median / max by kind") + for env in bands: + for k in ("control", "boundary", "log"): + e = [r["log_error"] for r in rows if r["env"] == env and r["kind"] == k and r["log_error"] is not None] + if e: + out.append(f" {env:12} {k:9} n {len(e):3} {min(e):9.1f} {st.median(e):9.1f} {max(e):9.1f}") + text = "\n".join(out) + (root / "summary.txt").write_text(text + "\n") + print(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/e57-clean-eval/verdicts.py b/experiments/e57-clean-eval/verdicts.py new file mode 100644 index 0000000..12de40b --- /dev/null +++ b/experiments/e57-clean-eval/verdicts.py @@ -0,0 +1,62 @@ +"""E57: judge PROTOCOL.md's checks and predictions from results/clean.csv. + + python verdicts.py [results] # prints, and writes results/verdicts.txt +""" + +import csv +import pathlib +import sys +from collections import defaultdict + +HERE = pathlib.Path(__file__).resolve().parent + + +def main() -> int: + root = pathlib.Path(sys.argv[1]) if len(sys.argv) > 1 else HERE / "results" + rows = list(csv.DictReader(open(root / "clean.csv"))) + for r in rows: + r["seed"], r["harmed"], r["episodes"] = int(r["seed"]), int(r["harmed"]), int(r["episodes"]) + r["clean"] = float(r["clean"]) + out = [] + + jobs = (root / "jobs.txt").read_text().split() + full = sum(r["episodes"] == 50 for r in rows) + out.append(f"V1: {'holds' if full == len(jobs) == len(rows) else 'FAILS'} - " + f"{len(rows)} evaluations of {len(jobs)} runs, {full} with 50 episodes") + + by_sha = defaultdict(set) + for r in rows: + by_sha[(r["env"], r["sha"])].add(r["clean"]) + shared = [k for k in by_sha if sum(1 for r in rows if (r["env"], r["sha"]) == k) > 1] + bad = [k for k in shared if len(by_sha[k]) > 1] + out.append(f"V2: {'holds' if not bad else 'FAILS'} - {len(shared)} weights shared by two or more runs; " + f"{len(bad)} with differing clean returns {bad[:5]}") + + p1 = {} + for env in sorted({r["env"] for r in rows}): + p1[env] = sum(r["harmed"] for r in rows if r["env"] == env and r["fault"] == "none" and r["seed"] >= 10) + out.append(f"P1: {'holds' if max(p1.values()) <= 1 else 'FALSIFIED'} - band seeds flagged: {p1}") + + p2 = {} + for env in sorted({r["env"] for r in rows}): + d = [r for r in rows if r["env"] == env and r["dose"] == "one" and r["kind"] == "boundary" and r["seed"] < 10] + p2[env] = (sum(r["harmed"] for r in d), len(d)) + ok = all(h <= 0.10 * n for h, n in p2.values()) + out.append(f"P2: {'holds' if ok else 'FALSIFIED'} - harmed at one value (harmed, runs): {p2}") + + d = [r for r in rows if r["env"] == "pendulum" and r["fault"] == "reward:zero" and r["dose"] == "1.0" and r["seed"] < 10] + ok = len(d) == 3 and all(r["harmed"] for r in d) and all(r["last"] and float(r["last"]) == 0.0 for r in d) + out.append(f"P3: {'holds' if ok else 'FALSIFIED'} - pendulum reward:zero 1.0: " + f"{[(r['seed'], r['clean'], r['last'], r['harmed']) for r in d]}") + + h = [r for r in rows if r["env"] == "hopper" and r["fault"] == "obs:f16" and r["dose"] == "1.0" and r["seed"] < 10] + out.append(f"reported: hopper obs:f16 1.0 (seed, clean, harmed, silent on the curve): " + f"{[(r['seed'], r['clean'], r['harmed'], r['silent_curve']) for r in h]}") + text = "\n".join(out) + (root / "verdicts.txt").write_text(text + "\n") + print(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())