Repository navigation
Conversation
(cherry picked from commit ba4509b)
(cherry picked from commit 9d2d649)
(cherry picked from commit 0f1af60)
(cherry picked from commit 271f027)
(cherry picked from commit bb9567f)
(cherry picked from commit 76ec11b)
(cherry picked from commit eaf180d)
(cherry picked from commit 067b250)
…ine eval (cherry picked from commit 3050d4e)
(cherry picked from commit 9965f82)
(cherry picked from commit 9751782)
(cherry picked from commit 44cddce)
(cherry picked from commit ce8543d)
(cherry picked from commit b86931a)
…val for E1 (M,16) tokens Aidan, 2026-09-15: fold the rl2 YAM stationery uploads into the ABC stationery set, hold the middle Step 0 tempo tertile out of training, and train two rollout-ready models -- the lab time-indexed recipe and the arcdur arc row -- scored with Aniketh's open-loop segment evaluator (open_loop_sim, 4481e3d). EVALUATOR open_loop_sim only knew the lab (M+1, 14) / (2M, 14) ARC layouts. Add token_layout (lab | e1_dur | e1_logdur | e1_profile) and truncate_e1_wide_token so an E1 wide (M, 16) prediction is truncated by waypoint rows (each row carries its own timing) and detokenized with TokenizeBimanualArcLengthE1 built exactly as robot/arc_decoder.BimanualArcDecoder builds that layout -- the scored decode is the deployed decode. The scoring loop is unchanged; lab-layout behaviour is unchanged (token_layout defaults to lab). Tests: truncation keeps waypoint/timing rows, and a truncated e1_dur decode equals the robot decoder's first 25 steps. CONFIGS (generated from the split manifest, not hand-edited) data/abc_arc/stationery_midtempo_{time,arcdur}.yaml and eval-only configs for val / test_mid / test_in, plus a *_smoke_* set on a 4-episode subset. Episode-hash frozensets; the task clause admits 'sort the stationery into containers' (ABC) and 'organize_stationary' (rl2 YAM station) -- no SQL relabel. time = Ryan's shorts_extreme_bc data block (Yam lab keymap, raw 100-frame eef_frame chunk); arcdur = shorts_extreme_arcdur (bimanual_arc, D 0.40 m, M 100, (100, 16)). experiment/abc_arc/stationery_midtempo_{time,arcdur}.yaml: the ported shorts-extreme recipe (e1/hpt_flow_wrists, 30k steps, batch 32, 1 GPU, checkpoint every 5k) with evaluator eval_open_loop_sim (10 val episodes every 5k in training, whole episodes). SCRIPTS (scripts/e1/) build_stationery_midtempo.py split v2: train = slow + fast tertiles minus val and test_in; val / test_mid identical to v1 make_stationery_midtempo_configs.py, make_smoke_manifest.py stationery_midtempo_{train,eval}.sbatch, stationery_midtempo_smoke.sh, stationery_midtempo_tests.sbatch Phoenix gpu-h200 / cpu-small launchers harvest_stationery_midtempo.py select on val open-loop paired MSE, report selected + final on test_in / test_mid package_stationery_midtempo.py rollout artefact: checkpoint (sha256), norm stats with normalizer_state, Hydra config, filled yam_rollout.yaml (base_T_model from the rl2 calibration, e1_dur decoder for arcdur), manifest replay_check_stationery_midtempo.py load the package via load_graph_policy and replay a recorded rl2 episode offline Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit ed120944eab5a485e934f104b1047143a19f2e62)
(cherry picked from commit b2f356359784fe68640d2edde31ca88f91aae7cd)
(cherry picked from commit ec32507b3eba2bb02c2680e4d32a8fc39f28d0d7)
(cherry picked from commit 902de75b813f792040239f1ed71b04d82272de45)
…C arc pretrain configs Loader workers are re-forked at every epoch boundary by default and this stack deadlocks there: job 13307279 wedged at step 99 of a 100-step epoch, workers idle in do_poll, main thread in futex_wait, after limping at ~19 s/step. persistent_workers=true crosses three boundaries cleanly at 4.2 it/s (job 13308744). Also adds the ABC-only arc pretrain (Elmo ckpt is variant=time, so it has no arc pretraining) and a per-job hydra run dir. (cherry picked from commit f189f60fc73499078c1bf3fe2ffe17203f58e0d8)
…ent base (ABC+RL2)
Fills the {time,arc} x {scratch,base} matrix. timepre_abc is the ABC-only time twin of arcpre_abc (Elmo ckpt is not a substitute: it trained on ABC + old RL2, 2264 eps). arcpre_abcrl2 mirrors the DATA POOL of Elmo checkpoint run (ABC + RL2 organize_stationary_updated, 2287 eps) with the arc-duration target, so a later RL2 fine-tune from it parallels the time arm.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit d810ec9861a089447705becafdee7c39d1c6327d)
…re all landing in a shared None/checkpoints) Copied from Elmo resolved config where output_dir had already resolved to null; dirpath then rendered as literal None/checkpoints relative to the repo root, shared by every run. Lightning versioned the collisions (-v1/-v2) so nothing was lost, but ownership was only recoverable by mtime. Each job now checkpoints under its own hydra run dir. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit a07d692d3e4d097a02c83209ce5413727bf1e194)
… land in None/checkpoints Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 9bfee8d17c32aaf784ae6180c545a73da8af1b9e)
… cells, ABC-only towel bases; towel sync
Aidan 09-18: never mix ABC and RL2 unless told to. From-scratch = RL2 YAM data of the task only; pretraining bases = ABC only. Adds scratch_rl2_{stationery,towels}_{time,arcdur}, base_abc_towels_{time,arcdur}, towel data configs, and sync_towels (rl2 task is fold_towels after the 09-18 relabel). arcpre_abcrl2 (ABC+RL2) violated the rule and was cancelled before it ran.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 19f18b9151fb25eab9a10d37a63ce3ec543a6aba)
…one submit helper Untyped gres lets PACE expand the partition list (a job once landed on a V100, no bf16), so grid_submit.sh excludes every non-H100/H200 GPU node and the job exits 42 if the GPU is not H100/H200. norm_stats.save_cache_dir was null (inherited from a resolved config), so no run was saving the stats a later fine-tune must reuse; now saved under the run dir. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 1dd56aecc1a8bc48e222294c5d84bbe3f24a5c8a)
… tasks; pinned-pool norm-stats recompute Aidan 09-18 asked for two from-scratch runs on BOTH abc and rl2 data with the velocity arc tokenizer (variant arcvel -> profile, (100,16) = 14 canonical + per-arm path speed). This is the explicit exception to the never-mix rule and applies to these two configs only. Also commits the pinned 2089-episode data config + sbatch used to recompute arcpre_abc norm stats exactly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 3d97d5ef827430ca797f0109711291c682197889)
…licitly authorised), both tasks; exclude hung H200 node Same pool filter and valid_ratio as the *_mix_arcvel configs; only the action target differs (14-D time chunk). atl1-1-03-018-14-0 hung job 13325462 in its first backward for 8.5 h (GPU 100 %, zero steps) -> added to the exclude list. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit ca5d2f1ff052bf64aa381ebdb198cabdb81b4e3c)
…h runs; arc evaluator on group_balance_sampler: WeightedRandomSampler over MultiDataset.index_map, each named group gets a fixed share of draws, uniform over frames within a group; membership from explicit episode lists (198 RL2 episodes per task). Hooked into MultiDataModuleWrapper.train_dataloader next to anchor_sampler. Four scratch_mix5050_* experiments supersede scratch_mix_* (cancelled); the arcvel pair now runs BimanualTempoEval so Valid/* is logged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 5af693db2487361b010c77135fe223372e38ce51)
…aws; end-to-end test against SQL ground truth Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 88e2e5a345383eb41e68fb6eb5ee96d5bff3bd57)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 6ddf673eadd9e3103b7d1cd7c4e559922295d9f9)
The venv interpreter links to the image's /usr/bin/python3.12, so test -x fails outside the container (the login node, where in-job steps are launched). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A relayed last.ckpt already in the run dir makes Lightning's rolling checkpoint save as last-v1.ckpt, so resuming from last.ckpt by name would rewind a requeued run to the relayed step. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…launcher)
stattempo_eval_{time,arcdur}: the training experiment with its inline BimanualTempoEval block replaced
by eval_open_loop_sim at the stationery mid-tempo settings (execute_fraction 0.25, DTW off, no videos),
because Hydra merges an experiment's inline evaluator block over any evaluator group selected on the
command line and OpenLoopSimEval forwards unknown kwargs to a parent that rejects them.
stattempo_eval.sbatch scores checkpoints on val_slow / val_medium / val_fast with the run's own norm stats.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…truth; stattempo_eval EXTRA overrides episode_dump_dir (default None = nothing written, metrics unchanged) saves one .npz per scored episode: segment starts, per-step frame index, and the executed (T, 14) prediction and ground truth exactly as scored. Used to plot per-model error/gripper timelines for the stationery tempo runs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Aidan 2026-09-27: two more arcdur models on the RL2 fold data, HPT at the larger
size and a DP at the same size. model/e1/hpt300_flow_wrists_arcdur = our arcdur
model at the lab HPT-300M dims (840 / 19 blocks x 10 heads / flow 6x320);
model/e1/dp300_wrists_arcdur = the lab graph DP-300M (UNet [512,1024,2048],
64-kp encoders) with front + both wrist cameras and the E1 arcdur (100, 16)
action. Experiments scratch_rl2_towels394_arcdur_{hpt300,dp300}_lambda are the
arcdur_lambda twin with only the model changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… pools train_slowpace = the 229 organize_stationary episodes (Elmo 200 + Aidan 29, collected at an instructed slow pace) minus the shared held-out set; train_all = every usable rl2 stationery episode (427) minus the same 24 held-out episodes, so both stay comparable with the tempo-bucket runs. Recipe unchanged. Generator gains --pools and takes the episode count from the manifest; check script takes a pool list (4/4 train + valid sets match the manifest). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…coders dp300pt_wrists_arcdur = dp300_wrists_arcdur with pretrained: true and norm_layer: batch (the group swap would re-initialise the pretrained norm layers; no EMA, so BN is safe). Experiment twin scratch_rl2_towels394_arcdur_dp300pt_lambda, W&B id ..._20260928_s42. Aidan 2026-09-28: all models should be inited from the pretrained resnet weights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings aidan/towels394-lambda (the RL2 fold_towels 394-episode campaign on the Lambda loaner) onto the consolidated branch next to the stationery tempo-bucket line, so one branch carries every run's code: - Ryan Punamiya's weighted dataset mixtures (PR #141, merged as-is) for the 40/60 ABC/RL2 mix runs - towels394 time / arcdur / arcvel / arcdur-w400 configs (Phoenix + Lambda twins), 40/60 mix configs - the yam_source_frames window knob, pooled two-source norm stats (trainHydra _source_embodiment) - Lambda pack_train.sbatch / setup_runtime.sbatch - HPT-300M, DP-300M and DP-300M-pretrained arcdur model + experiment configs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 12 towels394 Lambda experiment twins and 8 data twins become defaults-inheritance children holding only their delta (name, W&B id, tags, variant, model/data choice, dims, window). Parents: scratch arcdur, mix4060 arcdur, norm_mix4060 arcdur; data: towels_rl2_394 arcdur, towels_mix4060 arcdur, normpool arcdur. All 64 yam_arc_grid experiments compose byte-identical to before (sorted-key dump of every composed config, 0 differences). dp300pt stays a full file: its encoders sit in pipeline.stages[0], a list, which OmegaConf replaces on merge. trainHydra: Counter for the shared-embodiment check; bimanual_arc: one expression for the YAM window; pack_train: one-line resume. 111 tests pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
76 near-duplicate configs under experiment/yam_arc_grid and data/abc_arc (bucket, variant, task and Lambda twins, eval and smoke sets) are now `defaults: [<sibling>, _self_]` plus only the keys that differ, the pattern a56cf34 used for towels394. Parents are always in the same run family (the names differ only in variant / bucket / task / eval-set tokens), and a smoke or Lambda config is never the parent of one that is not. Headers are unchanged. The two config generators (make_stationery_tempo_configs.py, make_stationery_midtempo_configs.py) are removed: those runs are finished, the configs are the record, and re-running them would write the full copies back. The data headers that said "do not edit by hand" now point at git history. Verified: every yam_arc_grid experiment (64) and every abc_arc data config (103) composes byte-identically before and after (hydra compose, unresolved). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…key_map Under keymap_mode: cartesian the YAM window has been a code default (Yam.ACTION_HORIZON = 100 frames, 3.3 s) that ignores the yaml `horizon: 200`. Every grid run was trained on that default. Writing it into the configs makes the window visible in the saved and W&B configs, and keeps these configs reproducing their runs if the default or the horizon handling changes. The w400 cell keeps its explicit 400. Verified: 116 composed configs change, and in each one the only difference is the added key (238 YAM cartesian key_map blocks now read 100, the w400 cell's 4 read 400). Training tensors are unchanged because 100 is what the code already used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 2, 2026
Collaborator
Author
ElmoPA
changed the title
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Folded into #198; review the combined net diff there. This superseded PR is closed without merging to main.
Integrate #160's weighted and homogeneous graph execution, E1/YAM grids and campaign tooling with the canonical ARC and checkpoint-bound graph lifecycle. Preserve sample-mean mixture weighting, pooled/restored normalization, explicit stacked/wide/E1 layouts, independent per-arm clocks, and the validated YAM 100/400-frame override.
The branch points at the original assembled code commit
96df46f1427cd08e0dd64ac534af133ee37b193c. Its merge retains #160 tipbef887c4and all 34 source PR heads as ancestors. Review this diff against the canonical ARC integration layer below it; #198 adds only consolidation documentation and evidence above this code snapshot.Validation recorded for this exact code commit: 1,993 CPU tests passed with zero failures/skips; all 439 YAML/constructor contexts, installed-wheel checks, real two-process CPU Gloo checks, Ruff lint/format, lockfile and whitespace checks. Seventeen fixed-index YAM outputs match pinned #160, and all 64 grid recipes retain their resolved networks, optimizers, schedules and normalization. Stack construction additionally verified exact commit identity, unchanged final tree, and source-head ancestry.
Keep this PR in draft. The real-weight/data OSMO L40/L40S gate remains pending, as do applicable fresh CI/review. GPU jobs, tokenizer transfer, checkpoint conversion, deployment and merging are outside this stack-creation task.
Stack order: #150–#159 → #199 → #200 → #201 → #198.