Skip to content

Admit local LeRobot v3 datasets as training input (S-48) - #354

Merged
rylinjames merged 2 commits into
mainfrom
feat/local-dataset-admission
Oct 12, 2026
Merged

Admit local LeRobot v3 datasets as training input (S-48)#354
rylinjames merged 2 commits into
mainfrom
feat/local-dataset-admission

Conversation

@rylinjames

@rylinjames rylinjames commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

What

tether finetune --dataset-root now accepts a raw local LeRobot v3 dataset for ACT (from scratch) and SmolVLA LoRA. Before this, it accepted only a Studio export receipt. Studio's Train page can then train on a recorded or imported dataset without pushing it to the Hub.

How admission works

  • describe_lerobot_v3 (listing): reads meta/ only. It checks info.json, features, stats, tasks and the episode index. Tests prove it never opens a data or video file.
  • verify_lerobot_v3 (review and launch): runs the full check.
    • Every data row: index, episode, frame, task and timestamp, plus vector widths.
    • Shard placement, and safe image paths.
    • Decoded video frame count and duration.
    • Then a SHA-256 over every retained file. Launch re-verifies against the reviewed hash.
  • Video frames use LeRobot's own rule. The query is from_timestamp plus the row timestamp. The nearest decoded PTS must be less than tolerance_s=1e-4 away, compared in float32 as torch.cdist does. A float64 version disagreed with LeRobot on 8% of near-edge samples at long-shard timestamps (200k samples against torch).
  • Local configs refuse a tolerance_s override, so admission and training use the same tolerance.
  • SmolVLA: the raw path binds only max_state_dim/max_action_dim padding, since LeRobot derives features from the dataset.
  • Width-1 vectors stored as scalars (LeRobot 0.5.1) are admitted.
  • Every refusal names its field, for example episode_index[2].videos/observation.images.wrist/to_timestamp.

Evidence

  • Full suite after merging main (b1a8196): 4194 passed, 126 skipped, 0 failed. The video tests need PyAV and ran in the LeRobot 0.5.1 runtime: 148 passed across the v3 and local-dataset files.
  • Differential oracle: I wrote video datasets with LeRobot itself at 7, 10, 15, 25, 30, 50 and 60 fps, with uneven episode lengths and shared shards. Tether admits all 7, and LeRobotDataset decodes every frame of all 7.
  • Mutation checks: I removed each of these in turn, and each one fails at least one test: the row-timestamp query, the float32 rule, the tolerance_s refusal, the per-frame check, the frame-count check, the metadata-duration check, the decoded-duration check, and metadata-only listing.
  • Real ACT and SmolVLA runs on CPU, offline, through run_finetune. SmolVLA LoRA on lerobot/smolvla_base@d9f33c9 took 2 steps from a local 3-camera v3 video dataset, 81.1 s wall in total. Losses were 0.086 then 0.370, and it wrote an adapter checkpoint.

Not merged; for review.

🤖 Generated with Claude Code

rylinjames and others added 2 commits October 12, 2026 02:58
@rylinjames @claude
Admit local LeRobot v3 datasets as training input (S-48)
08b1ded 
`tether finetune --dataset-root` now admits a raw local LeRobot v3 dataset
for ACT (from scratch) and SmolVLA LoRA, not only a Studio export receipt.
Admission checks info.json, features, stats, tasks, the episode index,
every data row, and decoded video timestamps, then binds a SHA-256 over
every retained file. Training runs with HF offline flags.

- describe_lerobot_v3: metadata-only check for listings; reads meta/ only.
- verify_lerobot_v3: full admission. Video frames are checked with
  LeRobot's own rule: query = from_timestamp + the row timestamp, nearest
  decoded PTS distance < tolerance_s (1e-4), compared in float32 as
  torch.cdist does. float64 disagreed with LeRobot on 8% of near-edge
  samples at long-shard timestamps.
- Local configs refuse a tolerance_s override so admission and training
  use the same tolerance.
- SmolVLA raw path binds only max_state/action_dim padding; LeRobot
  derives features from the dataset.
- Width-1 vectors stored as scalars (LeRobot 0.5.1) are admitted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rylinjames
Merge remote-tracking branch 'origin/main' into feat/local-dataset-ad…
0f3c425 
…mission
@rylinjames
rylinjames merged commit cdb94dc into main Oct 12, 2026
5 of 9 checks passed
@rylinjames
rylinjames deleted the feat/local-dataset-admission branch October 12, 2026 11:09
rylinjames added a commit that referenced this pull request Oct 12, 2026
@rylinjames @claude
Parity vehicles: torch>=2.8 (onnx-diagnostic patches need it); LoRA s…
165102a 
…tub shape

The pi0/pi0.5/GR00T vehicles had been changed to torch==2.7.1. Every
onnx-diagnostic 0.9.x release (0.9.0-0.9.3, checked) patches
torch._export.non_strict_utils._constrain_user_specified_dimhint_range,
which exists from torch 2.8.0 and not in 2.7.1, so
torch_export_patches(patch_transformers=True) raises AttributeError before
any export: the paid GPU runs would have failed. Reproduced locally on
torch 2.7.1 + onnx-diagnostic 0.9.3. The vehicles now take torch>=2.8,<2.12
(lerobot 0.6.1's cap); uv resolves torch 2.11.0, transformers 5.3.0, no
pyyaml-include, for linux x86_64 py3.12.

test_lora_argv_parser: the local-dataset stub returns the real
verify_local_export shape (receipt key) that lerobot_v3.verify_local_dataset
reads since #354; three cases failed whenever lerobot was installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rylinjames rylinjames mentioned this pull request Oct 12, 2026
Merged
rylinjames added a commit that referenced this pull request Oct 13, 2026
@rylinjames @claude
LeRobot 0.6.1 + transformers 5.5.4: export port, golden reference, GP…
e196b79 
…U-qualified pi0/pi0.5 (#355)

* Move LeRobot pin to 0.6.1 with draccus 0.11.6 (Studio ADR 0003)

draccus 0.11.6 drops the GPL-3.0+ pyyaml-include that draccus 0.10.0 (required
by lerobot 0.5.1) pulled in. transformers stays 5.3.0 by override, so the
monolithic exporter's masking-API gate is unchanged.

- One QUALIFIED_LEROBOT_VERSION constant (src/tether/_lerobot_version.py) for
  the monolithic exporter gate, local dataset admission and preflight text.
- Local dataset admission pins the LeRobot 0.6.1 wheel, source commit
  7e241bd and 13 reviewed upstream file hashes (diffs reviewed in Studio
  docs/research/verification/lerobot-06-implementation-20261011/), plus the
  Studio v2 statistics adapter hash.
- lerobot.types -> lerobot.lerobot_types; GR00T probe imports follow N1.7.
- Resume refuses LeRobot 0.5.x train configs with a typed
  cross-version-resume-unsupported code.
- pi0/pi0.5/GR00T parity vehicles pin the 0.6.1 runtime, write to a new
  volume subtree and take --tether-revision/--watchdog-seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review fixes for the LeRobot 0.6.1 pin

- Keep the base torch/numpy requirements as on main (torch>=2.1.0,
  numpy>=1.24.0 off aarch64). The exact torch==2.7.1 / numpy==2.2.6 base pins
  were outside ADR 0003 (which pins Studio's evaluation runtime, not every
  public install) and LeRobot 0.6.1 already declares torch<2.12, numpy<2.3.
- accelerate floor >=1.14 in the explicit SmolVLA dependency sets, matching
  lerobot 0.6.1's accelerate-dep extra (>=1.14,<2) instead of a looser 1.10.
- Resume: an unreadable train_config.json raises ResumeConfigurationError
  instead of a bare JSONDecodeError; test added.
- New test: WRITER_FILES/NORMALIZATION_FILES equal the installed bytes when
  the qualified LeRobot is present (a wrong pin previously passed every
  Tether test; mutation-checked).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Parity vehicles: torch>=2.8 (onnx-diagnostic patches need it); LoRA stub shape

The pi0/pi0.5/GR00T vehicles had been changed to torch==2.7.1. Every
onnx-diagnostic 0.9.x release (0.9.0-0.9.3, checked) patches
torch._export.non_strict_utils._constrain_user_specified_dimhint_range,
which exists from torch 2.8.0 and not in 2.7.1, so
torch_export_patches(patch_transformers=True) raises AttributeError before
any export: the paid GPU runs would have failed. Reproduced locally on
torch 2.7.1 + onnx-diagnostic 0.9.3. The vehicles now take torch>=2.8,<2.12
(lerobot 0.6.1's cap); uv resolves torch 2.11.0, transformers 5.3.0, no
pyyaml-include, for linux x86_64 py3.12.

test_lora_argv_parser: the local-dataset stub returns the real
verify_local_export shape (receipt key) that lerobot_v3.verify_local_dataset
reads since #354; three cases failed whenever lerobot was installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Monolithic export: refuse torch<2.8; make dispatch test deterministic

onnx-diagnostic 0.8.11 and 0.9.0-0.9.3 patch
torch._export.non_strict_utils._constrain_user_specified_dimhint_range
on entering torch_export_patches(); the symbol first exists in torch
2.8.0. Reproduced: entering the patches fails on torch 2.7.1 with
AttributeError and succeeds on 2.8.0 and 2.11.0. _require_monolithic_deps
admitted torch 2.7.1, so every pi0/pi0.5/SmolVLA/GR00T export died mid
export instead of being refused. The gate now lists torch>=2.8 as a
missing dependency, and the [monolithic] extra declares the same floor
(the jetson extra is left alone: it does not export locally).

test_dispatch_by_model_id called the real SmolVLA exporter and expected
an ImportError, so it started a full export whenever the extras were
installed and failed on any torch<2.8 interpreter that had them. It now
stubs the family exporters and asserts routing for smolvla/pi0/pi05.
New test for the torch floor; removing the check fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Port patched export paths to LeRobot 0.6.1 shared VLA helpers

LeRobot 0.6.1 removed PI0Pytorch/PI05Pytorch._prepare_attention_masks_4d
(now lerobot.policies.common.vla_utils.prepare_attention_masks_4d), which
broke the pi0 GPU parity export. Port every patched prefix/denoise path
(pi0, pi0.5 monolithic, decomposed, SnapFlow pi0/pi0.5/state-out, training
adapters, standalone Modal scripts) to the shared helper. Drop the 0.5.1
embed_image sqrt(hidden_size) shim (0.6.1 native no longer scales). SnapFlow
pi0 clones the prefix cache like native; SnapFlow export freezes cache
updates and forwards explicit state for state-out students.

Adds tests/test_lerobot06_patched_forward.py: tiny real LeRobot 0.6.1
models, patched outputs vs native unpatched forward at rtol=atol=1e-6, plus
torch.export graphs and a private-API audit. Audit and review notes in
docs/research/verification/lerobot-06-export-port-20261012.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record: LeRobot 0.6.1 on transformers 5.3 is mis-scaled; pi0 GPU parity failed

GPU re-run at 5fb92df: pi0 cosine 0.465 / max_abs 0.403 (thresholds 0.999 /
0.1). LeRobot 0.6.1 requires transformers>=5.4; on the pinned 5.3.0 its
native image and language embeddings lose the openpi sqrt(hidden_size)
scale. The exporter now matches that native forward; the receipt harness
still applies the old image shim. Correction appended to the port audit.
Not for merge until the transformers pin decision is made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Move to transformers 5.5.4 (LeRobot 0.6.1's range); add pre-migration golden

LeRobot 0.6.1 requires transformers>=5.4,<5.6. On 5.3.0 its pi0/pi0.5 image
and language embeddings lose the openpi sqrt(hidden) scale, which failed GPU
parity at cosine 0.47 while the CPU tests passed against the same mis-scaled
native forward.

- Pins: monolithic/native/eval-local/jetson extras -> transformers==5.5.4,
  base <5.6, [tool.uv] 5.3.0 override removed; exporter gate driven by
  QUALIFIED_TRANSFORMERS_VERSION and refuses 5.3.0.
- Golden reference: tiny pi0/pi0.5 fixtures captured on LeRobot 0.5.1 +
  transformers 5.3.0 + Tether cdb94dc; native, patched export forward and the
  real onnx-diagnostic ONNX export are compared to it (negative control on
  0.6.1 + 5.3.0 fails, prefix norm ratio 1/sqrt(hidden)).
- Receipt harnesses use native LeRobot as the reference (shims removed).
- PaliGemmaBackbone.embed_text_openpi: table lookup * sqrt(hidden), so the
  Triton fast path and Pi0VLA/Pi05VLA do not double-scale on 5.4+.
- Parity vehicles: retries=0, subprocess phase timeouts from measured times
  instead of the SIGKILL watchdog (Modal re-ran the crashed input), restart
  guard, provenance-checked export reuse, new volume root, pi0.5 on A100-40GB.

Full suite: S11 4204 passed; LeRobot 0.6.1 + tf 5.5.4 4262 passed; export
venv subset 73 passed; 0 failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Format two files the quality ratchet flagged as new debt

ruff format on src/tether/finetune/resume.py and
tests/test_finetune_local_dataset.py (lines added earlier on this branch).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
rylinjames added a commit that referenced this pull request Oct 13, 2026
@rylinjames @claude
CI: add pyarrow to the dev extra so the LeRobot v3 finetune tests col…
143ca13 
…lect (#356)

Since #354 tests/test_finetune_lerobot_v3.py needs pyarrow; the CI install
omitted it, so collection failed and Pytest CI ran no tests on main.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rylinjames