Skip to content

Open the ZED take in front of the push, not the whole episode - #139

Open
amburger66 wants to merge 2 commits into
masterfrom
domino-record-from-push
Open

Open the ZED take in front of the push, not the whole episode#139
amburger66 wants to merge 2 commits into
masterfrom
domino-record-from-push

Conversation

@amburger66

@amburger66 amburger66 commented Aug 18, 2026

Copy link
Copy Markdown

Update: second commit, and the lead-in question is settled

f4873c4 draws the prompt boxes at the recording boundary instead of at run
start. It belongs here because THIS PR is what makes it necessary: the boxes are
SAM-2 prompts applied to the take's first frame, and while the take began at the
reset, "the arrangement at run start" and "the arrangement the first frame shows"
were the same thing. Recording from the Push separates them.

Observed on run_20260818_140211: boxes.json written at 14:02:37 before any
motion, take opened at 14:05:37 after the prologue had picked and placed two
dominoes. Two of the four prompts pointed at bare table. Nothing detected it
-- stage 2 does not report an empty box, it fits a mask to whatever is inside
the one it was given. A human caught it in the overlay, which is not a detection
mechanism.

The draw now happens after the prologue ships and before the take opens: the arm
is at rest, the row is final, and it is the only moment a human can draw boxes
that match the take. With no record_from_option the draw stays at run start,
which was already correct there. ensure_boxes is idempotent, since the new
call site runs every episode and a drag window per take is what
pick_boxes_at_start exists to avoid.

It trades rather than removes a cost: the human is needed ~50 s into the first
episode instead of before it. The unattended alternative -- projecting the
twin's geometry into image space -- needs no human and is considerably more work.

The lead-in: not needed, and for a better reason than "we chose to skip it"

The body below records two hazards taken deliberately, and says a lead-in would
fix the first. That is now known to be wrong, and the correction is
evidence-based rather than a change of mind.

Re-running stage 4 with visibility gates from 0.6 down to 0.2 closed the pushed
domino's occlusion gap from 1.4 s to 4 frames -- and its onset still swung
780 ms non-monotonically, against propagation intervals of 117-283 ms. The
reference onset is noise-limited, not occlusion-limited, so buying more
visibility with a lead-in would not have bought a usable onset.

The fix being pursued instead is excluding the pushed domino from interval
scoring, which is not on any branch yet.

The aliasing hazard is untouched by any of this and still stands.

#137 has merged

The "do not merge ahead of #137" caveat below is resolved -- it is on master as
b802ae3, and this branch sits above it.

tests/pybullet_helpers 170 passed on the current head.


Third in the stack: #137 (id matching) -> #138 (pipeline speed) -> this.

Records only the part of the episode that is actually scored.

Why

On run_20260818_092302 the first topple onset landed 107 s into a 131 s
track
. Everything before it was the pick-and-place that arranges the row --
roughly 80% of the video, none of it evidence. Post-processing scales with
frames, so that 80% is paid twice: once recording it, once fitting poses to it.

Push + Wait is about 25-30 s against 138 s, so this is ~4.5-5x fewer frames
-- a larger lever than everything in #138 combined.

What it does

real_robot_record_from_option names the option the take opens in front of.
_ship_episode sends the prologue as one unrecorded request, opens the take,
then sends the rest.

This does not give back what open-loop batching bought. The twin has already
simulated every option by the time _ship_episode runs, so splitting the
shipment costs one controller round trip and not a planner call. Splitting is
safe for the same reason per-boundary shipping was: _split_actions restarts its
gripper tracking per call and RealRobot dedups session-wide. Both requests
still go out with observe=False, so the write-only side-effect argument that
makes the twin trajectory bit-identical is untouched -- there is a test asserting
the split commands the same motion as one batch.

The arm coming to rest at the boundary is a small gain of its own: the free-run
is anchored at the last rest state before the push, and now there really is one.

Empty, or a name the episode never runs, records everything and warns. Too much
video is slow; too little is an episode whose first topple happened off camera,
and the first onset is what every interval is measured against.

Why it must sit on top of #137

This moves the track's first frame from the episode's initial arrangement to the
post-place one, and the id matching anchors on a position set.

Against #137 as first written -- anchored on the episode's first state -- that
was fatal: the two placed dominoes sit ~150-200 mm from the anchor and drop out,
taking the match from 4/4 to 2/4. 98e24cd moved both sides to the settled
arrangement immediately before the cascade, which is independent of when
recording starts, and push-only measures 4/4 again.

An earlier revision of this branch claimed recording from the push fixes the id
matching. That was wrong, and is corrected in the commit message. The dominant
cause was the base-frame/world-frame rotation #137 identified (144-307 mm against
a 40 mm tolerance); starting the take later does not touch it.

No lead-in, deliberately

Two known hazards ride on that, neither fixed here and both likelier on a short
take:

  • load_track takes each domino's first observed centre, so a start domino
    occluded by the gripper for the whole pre-cascade window contributes no
    position and drops out of the match. On a push-only take that is the domino
    whose onset every interval is measured against.
  • The offset vote's tie-break does not settle exact aliasing, so an evenly spaced
    row missing one domino can match completely and be shifted by one.

If either bites, splitting one option earlier is the fix. Called deliberately by
the user as a first cut, to be revisited on evidence rather than pre-emptively.

Tests

Five, plus three mutants: no prologue split, recording opened before the
prologue, and the unknown-option fallback recording nothing.

The third mutant initially passed, which is worth recording. Asserting "one
shipment of three chunks" cannot distinguish everything recorded from
everything shipped, then the take opened -- the latter being an episode with
no motion in the take at all. An ordering probe was added to pin it, and it
fails under that mutant now.

test_real_robot_executor.py 84 passed, mypy clean, pylint 10.00/10.

CI note

Expect the shard-8 test_gnn_action_policy_approach / dill failure documented
on #138. It is pre-existing pytest-split regrouping, not from this branch --
.test_durations is a two-week-old snapshot that already omits an entire module,
so any PR adding tests re-packs the groups and changes what runs before what in
a given process.

@amburger66 amburger66 self-assigned this Aug 18, 2026
@amburger66
amburger66 marked this pull request as ready for review August 18, 2026 16:07
@amburger66
amburger66 force-pushed the domino-record-from-push branch from 84baa0e to 4e1b76c Compare August 18, 2026 18:01

@yichao-liang yichao-liang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LTGM! Let's see how well this works!

@amburger66
amburger66 force-pushed the domino-record-from-push branch from f4873c4 to ae2cb7f Compare August 19, 2026 13:15
Base automatically changed from worktree-pipeline-speedup to master August 19, 2026 13:29
Only the cascade is scored. On run_20260818_092302 the first onset landed 107 s
into a 131 s track, so roughly 80% of the video was the pick-and-place that
arranges the row -- none of it evidence, all of it post-processing, which scales
with frames rather than seconds.

real_robot_record_from_option names the option the take opens in front of.
_ship_episode sends the prologue as one unrecorded request, opens the take, then
sends the rest. That costs one controller round trip and NOT a planner call: the
twin has already simulated every option by the time _ship_episode runs, so this
does not give back what open-loop batching bought. Splitting is safe for the
same reason per-boundary shipping was -- _split_actions restarts its gripper
tracking per call and RealRobot dedups session-wide -- and both requests still
go out with observe=False, so the write-only side-effect argument that makes the
twin trajectory bit-identical is untouched.

The arm coming to rest at the boundary is a small gain of its own: the free-run
is anchored at the last rest state before the push, and now there really is one.

Empty, or a name the episode never runs, records everything and warns. Too much
video is slow; too little is an episode whose first topple happened off camera,
and the first onset is what every interval is measured against.

STACKED ON #137. This moves the track's first frame from the episode's initial
arrangement to the post-place one, and the id matching anchors on a position
set. Against #137 as originally written -- anchored on the episode's first state
-- that was fatal: the two placed dominoes sit ~150-200 mm from the anchor and
drop out, taking the match from 4/4 to 2/4. 98e24cd moved both sides to the
settled arrangement immediately BEFORE the cascade, which is independent of when
recording starts, and push-only measures 4/4 again. This commit is only correct
with that fix underneath it.

An earlier version of this branch claimed recording from the push FIXES the id
matching. That was wrong. The dominant cause was the base-frame/world-frame
rotation #137 identified -- 144-307 mm against a 40 mm tolerance -- and starting
the take later does not touch it.

No lead-in, as a deliberate first cut. Two known hazards ride on that, neither
fixed here and both likelier on a short take: load_track takes each domino's
first OBSERVED centre, so a start domino occluded by the gripper for the whole
pre-cascade window contributes no position and drops out of the match; and the
offset vote's tie-break does not settle exact aliasing, so an evenly spaced row
missing one domino can match completely and be shifted by one. If either bites,
splitting one option earlier is the fix.

Tests: five, plus three mutants -- no prologue split, recording opened before
the prologue, and the unknown-option fallback recording nothing. The third
initially PASSED, because asserting one shipment of three chunks cannot tell
"everything recorded" from "everything shipped, then the take opened"; the
ordering probe was added to pin it, and it fails under that mutant now.
The boxes are SAM-2 prompts applied to the take's FIRST FRAME, so they have to
describe the arrangement that frame shows. While the take began at the reset
those were the same thing and the draw sat at run start. Recording from a later
option makes them different, and nothing detected it: stage 2 does not report an
empty box, it fits a mask to whatever is inside the one it was given.

Observed on run_20260818_140211. boxes.json was written at 14:02:37, before any
motion; the take opened at 14:05:37, after the prologue had picked and placed
two dominoes. Two of the four prompts pointed at bare table. A human noticed it
in the overlay, which is not a detection mechanism.

So the draw moves to the recording boundary: after the prologue ships, before
_start_recording. The arm is at rest and the row is in its final arrangement,
and it is also the only moment a human can draw boxes that match the take. Where
the take still opens at the reset -- no record_from_option -- the draw stays at
run start, which was already right there.

ensure_boxes is now idempotent. The boundary call site runs every episode and
the draw must not: a fixed-plan replay arranges the same row each time, and a
drag window opening per take is exactly what pick_boxes_at_start exists to
avoid.

This makes a real trade rather than removing a cost. The human is now needed
~50 s into the first episode instead of before it starts. The unattended
alternative is projecting the twin's geometry into image space, which needs no
human at all and is considerably more work.

Three tests, on the ordering rather than the fact of the draw -- asserting that
boxes were drawn cannot tell the right moment from the wrong one, which is the
whole defect. The probe records what had shipped and whether the take was open
at draw time, and asserts one shipment and no open take. Both mutations fail it:
drawing at run start regardless, and deleting the boundary call.
@amburger66
amburger66 force-pushed the domino-record-from-push branch from ae2cb7f to c01ca2e Compare August 19, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants