Open-loop execution and continuous recording for the real domino friction fit - #134
Merged
Conversation
The friction fit currently scores the twin against itself: of ~229 recorded states in the Stage-6 run, 6 were real camera looks and the rest are the twin integrating PyBullet at the true friction, so minimising that SSE recovers the true value by construction. The agent declined to declare, which was the right read of the evidence. This plan fixes the evidence. Three steps. Open-loop execution ships an episode's motion in one batch instead of one option at a time, which is safe because with observe=False shipping is a pure write-only side effect and deferring it leaves the twin trajectory bit-identical. Markerless perception supplies a dense real pose track, already measured at 1.03 deg median orientation error on the better camera. Scoring then moves to inter-domino propagation intervals, which are invariant to both the alignment offset and the ~25-38mm extrinsics calibration offset (measured displacements are accurate to ~1mm). Written against both remotes as of 2026-08-14, not against their docs.
The agent's 0.082 grasp offset failed on the real arm in run_20260807_102548:
look 2 put domino_4 0.1987 m from the twin's prediction while every other
domino stayed under 9 mm, so the gripper closed on nothing and the twin went
on believing it held the block. 0.082 puts the fingertips 8.2 mm below the top
of a 150 mm domino; in sim that grips as well as anything, because a grasp is a
JOINT_FIXED weld formed on 5e-4 proximity and grip depth costs nothing, and
higher offsets clear BiRRT more easily -- so a sim optimiser drives this
parameter to the worst real value. 0.0657 is _grasp_z_offset() for this hand.
The bridge sketches take their start and target from
domino_real_{start,target}_id rather than assuming them, and space the movables
across the measured 0.331 m gap. The 3-movable one exists because 2 movables at
0.110 m pitch only reached the target 3/8 at friction 0.1 -- low friction
shortens topple reach.
These live in scripts/plan_sketches/ and are named for what distinguishes them
-- scene, bridge size, and either the grasp offset or the fact that the roles
come from the scene's own ids -- rather than for which run happened to produce
them ("agent_best" ages badly, and said nothing about the 1-domino bridge or
the grasp it was tuned for).
Config: one exploration episode per cycle, since the fixed-plan explorer
replays the same plan and a second episode costs a full hardware run plus a
human scene reset for a near-duplicate.
Eight PRs of catch-up. The pointer sat at #63 ("use the measured table height"), which predates the entire perception stack, so predicators could not import any markerless or recorder code at all -- nothing downstream of it could even be prototyped. What this brings in: #66 ZedRecorderSession (open the ZEDs once, start/stop many SVO takes), #67 markerless pose estimation, #68 depth-free bundles (1.4 GB -> 48 MB, poses unchanged), #71 the stage-1 extrinsics check, #72 stage-3 fp16 + crop (20x -> ~3.2x real time) and the occlusion visibility gate, #73 markerless scene capture as the default, #75 the pre-5.3 pyzed fix that lets the recorder run on this machine's SDK 3.8.2 at all, and #76 output paths that stop derived artifacts overwriting the bundle they came from. The submodule is not checked out in this worktree, so this moves the gitlink only; run `git submodule update --init` to populate it.
Today the twin simulates an option, ships it to the arm, then simulates the next -- so the arm idles through the next option's motion planning before it moves again. real_robot_open_loop_episode (off by default) holds each completed option and ships the whole episode as a single request once it has all been simulated. Why deferring is safe, and not merely acceptable: with the boundary look off, execute_chunks(observe=False) returns [], the absorb loop never runs, and after_step already returns obs UNCHANGED. Shipping is a pure write-only side effect, so *when* it happens cannot be observed by the rollout -- the twin trajectory is bit-identical either way. A test asserts that equality directly, driving both paths with a distinct observation per step so it cannot pass by both sides being constant. The port needed a fourth method. ActionExecutor had tasks_for/after_reset/ after_step and no end-of-episode call, which is the only reason shipping had to happen inside after_step in the first place. after_episode(completed) closes that gap; BaseEnv.finish_execution is a no-op so cogman can end an episode without knowing whether the env drives anything, and PyBulletEnv delegates. completed=False drops the buffer instead of shipping it. What survives an abnormal end is a prefix -- half a bridge, or a transport with no place at the end of it -- and the arm would run it with nobody having decided that was a good idea. The information that it was partial exists only at that call, so it is the last place the judgement can be made. after_reset drops anything left over as a backstop, since a plan shipped against the next episode's scene is the same hazard one episode later. Mutually exclusive with real_robot_observe_at_option_boundary, asserted at construction: a boundary look has to happen between the two options it separates, and batching leaves no such moment. Batch start/end are logged from both time.monotonic_ns() and time.time_ns(); the wall stamp is what pairs with a recorder's own host stamp, the monotonic one survives an NTP step. Kept off by default because it is a real regression in supervisability: the arm runs the whole plan with the e-stop as the only intervention, where today a bad first option is visible before the second ships. Two duck-typed env mocks in test_cogman gain the no-op, since cogman now calls it on the BaseEnv it is declared to take. 9 tests. Verified they fail against the old behaviour: disabling the deferral reds 4 of them, and the bit-identical test reds when the deferred path hands back anything other than what it was given. Full suite 1497 passed, mypy clean, pylint 10.00/10.
…ation Step 2 of the open-loop plan: the cameras record the whole execution and the poses are recovered afterwards, instead of six option-boundary looks. Nothing here estimates a pose. The markerless pipeline runs at roughly 3x real time, so a result cannot come back inside the episode that produced it -- which is the same fact that makes open-loop execution necessary rather than merely nice. real_robot_record_episodes (off by default) wires ZedRecorderSession onto the executor's lifecycle: open() once when the executor is built, start_take after each reset, stop_take at after_episode, close at exit. Opening once matters because a learning cycle is many episodes and per-episode camera init and warmup would otherwise be paid every time. Recording stops in a finally, shipping does not. They have opposite defaults on an abnormal end: a partial plan must NOT reach the arm, but a take left open records until the disk fills. Two tests pin the pair -- one where the episode did not complete, one where execute_chunks itself raises. Failures are asymmetric for the same reason. start_take raises: an episode that cannot record spends hardware time and a human scene reset for nothing, so it is better to say so before the arm moves. stop_take does not: by then the arm has already moved, and a recording problem must not destroy the run around it, so the take is logged and marked unusable. meta.json's errors list gets the same treatment -- a camera that dropped out mid-episode yields a short track that is perfectly well formed, which is exactly the failure worth being loud about. Recording and a live "zed" perception are refused together, before the env or the hardware is examined: both open the same cameras and a ZED admits one owner. This is a config contradiction, so it is reported as one. Defaults chosen from the pipeline's own measurements: HD720 (what it was measured on) at 60 fps, not 30. A real cascade's topple onsets came 6, 4 and 2 frames apart at 30 fps, and those inter-domino intervals are what the friction fit is scored on -- at 30 fps a one-frame detection error is half the shortest interval. HD720 already runs at 60. Exports stay off during a run; stop_take can write depth inline, but that is the expensive offline work and doing it in the episode loop would undo open-loop execution. babyrobot stays a lazy import, and the recorder gets its own module-level-import test rather than relying on the executor's: the executor imports this module at module level, so a top-level import here would break a submodule-less checkout just as surely, and the ZED recorder lives under pose_estimation rather than babyrobot. mypy.ini gains the matching stanza for the same reason. The recorder's own tests drive a stub session, which is what keeps them hardware-free and is also how a stub silently drifts. test_real_robot_bridge gains a contract test that pins the four calls and their keywords against the real ZedRecorderSession, skipping without the submodule like its neighbours. It passes against b45ac97. 12 tests. Verified load-bearing: removing the finally reds the raise case, and ignoring meta's errors reds the unusable case. Full suite 1510 passed, mypy clean, pylint 10.00/10.
Restores per-episode scene rebuild alongside episode recording, which the previous commit had to refuse: recording and a live "zed" perception both want the same cameras and a ZED admits one owner. The refusal gave up less than it appeared to. Live "zed" perception IS the marker pipeline -- _live_records loads a marker registry and a dictionary name and calls capture_frames -- and the 20mm ArUco markers are not resolvable at this camera distance: 1 of ~7 detected on 30264679 and 0 on 32294776. So the capability being sacrificed could not see these dominoes anyway. This replaces it with the thing markerless actually offers. The trick is that a snapshot opens no cameras. It is a second, short take on the recorder's already-open session, taken between episodes while no episode take is running -- so there is no second owner and nothing to arbitrate. A test pins the sequencing, and the stub session now refuses a concurrent take the way the real one does, so that test can fail. It plugs in without touching the reset flow. RealRobot.reset_env homes the arm, blocks until a human confirms the scene is arranged, and only THEN calls perception.observe() -- which is exactly when a snapshot should be taken. So MarkerlessSnapshotPerception duck-types the perception protocol and is injected through make_real_robot(perception=...); open() and close() are no-ops, because owning cameras here is the collision being avoided. attach_real_robot builds the recorder before the robot so the session exists to hand over. run_stages reads config.boxes and NOT config.boxes_json, so the runner resolves the file itself. Getting that wrong would not fail -- it would quietly fall back to the drag window on every episode, which is the difference between unattended and not. A contract test pins it, along with the MarkerlessCapture fields and run_stages' signature, and asserts run_stages' source mentions config.boxes and not config.boxes_json. z_mode defaults to "contact" here, opposite to the cascade default: a rebuild looks at a scene a human has just arranged upright on the table, where z from the table is right and better constrained. "free" is for dominoes at rest on each other. table_z is passed rather than measured per capture, because the twin's base->world transplant and the fit have to agree on where the table is. Snapshot take directories carry a counter, not just a timestamp: start_take makes the directory with exist_ok, so two snapshots in the same second would write into one and the second would inherit the first's frames. Found by a test that took two snapshots in a row. Snapshots are tracked separately from episode takes. ``takes`` is what the fit consumes; a snapshot is an input to a task, not a record of an execution. 10 tests, with the pipeline and scene loader injected so none of it needs a GPU, a camera, or the submodule. The two contract tests were run against the bumped submodule (b45ac97) via PYTHONPATH, since the installed babyrobot resolves to the main checkout's older working tree. Full suite 1519 passed, mypy clean, pylint 10.00/10.
CI runs docformatter 1.4 and isort as their own required checks, and I had run only yapf and pylint over these files. docformatter rewraps the summary lines and one-line docstrings its own way, and isort wanted the two new pybullet_helpers imports reordered in the executor's test. Produced by run_autoformat.sh's tools in its order (yapf, then docformatter, then isort), so the three agree rather than each undoing the last. No wording or behaviour changed. Full suite 1519 passed, mypy clean, pylint 10.00/10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Steps 1 and 2 of the plan in
docs/domino-openloop-continuous-perception.md: move real-robot execution from per-option shipping to one batch per episode, and record each episode for offline markerless pose estimation instead of taking six option-boundary looks.Every new flag is off by default. With them off, behaviour is byte-for-byte what it was.
Why
The Stage-6 friction fit currently scores the simulator against itself. Of ~229 recorded states in the completed run, 6 were real camera looks; the rest are the twin integrating PyBullet at the true friction, so minimising that SSE recovers the true value by construction. The agent declined to declare, which was the correct read of that evidence. This series fixes the evidence rather than the decision.
Two facts shape the design:
30264679, 0 on32294776. The live"zed"perception is the marker pipeline, so it cannot see these dominoes.The commits
8b6adcfd7526170d8e690submodules/BabyRobotPredicator396094e→b45ac97(eight PRs)f007e39421ffcb62f293cStep 1 — open-loop execution
real_robot_open_loop_episodeholds each completed option and ships the whole episode as one request once it has all been simulated, instead of shipping option-by-option while the arm idles through the next option's motion planning.Why deferring is safe. With the boundary look off,
execute_chunks(observe=False)returns[], the absorb loop never runs, andafter_stepalready returnsobsunchanged. Shipping is a pure write-only side effect, so when it happens cannot be observed by the rollout. A test asserts the twin trajectory is bit-identical, driving both paths with a distinct observation per step so it cannot pass by both sides being constant.The port needed a fourth method.
ActionExecutorhadtasks_for/after_reset/after_stepand no end-of-episode call, which is the only reason shipping had to live insideafter_step.after_episode(completed)closes that gap;BaseEnv.finish_executionis a no-op socogmanstays env-agnostic.completed=Falsedrops the buffer. What survives an abnormal end is a prefix — half a bridge, or a transport with no place at the end of it — and the arm would run it with nobody having decided that was a good idea.Step 2 — recording, and the snapshot rebuild
real_robot_record_episodeswiresZedRecorderSessiononto the executor lifecycle:open()once per run, a take per episode,close()at exit.Recording stops in a
finally; shipping does not. On an abnormal end these want opposite things: a partial plan must not reach the arm, but a take left open records until the disk fills. Failure handling is asymmetric for the same reason —start_takeraises (an episode that cannot record burns hardware time and a human scene reset for nothing),stop_takenever does (by then the arm has moved, and a recording problem must not destroy the run around it, so the take is logged and marked unusable).Recording and live
"zed"perception are refused together: both open the same cameras and a ZED admits one owner.real_robot_snapshot_rebuildthen restores per-episode task rebuild without reopening that conflict. A snapshot opens no cameras — it is a second, short take on the recorder's already-open session, taken between episodes while no episode take is running. It plugs in atRealRobot.reset_env, which homes the arm, waits for the human, and only then callsperception.observe()— exactly when a snapshot should be taken.Defaults come from the pipeline's own measurements: HD720 at 60 fps, not 30, because a real cascade's topple onsets came 6, 4 and 2 frames apart at 30 fps and those inter-domino intervals are what the friction fit will be scored on.
What this does NOT do yet
The learner still does not see real observations. The fit reads
_fit_trajectories, which is theobsstream out ofenv.step. This series produces.svotakes and marks which are usable; nothing yet runs the markerless pipeline over them or hands the resulting pose track to the fit.That is Step 3, and it is where the real poses reach the learner — through the objective, not the trajectory. Concretely still missing: running
reconstruct_dominoes_markerless.pyover each take, the scope fix (§3.1), event alignment (§3.2), the propagation-interval residual (§3.3), andcode_sim_learning_rollout_score_observed_only(§3.7).Running with
real_robot_open_loop_episodeon and Step 3 absent would make the fit worse, not better — zero real observations instead of six.Verification
finallyreds the shipping-raises test;meta["errors"]reds the unusable-take test.ZedRecorderSessionand the markerless staging, run againstb45ac97viaPYTHONPATH. They skip where the submodule is absent, matching the convention intest_real_robot_bridge.py.No hardware validation. None of this has been run on the arm or the cameras.
Reviewer's attention
cogman.pygained four call sites, not the two the plan named. Reading the control flow, tworaisepaths leave the function without either — harmless for Step 1, but it would strand a recording under Step 2.test_cogman.pygained the no-op, since cogman now callsfinish_executionon theBaseEnvit is declared to take.real_robot_snapshot_z_modedefaults to"contact", opposite the cascade default, because a rebuild looks at a scene just arranged upright on the table.mypy.inigains apose_estimation.*stanza for the same reasonbabyrobot.*has one.scripts/plan_sketches/, named for what distinguishes them (scene, bridge size, grasp offset or scene-derived roles) rather than for the run that produced them.exp_domino_real.yaml'sfixed_plan_explorer_pathfollows.