Open-loop execution and continuous recording for the real domino friction fit - #133
Closed
amburger66 wants to merge 24 commits into
Closed
Open-loop execution and continuous recording for the real domino friction fit#133amburger66 wants to merge 24 commits into
amburger66 wants to merge 24 commits into
Conversation
A task built from the real scene carried no evaluator, so every episode scored reward 0.0 and an over-built chain was indistinguishable from a minimal one. That difference is exactly what the friction-mismatch experiments are read off, and it is what DominoEvaluator computes: each movable domino the cascade consumes costs domino_block_cost. Opt-in via domino_real_attach_evaluator, so existing runs score as before. The movable count comes from the perceived scene rather than domino_min_block_num_blues -- that flag is a generator budget, and a real scene stages whatever the person put on the table. The evaluator asserts that a success outscores any failure; that would fire mid-episode on the real robot, so the bound is checked at task construction and reports the scene's own numbers instead.
stage6_domino_real.yaml is Stage 6 proper -- exp_domino_real.yaml with execution on. Everything else the stage needs (live cameras, a look at every option boundary, a human reset per episode) is already the settings.py default, so it sets only what differs. stage6_domino_real_underreach.yaml adds the REVERSE friction mismatch: the planner believes the dominoes are far more slippery than they are, under-estimates reach, over-builds, and pays for the surplus. Only domino_planning_friction moves -- the twin keeps modelling the real table. Its mirror does not transfer to hardware, where the twin is not the ground truth; the file says so. Both cap wait_option_max_steps, whose shipped bound is inf: a Wait that never sees the scene settle otherwise runs to the horizon and takes a human reset with it. The rehearsal now attaches the evaluator, which is the cheap place to learn whether the cascade certificate accepts an episode assembled from perception rather than physics.
utils.string_to_python_object maps both "None" and "none" to Python None, so real_robot_perception: "none" from a launcher config never arrives as the string _make_perception was matching on -- it arrived as None and fell through to the unknown-kind ValueError. The mode was reachable only from a direct reset_config call, which is why the existing test (which does exactly that) never caught it. Also adds the evaluator-check launcher: perception off and no looks, so the twin keeps what the plan built and the cascade can actually run. That is the cheap place to learn whether DominoEvaluator's certificate accepts a real-shaped episode, which the rehearsal cannot answer because scene_file perception undoes every Place.
Replaces domino_real_attach_evaluator with the condition DominoTaskGenerator._make_task already uses. What decides whether a verdict means anything is the certificate's causal model, not which env built the scene, so the real env should not need a flag the simulated ones do not have -- and defaulting it off meant the real path silently scored 0.0 while every sim path scored. Both of the generator's conditions hold for a real scene: every shipped domino config sets domino_use_domino_blocks_as_target, and a real scene is dominoes and nothing else, so there is no additional dynamic component to topple them without a Push. Also carries across the scoring text the generator appends to goal_nl. Its comment records why it exists (run_20260716_215533 burned its budget theorizing that any disturbed blue disqualifies a solve); an agent scored on the real robot would have hit the same misreading.
Brings in the auto-launch (#59), so a restarted droid server binds its robot handles instead of failing every read with 'FrankaRobot object has no attribute _robot', and the safety box (#61), which confines the end-effector and resolves fail-closed. The pin was at a 2026-07-28 commit that predated both. Cherry-picked from master; its exp_domino_real.yaml scene line is dropped in favour of the newer one already on this branch. Note this does NOT include BabyRobotPredicator #63 (the measured table height), which is still open -- the submodule wants bumping again once it lands, or a scene captured through the submodule's own scripts will carry the 4 mm z error back.
… less far Two measured fixes to the same skill pair. Pick commanded closed_fingers - 0.01, which with the domino env's 0.02 put the target 10 mm past closed. A 0.029 m domino blocks the fingers well before that, so the position motor drove to the joint limit: 18 mm of penetration, measured. A grasp here is a JOINT_FIXED weld created on 0.5 mm proximity, so the overshoot buys nothing once the fingers touch -- and past that it only stores contact impulses that kick the domino when the weld is released. The overshoot is now a SkillConfig field (default 0.01, so nothing else changes) with a CFG override. Place released the domino from an EE height of 0.58, about 4 cm above where it rests, and the fall is what scattered it. Over 3 placements each, mean error was 43 mm at 0.580, 15 mm at 0.568, 10 mm at 0.565. The floor is the collision-aware Place goal rather than the physics: below ~0.565 BiRRT cannot route the held domino to a goal that puts it through the table and raises OptionExecutionFailure (0.560 fails). 0.568 keeps 8 mm of margin above that edge. Only the release height is shown to improve placement; the overshoot change removes a real defect but its effect on placement error was within run-to-run noise.
tasks_for consumed a single _reset_pending token, so whichever split asked first got the perceived scene and the other kept the captured-scene task. With the online loop off, main.py requests the train tasks during setup -- so the TEST task, the one that actually gets solved, silently stayed on the scene JSON while the arm executed in the real scene. Seen in run_20260806_132702: the agent's sim.state matched the JSON-derived pose to 1e-16 (bit-identical, where a camera would differ by millimetres). It tuned a single-bridge plan to 20/20 against those poses; on the arm the Push was refused before moving, its approach waypoint 9.4 mm inside the bridge domino it had just placed. The look is now kept and both splits rebuild from it. A physical reset arranges one scene and the person who arranged it meant it for whatever runs next, so train and test are the same task -- which is what a real bench means anyway.
The scene JSON's poses are a snapshot. Planning against them while the arm works a scene nobody looked at plans for a world that is not there, and it is silent: planning succeeds, and the twin only jumps to the truth at the first option boundary. tasks_for now raises when live perception is configured and no look has happened, naming both ways out. Replaying a recorded plan is the one case that wants those exact poses -- the plan was written against them -- so replay_plan opts in via real_robot_allow_captured_scene_task, and the two test suites that deliberately run against a capture say so too. Does not remove the JSON: _scene_ids (the capture-id to slot map, without which a live observation names nothing) and the slot counts are read in __init__, before any camera exists. Dropping it entirely means capturing at construction, which is a separate change.
The shipped grasp_z_offset box (0, 0.1) was drawn around the Fetch: its top edge IS the Fetch's reach edge, and the collision edge at 0.045 sits 45% up, leaving the top 55% feasible. Sweeping a real domino at 5 mm, the Panda collides at the same 0.045 -- that edge belongs to the domino, not the hand -- but stops reaching it past 0.080, so only 35% of the same box can work and a sampler spends most of its budget on offsets that cannot. Give the Panda the Fetch's proportions around its own reach edge rather than shrink-wrapping its band: same feasible fraction, same shape of learning problem, so a sampler compared across the two arms is comparing embodiment and not box width. The reach edge comes from _hand_z_correction, which is zero on the Fetch -- so the Fetch keeps the shipped box and its parameter description untouched, and a future hand gets bounds without another sweep. Panda: [0.0137, 0.0832], 35% -> 50% feasible. The tuned sampler value (0.0657) and every Pick in plans/*.txt stay inside it. create_pick_skill grows an optional param_defs that defaults to the canonical box, so coffee, bridge, grow and boil are unchanged -- their objects were never measured, and guessing bounds for them would be worse than leaving them.
2403bf2 lowered _DOMINO_DROP_Z to 0.568 and left this assertion behind, so the sampler test has been failing since. What it means to check is that the sampler uses the canonical drop height, not that the height has one particular value.
real_robot_dry builds the RealRobot without a RobotInterface, so the arm is never contacted and no polymetis server is needed, while the option loop, the twin correction, the divergence logging and the learning all run as they would live. Paired with real_robot_perception "none" deliberately. Live cameras on a motionless arm report a static scene, so every look would correct the twin back to the initial layout and the learner would see nothing but no-ops; blind keeps the twin evolving in sim, which is what exercising the pipeline wants.
Brings the branch onto master through #116 (squash-merged from this branch's own evaluator work) and #118 (the BabyRobotPredicator bump that lands #63's measured table height). Merging rather than rebasing because #116 arrived as a squash: replaying this branch's six pre-#116 commits would mean hand-resolving conflicts against content master already has. Conflicts, both taken deliberately: - submodules/BabyRobotPredicator -> master's 396094e. The branch was pinned a commit behind, at d53b02b, which predates #63; the whole point of #118 was to stop a scene captured through the submodule's own scripts carrying the 4 mm z error back. - exp_domino_real.yaml -> the branch's side wholesale. It is a superset of master's: same scene, execute and divergence tolerance, plus the friction mismatch, the MCMC steps, the dry-run rung and the agent_oracle_hybrid_sim block. The one master value not carried is real_robot_observation_dump_dir, which this branch repointed from logs/stage6_looks to logs/friction_0.1 on purpose.
real_robot_perception "none" on its own raises at executor construction: the option-boundary look asks the robot to perceive, and it has no cameras. Turning the look off is not enough either -- with human resets on, reset_env perceives after the human confirms, so it fails at the first episode boundary instead. All three go together, and now say so in the file.
Swaps the exploration half for a constant so the run measures what it is meant to: whether perception feeds the learner well enough to move the friction belief. A planning explorer varies episode to episode and costs minutes each; a fixed plan makes every episode comparable and puts any failure on the loop rather than the planner. The plan is the one the agent explorer found in run_20260807_091533 (12/12 at reward 0.95, the optimal score). Verified 8/8 in the BELIEF simulator -- the env built with skip_process_dynamics=True, whose PyBullet bodies really do carry domino_planning_friction 0.1 -- which is the requirement: the agent has to believe the plan works. Checked against the hand-written bridges on the same scene: solve_v2_bridge2 is 5/8 there and solve_v2 is 0/5 (Push IK). Also flips agent_planner_use_base_simulator on. The agent's plan-testing simulator is built with skip_process_dynamics=that flag, and only a skip_process_dynamics=True env picks up domino_planning_friction -- so left at its default the agent was testing plans against the TRUE friction and its belief was never mismatched at all. Nothing for a sysID to discover. Live rung: the arm moves and the cameras look. A real cascade has to happen to be observed, so neither half can be faked. Human resets are on so each task is rebuilt from a look, which is the thing under test; Pick and Push follow perceived poses, only Place's drop point is absolute.
#117 was closed rather than merged: its config flag was never actually used, and it lowered the oracle Place drop height for every embodiment on the strength of measurements taken only on the Panda. This branch had been carrying that work locally ever since. Reverted to master exactly: * _DOMINO_DROP_Z back to 0.58 from 0.568 * SkillConfig.pick_close_overshoot dropped; Pick's close target is the literal closed_fingers_joint - 0.01 again * the skill_pick_close_overshoot setting dropped * _build_skill_config back to returning SkillConfig directly * the sampler test back to asserting the literal 0.58, which is correct again now the constant is 0.58 Nothing else was mixed into those files: the whole local delta on all six was #117's, so restoring master's copies is the entire revert. The #119 fixes they also contain (the Pick grasp box, the captured-scene guard) are master's and stay. Worth knowing before the next hardware run: 0.580 measured a 43 mm mean placement error against 15 mm at 0.568, over 3 placements each. The drop-height change is still wanted -- scoped to the Panda, the way the Pick box is -- rather than abandoned.
The friction fit currently scores the twin against itself: of ~229 recorded states in the Stage-6 run, 6 were real camera looks and the rest are the twin integrating PyBullet at the true friction, so minimising that SSE recovers the true value by construction. The agent declined to declare, which was the right read of the evidence. This plan fixes the evidence. Three steps. Open-loop execution ships an episode's motion in one batch instead of one option at a time, which is safe because with observe=False shipping is a pure write-only side effect and deferring it leaves the twin trajectory bit-identical. Markerless perception supplies a dense real pose track, already measured at 1.03 deg median orientation error on the better camera. Scoring then moves to inter-domino propagation intervals, which are invariant to both the alignment offset and the ~25-38mm extrinsics calibration offset (measured displacements are accurate to ~1mm). Written against both remotes as of 2026-08-14, not against their docs.
… roles
The agent's 0.082 grasp offset failed on the real arm in run_20260807_102548:
look 2 put domino_4 0.1987 m from the twin's prediction while every other
domino stayed under 9 mm, so the gripper closed on nothing and the twin went
on believing it held the block. 0.082 puts the fingertips 8.2 mm below the top
of a 150 mm domino; in sim that grips as well as anything, because a grasp is a
JOINT_FIXED weld formed on 5e-4 proximity and grip depth costs nothing, and
higher offsets clear BiRRT more easily -- so a sim optimiser drives this
parameter to the worst real value. 0.0657 is _grasp_z_offset() for this hand.
bridge2/bridge3 take their start and target from domino_real_{start,target}_id
rather than assuming them, and space the movables across the measured 0.331 m
gap. bridge3 exists because 2 movables at 0.110 m pitch only reached the
target 3/8 at friction 0.1 -- low friction shortens topple reach.
Config: one exploration episode per cycle, since the fixed-plan explorer
replays the same plan and a second episode costs a full hardware run plus a
human scene reset for a near-duplicate.
Eight PRs of catch-up. The pointer sat at #63 ("use the measured table height"), which predates the entire perception stack, so predicators could not import any markerless or recorder code at all -- nothing downstream of it could even be prototyped. What this brings in: #66 ZedRecorderSession (open the ZEDs once, start/stop many SVO takes), #67 markerless pose estimation, #68 depth-free bundles (1.4 GB -> 48 MB, poses unchanged), #71 the stage-1 extrinsics check, #72 stage-3 fp16 + crop (20x -> ~3.2x real time) and the occlusion visibility gate, #73 markerless scene capture as the default, #75 the pre-5.3 pyzed fix that lets the recorder run on this machine's SDK 3.8.2 at all, and #76 output paths that stop derived artifacts overwriting the bundle they came from. The submodule is not checked out in this worktree, so this moves the gitlink only; run `git submodule update --init` to populate it.
Today the twin simulates an option, ships it to the arm, then simulates the next -- so the arm idles through the next option's motion planning before it moves again. real_robot_open_loop_episode (off by default) holds each completed option and ships the whole episode as a single request once it has all been simulated. Why deferring is safe, and not merely acceptable: with the boundary look off, execute_chunks(observe=False) returns [], the absorb loop never runs, and after_step already returns obs UNCHANGED. Shipping is a pure write-only side effect, so *when* it happens cannot be observed by the rollout -- the twin trajectory is bit-identical either way. A test asserts that equality directly, driving both paths with a distinct observation per step so it cannot pass by both sides being constant. The port needed a fourth method. ActionExecutor had tasks_for/after_reset/ after_step and no end-of-episode call, which is the only reason shipping had to happen inside after_step in the first place. after_episode(completed) closes that gap; BaseEnv.finish_execution is a no-op so cogman can end an episode without knowing whether the env drives anything, and PyBulletEnv delegates. completed=False drops the buffer instead of shipping it. What survives an abnormal end is a prefix -- half a bridge, or a transport with no place at the end of it -- and the arm would run it with nobody having decided that was a good idea. The information that it was partial exists only at that call, so it is the last place the judgement can be made. after_reset drops anything left over as a backstop, since a plan shipped against the next episode's scene is the same hazard one episode later. Mutually exclusive with real_robot_observe_at_option_boundary, asserted at construction: a boundary look has to happen between the two options it separates, and batching leaves no such moment. Batch start/end are logged from both time.monotonic_ns() and time.time_ns(); the wall stamp is what pairs with a recorder's own host stamp, the monotonic one survives an NTP step. Kept off by default because it is a real regression in supervisability: the arm runs the whole plan with the e-stop as the only intervention, where today a bad first option is visible before the second ships. Two duck-typed env mocks in test_cogman gain the no-op, since cogman now calls it on the BaseEnv it is declared to take. 9 tests. Verified they fail against the old behaviour: disabling the deferral reds 4 of them, and the bit-identical test reds when the deferred path hands back anything other than what it was given. Full suite 1497 passed, mypy clean, pylint 10.00/10.
…ation Step 2 of the open-loop plan: the cameras record the whole execution and the poses are recovered afterwards, instead of six option-boundary looks. Nothing here estimates a pose. The markerless pipeline runs at roughly 3x real time, so a result cannot come back inside the episode that produced it -- which is the same fact that makes open-loop execution necessary rather than merely nice. real_robot_record_episodes (off by default) wires ZedRecorderSession onto the executor's lifecycle: open() once when the executor is built, start_take after each reset, stop_take at after_episode, close at exit. Opening once matters because a learning cycle is many episodes and per-episode camera init and warmup would otherwise be paid every time. Recording stops in a finally, shipping does not. They have opposite defaults on an abnormal end: a partial plan must NOT reach the arm, but a take left open records until the disk fills. Two tests pin the pair -- one where the episode did not complete, one where execute_chunks itself raises. Failures are asymmetric for the same reason. start_take raises: an episode that cannot record spends hardware time and a human scene reset for nothing, so it is better to say so before the arm moves. stop_take does not: by then the arm has already moved, and a recording problem must not destroy the run around it, so the take is logged and marked unusable. meta.json's errors list gets the same treatment -- a camera that dropped out mid-episode yields a short track that is perfectly well formed, which is exactly the failure worth being loud about. Recording and a live "zed" perception are refused together, before the env or the hardware is examined: both open the same cameras and a ZED admits one owner. This is a config contradiction, so it is reported as one. Defaults chosen from the pipeline's own measurements: HD720 (what it was measured on) at 60 fps, not 30. A real cascade's topple onsets came 6, 4 and 2 frames apart at 30 fps, and those inter-domino intervals are what the friction fit is scored on -- at 30 fps a one-frame detection error is half the shortest interval. HD720 already runs at 60. Exports stay off during a run; stop_take can write depth inline, but that is the expensive offline work and doing it in the episode loop would undo open-loop execution. babyrobot stays a lazy import, and the recorder gets its own module-level-import test rather than relying on the executor's: the executor imports this module at module level, so a top-level import here would break a submodule-less checkout just as surely, and the ZED recorder lives under pose_estimation rather than babyrobot. mypy.ini gains the matching stanza for the same reason. The recorder's own tests drive a stub session, which is what keeps them hardware-free and is also how a stub silently drifts. test_real_robot_bridge gains a contract test that pins the four calls and their keywords against the real ZedRecorderSession, skipping without the submodule like its neighbours. It passes against b45ac97. 12 tests. Verified load-bearing: removing the finally reds the raise case, and ignoring meta's errors reds the unusable case. Full suite 1510 passed, mypy clean, pylint 10.00/10.
Restores per-episode scene rebuild alongside episode recording, which the previous commit had to refuse: recording and a live "zed" perception both want the same cameras and a ZED admits one owner. The refusal gave up less than it appeared to. Live "zed" perception IS the marker pipeline -- _live_records loads a marker registry and a dictionary name and calls capture_frames -- and the 20mm ArUco markers are not resolvable at this camera distance: 1 of ~7 detected on 30264679 and 0 on 32294776. So the capability being sacrificed could not see these dominoes anyway. This replaces it with the thing markerless actually offers. The trick is that a snapshot opens no cameras. It is a second, short take on the recorder's already-open session, taken between episodes while no episode take is running -- so there is no second owner and nothing to arbitrate. A test pins the sequencing, and the stub session now refuses a concurrent take the way the real one does, so that test can fail. It plugs in without touching the reset flow. RealRobot.reset_env homes the arm, blocks until a human confirms the scene is arranged, and only THEN calls perception.observe() -- which is exactly when a snapshot should be taken. So MarkerlessSnapshotPerception duck-types the perception protocol and is injected through make_real_robot(perception=...); open() and close() are no-ops, because owning cameras here is the collision being avoided. attach_real_robot builds the recorder before the robot so the session exists to hand over. run_stages reads config.boxes and NOT config.boxes_json, so the runner resolves the file itself. Getting that wrong would not fail -- it would quietly fall back to the drag window on every episode, which is the difference between unattended and not. A contract test pins it, along with the MarkerlessCapture fields and run_stages' signature, and asserts run_stages' source mentions config.boxes and not config.boxes_json. z_mode defaults to "contact" here, opposite to the cascade default: a rebuild looks at a scene a human has just arranged upright on the table, where z from the table is right and better constrained. "free" is for dominoes at rest on each other. table_z is passed rather than measured per capture, because the twin's base->world transplant and the fit have to agree on where the table is. Snapshot take directories carry a counter, not just a timestamp: start_take makes the directory with exist_ok, so two snapshots in the same second would write into one and the second would inherit the first's frames. Found by a test that took two snapshots in a row. Snapshots are tracked separately from episode takes. ``takes`` is what the fit consumes; a snapshot is an input to a task, not a record of an execution. 10 tests, with the pipeline and scene loader injected so none of it needs a GPU, a camera, or the submodule. The two contract tests were run against the bumped submodule (b45ac97) via PYTHONPATH, since the installed babyrobot resolves to the main checkout's older working tree. Full suite 1519 passed, mypy clean, pylint 10.00/10.
Collaborator
There was a problem hiding this comment.
can we put these plans in the docs subdir?
Collaborator
There was a problem hiding this comment.
For plans like these, let's move them to
https://github.com/BasisResearch/predicators/tree/master/scripts/plan_sketches or https://github.com/BasisResearch/predicators/tree/master/scripts/scripted_option_policies?
yichao-liang
approved these changes
Aug 17, 2026
yichao-liang
left a comment
yichao-liang
left a comment
Collaborator
There was a problem hiding this comment.
Let's organize the claude_plan/plan files a bit, but otherwise they look good! Thanks!
Author
|
Superseded by #134, which is branched cleanly off master. This branch's commit list included ~18 older commits whose content already reached master via the squash-merge of #119 — git could not see them as ancestors, so they showed up as new. #134 carries the same changes as six commits with no such noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Steps 1 and 2 of the plan in
claude_plans/domino-openloop-continuous-perception.md: move real-robot execution from per-option shipping to one batch per episode, and record each episode for offline markerless pose estimation instead of taking six option-boundary looks.Every new flag is off by default. With them off, behaviour is byte-for-byte what it was.
Why
The Stage-6 friction fit currently scores the simulator against itself. Of ~229 recorded states in the completed run, 6 were real camera looks; the rest are the twin integrating PyBullet at the true friction, so minimising that SSE recovers the true value by construction. The agent declined to declare, which was the correct read of that evidence. This series fixes the evidence rather than the decision.
Two facts shape the design:
30264679, 0 on32294776. The live"zed"perception is the marker pipeline, so it cannot see these dominoes.The commits
3b41dd871764307b81ce1origin/masterdf7d108submodules/BabyRobotPredicator396094e→b45ac97(eight PRs)237bc616f9bc15d2e3b41The branch also carries earlier config and experiment work that predates this series.
Step 1 — open-loop execution
real_robot_open_loop_episodeholds each completed option and ships the whole episode as one request once it has all been simulated, instead of shipping option-by-option while the arm idles through the next option's motion planning.Why deferring is safe. With the boundary look off,
execute_chunks(observe=False)returns[], the absorb loop never runs, andafter_stepalready returnsobsunchanged. Shipping is a pure write-only side effect, so when it happens cannot be observed by the rollout. A test asserts the twin trajectory is bit-identical, driving both paths with a distinct observation per step so it cannot pass by both sides being constant.The port needed a fourth method.
ActionExecutorhadtasks_for/after_reset/after_stepand no end-of-episode call, which is the only reason shipping had to live insideafter_step.after_episode(completed)closes that gap;BaseEnv.finish_executionis a no-op socogmanstays env-agnostic.completed=Falsedrops the buffer. What survives an abnormal end is a prefix — half a bridge, or a transport with no place at the end of it — and the arm would run it with nobody having decided that was a good idea.Step 2 — recording, and the snapshot rebuild
real_robot_record_episodeswiresZedRecorderSessiononto the executor lifecycle:open()once per run, a take per episode,close()at exit.Recording stops in a
finally; shipping does not. On an abnormal end these want opposite things: a partial plan must not reach the arm, but a take left open records until the disk fills. Failure handling is asymmetric for the same reason —start_takeraises (an episode that cannot record burns hardware time and a human scene reset for nothing),stop_takenever does (by then the arm has moved, and a recording problem must not destroy the run around it, so the take is logged and marked unusable).Recording and live
"zed"perception are refused together: both open the same cameras and a ZED admits one owner.real_robot_snapshot_rebuildthen restores per-episode task rebuild without reopening that conflict. A snapshot opens no cameras — it is a second, short take on the recorder's already-open session, taken between episodes while no episode take is running. It plugs in atRealRobot.reset_env, which homes the arm, waits for the human, and only then callsperception.observe()— exactly when a snapshot should be taken.Defaults come from the pipeline's own measurements: HD720 at 60 fps, not 30, because a real cascade's topple onsets came 6, 4 and 2 frames apart at 30 fps and those inter-domino intervals are what the friction fit will be scored on.
What this does NOT do yet
The learner still does not see real observations. The fit reads
_fit_trajectories, which is theobsstream out ofenv.step. This series produces.svotakes and marks which are usable; nothing yet runs the markerless pipeline over them or hands the resulting pose track to the fit.That is Step 3, and it is where the real poses reach the learner — through the objective, not the trajectory. Concretely still missing: running
reconstruct_dominoes_markerless.pyover each take, the scope fix (§3.1), event alignment (§3.2), the propagation-interval residual (§3.3), andcode_sim_learning_rollout_score_observed_only(§3.7).Running with
real_robot_open_loop_episodeon and Step 3 absent would make the fit worse, not better — zero real observations instead of six.Verification
finallyreds the shipping-raises test;meta["errors"]reds the unusable-take test.ZedRecorderSessionand the markerless staging, run againstb45ac97viaPYTHONPATH. They skip where the submodule is absent, matching the convention intest_real_robot_bridge.py.No hardware validation. None of this has been run on the arm or the cameras.
Reviewer's attention
cogman.pygained four call sites, not the two the plan named. Reading the control flow, tworaisepaths leave the function without either — harmless for Step 1, but it would strand a recording under Step 2.test_cogman.pygained the no-op, since cogman now callsfinish_executionon theBaseEnvit is declared to take.real_robot_snapshot_z_modedefaults to"contact", opposite the cascade default, because a rebuild looks at a scene just arranged upright on the table.mypy.inigains apose_estimation.*stanza for the same reasonbabyrobot.*has one.