[bokalnv]

Summary

Report dataset KPIs from generation records

Detailed description

• Why: a run reports only success and attempt counts. Dataset users also need yield and cost, demo quality, coverage of the reset ranges, and why attempts fail.
• What changed:
  • generate_dataset.py --result_file adds a run block to the result JSON (seed, stop rule, device, env count, control step, setup and generation time, source digest, reset ranges) and writes <result_file stem>.attempts.jsonl, one line per attempt (outcome, failure cause, failed subtask, timing, initial object poses, motion-planner call summaries).
  • Every generated episode records intended_eef_pose and eef_pose per step under obs/datagen_info. Channels whose action commands no pose (tracking-only) record no intended pose, via a new EmbodimentAdapter.get_eef_names_with_pose_action() hook.
  • The cuRobo planner keeps last_plan_summary; annotate_demos.py --summary_file saves annotation counts and rejection reasons.
  • validate_dataset.py --report computes 12 KPIs offline: success rate (Wilson interval, corrected for attempts cancelled at the stop), demos per hour, demos per human demo, annotation yield, tracking error, action saturation, motion smoothness, initial-state coverage, success by initial configuration, failure share, planner failure types, planning time. Definitions and formulas: docs/pages/references/kpis.rst.
• Impact: generation behavior is unchanged; the new files are written only with --result_file. Each episode gains two small per-step pose datasets per end effector.

Testing

• Unit tests (-m "not with_subprocess"): 355 passed, three runs.
• End-to-end tests (autodata_tests/e2e, 22 tests including the Arena workflows): all passed, two runs.
• KPI numbers from Franka MimicGen, GR1 DexMimicGen, SkillGen bin and G1 loco-manipulation runs will follow in a comment.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}