Repository navigation
cosmos3-generate-media: fix the diffusers distilled payload KeyError (NVBug 7034691) + CPU repro eval case - #358
christinayyw wants to merge 2 commits into
Conversation
…load helper (NVBug 7034691) Adds diffusers-distilled-payload-cpu: executes the five reference blocks of references/payloads.md verbatim (section 4.1 paths, the diffusers FIXED_SAMPLING and ASSET_SETS, section 9.1 helpers, the diffusers create_payload) and calls create_payload for t2i (control), t2i_distilled and i2v_distilled, the way QA's reproducer does. No GPU, no model download, no torch. Three verifier checks require the as-written calls to return. On main the distilled calls raise KeyError 'num_steps' (payloads.md line 1446 prints keys the distilled branch removed), so the case fails until the helper is fixed: repro group nvbug7034691-repro, claude/codex/cursor FAIL. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Christina Wang <christwang@nvidia.com>
christinayyw
added
bug
|
/run-skill-eval force cells=*/diffusers-distilled-payload-cpu |
Skill eval — 7e52570 · run #41771138511 · mode force · cells=
|
| Case | claude | codex | cursor | opencode |
|---|---|---|---|---|
diffusers-edge-t2i |
— not selected | — not selected | — not selected | — not selected |
diffusers-edge-t2v |
— not selected | — not selected | — not selected | — not selected |
framework-edge-t2i |
— not selected | — not selected | — not selected | — not selected |
vllm-omni-edge-t2i |
— not selected | — not selected | — not selected | — not selected |
sglang-edge-t2i |
— not selected | — not selected | — not selected | — not selected |
trtllm-edge-t2i |
— not selected | — not selected | — not selected | — not selected |
distilled-trtllm-t2i |
— not selected | — not selected | — not selected | — not selected |
nim-generator-i2v |
— not selected | — not selected | — not selected | — not selected |
diffusers-nano-t2i |
— not selected | — not selected | — not selected | — not selected |
framework-nano-t2i |
— not selected | — not selected | — not selected | — not selected |
trtllm-nano-t2i |
— not selected | — not selected | — not selected | — not selected |
diffusers-distilled-payload-cpu |
❌ FAIL · report | ❌ FAIL · report | ❌ FAIL · report | ✅ PASS · report |
Legend: ⏳ queued · 🟢 running · ✅ PASS · ❌ FAIL · cells= with no result of this skill version yet · (not re-run: n report-less attempts) = 3 consecutive NO-REPORT/ERROR results for this skill version, treated as a harness-side failure (force or cells= re-runs it) · not run = the runner lost docker (job stopped)
❌ done — 1/4 passed (3 failed, 0 no report, 0 errors)
…iffusers payload helper (NVBug 7034691)
references/payloads.md section 9.2, diffusers create_payload: the distilled
specs (t2i_distilled, i2v_distilled) remove num_steps, guidance and shift from
the payload, and the summary print right before the guarded one indexed those
keys unconditionally, so create_payload("t2i_distilled") and
create_payload("i2v_distilled") raised KeyError: 'num_steps' just after writing
the payload file. The guarded summary (summary_keys, if k in payload) prints the
same information for every spec, so the unconditional print is removed. Nothing
else changes: payload content and the return value are the same as before.
CPU check over every payloads.md block and use case: 61 OK / 2 CRASH before,
63 OK / 0 CRASH after (7 video2video use cases skip for assets the skill does
not ship).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Christina Wang <christwang@nvidia.com>
|
/run-skill-eval cells=*/diffusers-distilled-payload-cpu |
Skill eval — 1fd8dc4 · run #4177665193048 / 48 cells done · 48 passed · 0 failed · 0 no report · 0 errors · 4 reused · started 2026-11-04 05:01:30 UTC · updated 2026-11-04 10:37:04 UTC
|
| Case | claude | codex | cursor | opencode |
|---|---|---|---|---|
diffusers-edge-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
diffusers-edge-t2v |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
framework-edge-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
vllm-omni-edge-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
sglang-edge-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
trtllm-edge-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
distilled-trtllm-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
nim-generator-i2v |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
diffusers-nano-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
framework-nano-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
trtllm-nano-t2i |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
diffusers-distilled-payload-cpu |
✅ PASS · report | ✅ PASS · report | ✅ PASS · report | ✅ PASS · report |
Previous runs on this PR
- 1fd8dc4 · run #41775651090 · ✅ done — 4/4 passed (0 failed, 0 no report, 0 errors)
- 1fd8dc4 · run #41774597407 · ❌ done — 3/4 passed (0 failed, 1 no report, 0 errors)
- 7e52570 · run #41771138511 · ❌ done — 1/4 passed (3 failed, 0 no report, 0 errors)
Legend: ⏳ queued · 🟢 running · ✅ PASS · ❌ FAIL · cells= with no result of this skill version yet · (not re-run: n report-less attempts) = 3 consecutive NO-REPORT/ERROR results for this skill version, treated as a harness-side failure (force or cells= re-runs it) · not run = the runner lost docker (job stopped)
✅ done — 48/48 passed (0 failed, 0 no report, 0 errors)
|
/run-skill-eval cells=opencode/diffusers-distilled-payload-cpu |
|
/run-skill-eval |
Summary
Skill-only change to
cosmos3-generate-media(NVBug 7034691, found by QA on the merged skill):diffusers-distilled-payload-cpu, first commit7e52570). It mirrors the QA reproduction steps: execute the reference blocks ofreferences/payloads.mdverbatim (section 4.1 paths, the diffusersFIXED_SAMPLINGandASSET_SETStables, the section 9.1 helpers and the diffuserscreate_payload), then callcreate_payload("t2i")as the control pluscreate_payload("t2i_distilled")andcreate_payload("i2v_distilled"), all withbackend="diffusers". No GPU, no model download, no torch. Three verifier checks require every as-written call to return; the prompt forbids patching the skill, so a crash in the reference block is the case's result.1fd8dc4): one deleted line, the minimal change that makes the documented calls return.The bug
references/payloads.md, section 9.2, diffuserscreate_payload: for the two distilled specs (t2i_distilled,i2v_distilled, the only specs with"distilled": True) the helper removesnum_steps,guidanceandshiftfrom the payload (the 4-step checkpoints supply them), writes the payload file, and then prints a summary that indexes those three keys unconditionally:The guarded summary two lines below (
for k in summary_keys if k in payload) already handles both cases, so the unconditional print is a duplicate that only differs by crashing. The payload file is written before the crash, which is why the GPU notebooks still run when started from the written files, and why the bug only shows when the helper's return value is used as documented. Every other use case (61 of the 70 in the matrix) returns normally.The same text came from the upstream
run_with_diffusers.ipynb(the print was re-added there after a V2V merge); the skill migration carried it as-is, so the bug predates the skill.Why the existing eval did not catch it
The ten GPU cases run
diffusers-edge-*anddiffusers-nano-*but no diffusers distilled case (the distilled case runs on TensorRT-LLM), the smoke snippets build payloads by hand instead of calling the helper, and no tier of the eval executes the reference code on its own. The new case closes exactly that gap and runs in minutes on any runner.Reproduction (eval CI on this PR, before the fix)
Head
7e52570= the eval case only, skill text as onmain. Trigger/run-skill-eval force cells=*/diffusers-distilled-payload-cpu(forcebecause the case's skill content matched the manual pre-run, which the CI would otherwise have reused) → Actions run 41771138511, CI report comment, dashboard group ci-pr358-7e52570.diffusers-distilled-payload-cpuAll four backends hit the bug exactly as reported:
create_payload("t2i")returns,create_payload("t2i_distilled")andcreate_payload("i2v_distilled")write their payload files and then raiseKeyError: 'num_steps'from the unconditional print (payloads.md line 1446). claude, codex and cursor report FAIL on the first attempt. opencode's tester reproduced the sameKeyError(its assertion table marks the distilled-calls assertion as failed, 4/5) but still wrote a PASS verdict after a cwd self-correction of its own driver; the case asks for FAIL when a documented call raises, so that cell is tester leniency, not evidence that the skill works.A manual pre-run of the same case on
main(d04e506, same skill content) is recorded as dashboard group nvbug7034691-repro: claude, codex and cursor FAIL the same way; opencode's PASS there is a tester error (its driver never executed the unconditional print).Fix
Commit
1fd8dc4: one deleted line inreferences/payloads.md(section 9.2, diffuserscreate_payload), the unconditional summary print. The guarded summary right below prints the same keys when present, followed by the existing "distilled: step count and guidance come from the checkpoint config" note. Payload content, environment variables and the return tuple are unchanged; non-distilled use cases now print the summary once instead of twice.The block's ending now matches the upstream notebook revision from NVIDIA/cosmos PR #292 (the duplicate print came back with #299). Checked with a CPU matrix that executes every
create_payloadblock of payloads.md for every use case: 61 OK / 2 CRASH before, 63 OK / 0 CRASH after (the 7 video2video use cases skip for an asset the skill does not ship). The other blocks keep their unconditional prints because none of them removes keys for any spec in itsASSET_SETS(framework, vllm_omni) or already guards the print (sglang, TRT-LLM).Not in this PR, as follow-ups: the same cell in the private notebook source; the section 9 lead-in ("every other line is verbatim") now has this one deliberate divergence; the authoring skill's "11 eval cases" counts.
Validation
Fixed case, all four backends (
/run-skill-eval cells=*/diffusers-distilled-payload-cpuon1fd8dc4): Actions runs 41774597407 and 41775651090, dashboard group ci-pr358-1fd8dc4. The second run re-ran only the opencode cell: in the first run that agent stopped after 120 output tokens without writing a report (NO-REPORT, a tester flake with no skill content involved).diffusers-distilled-payload-cpu4/4 PASS, 4/4 first-attempt. The reference blocks ran verbatim and every documented call returned; opencode's report notes a first-attempt stumble of its own driver (launched from the wrong working directory), which the grading attributes to the tester, not to the skill.
Full matrix (12 cases × 4 backends,
/run-skill-evalon1fd8dc4): running; the results table is added here when it completes, with any cell re-run until every cell passes on the first attempt.🤖 Generated with Claude Code