Skip to content

feat(nr-ocx): one compare_mesen2 command owns the Mesen2 comparison recipe and its confounds - #3335

Merged
rmstdope merged 6 commits into
mainfrom
nr-ocx
Oct 4, 2026
Merged

rmstdope merged 6 commits into
mainfrom
nr-ocx

Conversation

@rmstdope

@rmstdope rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Bead: nr-ocx — the Mesen2 comparison recipe is pasted into every sweep bead, and its known confounds keep costing hours.

What changes

  • New python -m scripts.reference_capture.compare_mesen2 <rom> --frames N [N ...] [--out-dir DIR]: owns the Mesen2 flags per system (frame skip off, zero RAM, a standard pad in both ports), runs every Mesen2 and NESER invocation on its own ROM copy and deletes Mesen2's Saves/RecentGames/SaveStates entries for it afterwards, runs NESER with an empty --config, prints Mesen2's [iNes]/[DB] lines (or SNES header block) beside NESER's Loaded rom ... mapper= and Hardware: lines, writes both PNGs and prints differing pixels per frame (exit 0/1/2). --no-game-database passes --nes.DisableGameDatabase=true.
  • scripts/test_compare_mesen2.py: 18 unit tests against fake Mesen/neser executables, run by the gate.
  • scripts/test_mesen2_capture.py imports its flags and wait_for_other_mesen2 from the module (TestMesen2SnesFlags now pins the module's constant).
  • README "Mesen2" section leads with the command and explains each confound; nes/snes research skills and architecture.md name it.

Validation

  • python -m unittest scripts.test_compare_mesen2 scripts.test_mesen2_capture: OK (18 + 3, opt-in ones run too with NESER_MESEN2_CAPTURE_TEST=1: OK).
  • Real run, Mesen2 2.1.1: demo_ntsc.nes --frames 61 120 → 0 / 0 differing pixels, [iNes]/[DB] and NESER's mapper/Hardware lines printed; window-precalculated-single.sfc --frames 72 120 → 0 / 0, SNES header block printed; GTROM_CC_Test1.nes (battery) 0 / 0. Saves, RecentGames, SaveStates listings identical before and after.
  • ./scripts/gate-full.sh: passed.

Deviations from the plan

  • The real runs used main's release NESER binary: this diff touches no Rust.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XgD2SsQeDqM4yvgncxo7vw

@rmstdope

rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Review (cold read, reviewed_head: 3155a5040921b8d371ac2c404d784a2c2578478a) — reviewed before merge under the producer's
review practice, by an agent given the diff and the bead, and not the producer's reasoning.

  1. Blocking — a failed Mesen2 capture reads as a match when --out-dir is reused (scripts/reference_capture/compare_mesen2.py, capture_mesen2). Neither capture deletes out first, and failure is detected only by not out.is_file(), so a rerun into the same --out-dir whose Mesen2 run fails (Lua ERROR:, concurrent testRunner exiting 0 silently, timeout) diffs the previous run's PNG and prints "0 differing pixels". Reproduced with the PR's fixtures: a normal run, then FAKE_LUA_ERROR=1 into the same out dir, exits 0. Fix: out.unlink(missing_ok=True) before each capture, plus a test rerunning into the same out dir.
  2. Blocking — a capture that fails with an exception exits 1 ("a frame differs"), not 2 (compare/capture_*). Only CaptureFailed is caught; FileNotFoundError (missing binary, e.g. unbuilt target/release/neser), subprocess.TimeoutExpired and TimeoutError from wait_for_other_mesen2 escape as a traceback with exit 1. Reproduced: --neser-bin /nonexistent --mesen2-bin /nonexistent exits 1. Catch them (and OSError) as capture failures, with a test.
  3. Copies of the recipe left in place, one already stale: .github/skills/nes-hardware-research/references/source-priority.md (Tier 3, ~line 49) still gives the full Mesen2 command without the NES port flags and without naming compare_mesen2; README-SNES.md (~572–585) and .github/skills/snes-hardware-research/SKILL.md (~105–125) still give hand-copied flag lists and tell people to pin NESER with --snes-controller-port1/2 standard. Beyond the plan, but a copy that already disagrees with the command.
  4. Suggestion — the pgrep guard is narrower than the one it replaces (wait_for_other_mesen2): f"{mesen2} --testRunner" (full path, unescaped regex) no longer sees a concurrent Mesen --testRunner started through PATH or a symlink, the form the snes skill's own loops use. Match on the basename.
  5. Question — SNES Mouse/Super Scope games: the README's workaround appends --mesen2-arg=--snes.port2.type=SnesMouse after the pinned --snes.port2.type=SnesController. Has anyone confirmed Mesen2 lets the later duplicate win? If not, the workaround is unverified.

@rmstdope

rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Answers to the cold review, in 9b1d903:

  1. Fixed. Both captures delete their output PNG before running, so a failed rerun into the same --out-dir cannot diff the last run's file. Test: test_failed_capture_into_a_reused_out_dir_is_not_a_match.
  2. Fixed. OSError (missing binary), subprocess.TimeoutExpired and TimeoutError are caught with CaptureFailed and exit 2 with "capture failed: …". Test: test_missing_binary_is_a_capture_failure (both binaries).
  3. Fixed. references/source-priority.md now names the command instead of the flag list; README-SNES.md step 2/3 and the snes skill's "Plug in the same controllers" bullet name it, keep the flags as the why, and no longer tell people to pin NESER with --snes-controller-portN standard (the command's empty --config does that).
  4. Fixed. testrunner_pattern matches the binary's escaped basename (Mesen --testRunner), as before. Test: test_concurrent_testrunner_is_matched_by_basename.
  5. Rather than rely on Mesen2's handling of a duplicate flag, --mesen2-arg=--key=value now replaces the pinned flag with the same key (merge_flags), so only one port flag reaches Mesen2. Tests: test_an_extra_flag_replaces_the_pinned_flag_with_its_key, test_extra_mesen2_arg_replaces_a_pinned_port. README wording updated.

@rmstdope

rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Review (delta, reviewed_head: 9b1d9034179342d7d8248a3a7247314e91e6a56e, from 3155a504) — reviewed before merge under the producer's
review practice, by an agent given the delta, the previous findings and the answers, and not the producer's reasoning.

Answers 1, 2, 4 and 5 verified against the code; answer 3 partly.

  1. Not blocking (docs): one stale copy remains. .github/skills/snes-hardware-research/references/source-priority.md (Tier 3, lines 124–132) still gives the hand-copied Mesen2 command and says to pin NESER with --snes-controller-port1/2 standard, contradicting answer 3. Optional: snes-hardware-research/SKILL.md lines 89–91 and 388–391 list Mesen2 flags by hand without RamPowerOnState or a pointer to the command.
  2. Suggestion: merge_flags matches keys case-sensitively, and only over the system flags. Pinned flags already mix case (--nes.DisableFrameSkipping / --snes.disableFrameSkipping), so --snes.Port2.Type=SnesMouse would leave the pinned port flag in place; and --mesen2-arg=--timeout=60 yields both --timeout=30 and --timeout=60, since COMMON_MESEN2_FLAGS bypass the merge. Compare keys lower-cased and run the common flags through the merge too.

The delta adds nothing else new and does not need a fresh full review.

@rmstdope

rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

Answers to the delta review, in 938689c:

  1. Fixed. snes-hardware-research/references/source-priority.md Tier 3 now names compare_mesen2 instead of the hand-copied command and the --snes-controller-portN standard pin; the two flag lists in the SNES SKILL.md gain --snes.RamPowerOnState=AllZeros and a pointer to the command.
  2. Fixed. merge_flags compares keys lower-cased, and the common flags (--testRunner, --timeout=30, video) go through the merge too. Tests: test_extra_flag_keys_match_regardless_of_case, test_extra_mesen2_arg_replaces_a_common_flag. Real Mesen2 re-run after the change: demo_ntsc frame 61 and window-precalculated-single frame 72, 0 differing pixels each.

Small, self-contained answers; no further review requested.

rmstdope and others added 4 commits October 4, 2026 12:46
…ecipe and its confounds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XgD2SsQeDqM4yvgncxo7vw
…replace pinned ones, stale recipe copies name the command

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XgD2SsQeDqM4yvgncxo7vw
…e, common flags included; SNES skill copies name the command

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XgD2SsQeDqM4yvgncxo7vw
@rmstdope

rmstdope commented Oct 4, 2026

Copy link
Copy Markdown
Owner Author

CI fix (attempt 1 of 3) in 7fe68ca: the rebase onto main brought in scripts/test_mesen2_traces.py (nr-ggx), which imported NES_FLAGS/SNES_FLAGS from test_mesen2_capture; it now takes NES_MESEN2_FLAGS/SNES_MESEN2_FLAGS and wait_for_other_mesen2 from compare_mesen2. Its NES runs therefore gain the two port flags; the opt-in trace tests (NESER_MESEN2_TRACE_TEST=1) pass with them on 3 of 4 runs here, the one miss a power-on exec-trace case at load average ~40 that passed on rerun and also passes without the flags. Small mechanical change; no further review.

…nstants this PR moved

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XgD2SsQeDqM4yvgncxo7vw
@rmstdope
rmstdope merged commit f6b5463 into main Oct 4, 2026
8 checks passed
@rmstdope
rmstdope deleted the nr-ocx branch October 4, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant