diff --git a/.github/skills/nes-hardware-research/SKILL.md b/.github/skills/nes-hardware-research/SKILL.md index 4e19a03b4..4a839002d 100644 --- a/.github/skills/nes-hardware-research/SKILL.md +++ b/.github/skills/nes-hardware-research/SKILL.md @@ -20,7 +20,7 @@ Every hardware research skill in this repository uses the same three tiers. The Used only when the specification is missing, incomplete, or ambiguous, to see how a specific behavior can be implemented. Implementation evidence is never equal authority with the specification. Where Mesen2 makes a choice the specification does not settle, say so instead of presenting it as hardware fact. No second implementation reference is designated for the NES; the legacy `SourMesen/Mesen` repository is history, not an authority. 3. **Screenshot reference: Mesen2.** - When a visual test ROM needs a reference image, capture it with Mesen2 at the same frame as NESER and pixel-diff the two captures with `python -m scripts.diff_screenshots`. An exact match approves the golden. If the captures differ and Mesen2 itself is suspect, ask the navigator instead of approving either side. The verified headless recipe (Mesen2 release binary in `/Applications/Mesen.app`, testRunner mode, `scripts/reference_capture/mesen2_capture.lua`, `--nes.DisableFrameSkipping=true --nes.RamPowerOnState=AllZeros`) is in `scripts/reference_capture/README.md`; it produced a 0-px match against NESER on instr_test-v5 at frame 120 on 2026-09-25. + When a visual test ROM needs a reference image, capture it with Mesen2 at the same frame as NESER and pixel-diff the two captures with `python -m scripts.diff_screenshots`. An exact match approves the golden. If the captures differ and Mesen2 itself is suspect, ask the navigator instead of approving either side. Run the comparison with `python -m scripts.reference_capture.compare_mesen2 --frames N` (Mesen2 release binary in `/Applications/Mesen.app`, testRunner mode, `scripts/reference_capture/mesen2_capture.lua`); it owns the flags, isolates battery saves per run and prints Mesen2's `[iNes]`/`[DB]` lines beside NESER's mapper and region, and `scripts/reference_capture/README.md` explains each confound it removes. Name that command in a bead rather than copying flags. The underlying recipe produced a 0-px match against NESER on instr_test-v5 at frame 120 on 2026-09-25. ## Instructions diff --git a/.github/skills/nes-hardware-research/references/source-priority.md b/.github/skills/nes-hardware-research/references/source-priority.md index 48fb487eb..9b92a9967 100644 --- a/.github/skills/nes-hardware-research/references/source-priority.md +++ b/.github/skills/nes-hardware-research/references/source-priority.md @@ -45,10 +45,9 @@ specification authority, one or two implementation references, one screenshot re - The only emulator whose captures approve a NESER golden frame. - Binary: `/Applications/Mesen.app/Contents/MacOS/Mesen` (official release zip; no Homebrew cask). Verified recipe, scripts and the `AllowIoOsAccess` toggle: - `scripts/reference_capture/README.md`. In short: - `CAPTURE_FRAME= CAPTURE_OUT= Mesen --testRunner --enableStdout --timeout=30 - --Video.VideoFilter=None --Video.AspectRatio=NoStretching --nes.DisableFrameSkipping=true - --nes.RamPowerOnState=AllZeros scripts/reference_capture/mesen2_capture.lua`. + `scripts/reference_capture/README.md`. Compare with one command, which owns the flags + (frame skip off, zero RAM, a standard pad in both ports) and isolates battery saves per run: + `python -m scripts.reference_capture.compare_mesen2 --frames --out-dir `. - Diff with `python -m scripts.diff_screenshots --shift-search 1`. ## Reporting rules diff --git a/.github/skills/snes-hardware-research/SKILL.md b/.github/skills/snes-hardware-research/SKILL.md index b52a96e4c..cfc6a597c 100644 --- a/.github/skills/snes-hardware-research/SKILL.md +++ b/.github/skills/snes-hardware-research/SKILL.md @@ -20,7 +20,7 @@ Every hardware research skill in this repository uses the same three tiers. The Used only when the specification is missing, incomplete, or ambiguous, to see how a specific behavior can be implemented. Mesen2 (`https://github.com/SourMesen/Mesen2`, `Core/SNES/`) is consulted first, because it is also the project's screenshot reference and a deliberate divergence from it must be commented at the call site; ares (`https://github.com/ares-emulator/ares`, `ares/sfc/`) is the second, independent lineage, consulted when Mesen2 has no model for the behavior or when two implementations agreeing would settle a question. Implementation evidence is never equal authority with the specification. Where an emulator makes a choice the specification does not settle, say so instead of presenting it as hardware fact. bsnes, higan and ares-performance are the same lineage as ares and count as one opinion; Snes9x is not an authority but may break a Mesen2-vs-ares tie (see `references/source-priority.md`). 3. **Screenshot reference: Mesen2** (navigator decision in #3000). - When a visual test ROM needs a reference image, capture it with Mesen2 at the same frame as NESER and pixel-diff the two captures with `python -m scripts.diff_screenshots`. An exact match approves the golden. If the captures differ and Mesen2 itself is suspect, ask the navigator instead of approving either side. ares is never used for screenshots. The headless capture recipe is in "Automating Screenshot Capture at Specific Frames" below; the reusable script is `scripts/reference_capture/mesen2_capture.lua` (see `scripts/reference_capture/README.md`, shared with the NES). + When a visual test ROM needs a reference image, capture it with Mesen2 at the same frame as NESER and pixel-diff the two captures with `python -m scripts.diff_screenshots`. An exact match approves the golden. If the captures differ and Mesen2 itself is suspect, ask the navigator instead of approving either side. ares is never used for screenshots. The headless capture recipe is in "Automating Screenshot Capture at Specific Frames" below; the reusable script is `scripts/reference_capture/mesen2_capture.lua`, and a NESER-against-Mesen2 comparison is one command, `python -m scripts.reference_capture.compare_mesen2 --frames N` (see `scripts/reference_capture/README.md`, shared with the NES). Name that command in a bead rather than copying its flags. ## Instructions @@ -87,7 +87,7 @@ Every hardware research skill in this repository uses the same three tiers. The - Capture a Mesen2 screenshot at the same frame as NESER and pixel-diff programmatically; exact matches become the reference for NESER comparison. - If NESER and Mesen2 disagree and the divergence is suspected to be a Mesen2 quirk, **ask the user** how to proceed rather than approving either side unilaterally. - Screenshot settings for comparable captures: - - Mesen2: `--Video.VideoFilter=None --Video.AspectRatio=NoStretching --snes.disableFrameSkipping=true --snes.port1.type=SnesController --snes.port2.type=SnesController` + - Mesen2: `--Video.VideoFilter=None --Video.AspectRatio=NoStretching --snes.disableFrameSkipping=true --snes.port1.type=SnesController --snes.port2.type=SnesController` plus `--snes.RamPowerOnState=AllZeros`; `python -m scripts.reference_capture.compare_mesen2` passes all of them - Mesen2 headless mode: `Mesen --testRunner --enableStdout --timeout=N ` - **`--snes.disableFrameSkipping=true` is mandatory for animated content** (found in #2990): headless testRunner emulation runs >100 fps, engaging `_skipRender` (SnesPpu.cpp) which @@ -113,11 +113,13 @@ Every hardware research skill in this repository uses the same three tiers. The - **Plug in the same controllers on BOTH sides** (nr-0an). Mesen2's testRunner takes its SNES ports from `settings.json`, and a local install may have port 2 empty, while NESER has a standard pad in each port by default. Pass - `--snes.port1.type=SnesController --snes.port2.type=SnesController` to Mesen2, and pin - NESER's side with `--snes-controller-port1 standard --snes-controller-port2 standard`, - since a `neser.conf` port line (e.g. `multitap`) would otherwise apply. For a game NESER - recognises as a Mouse or Super Scope game, NESER picks that device itself; give Mesen2 - the same type (`SnesMouse`, `SuperScope`) instead. Games that read which pads are + `--snes.port1.type=SnesController --snes.port2.type=SnesController` to Mesen2, and keep + a `neser.conf` port line (e.g. `multitap`) from applying on NESER's side. + `python -m scripts.reference_capture.compare_mesen2` does both (it runs NESER with an + empty `--config`). For a game NESER recognises as a Mouse or Super Scope game, NESER + picks that device itself; give Mesen2 the same type instead with + `--mesen2-arg=--snes.port2.type=SnesMouse` (or `SuperScope`), which replaces the pinned + flag for that port. Games that read which pads are connected play differently otherwise: Super Bomberman 3's attract demo took a lag frame on Mesen2 only, which looked like a CPU-timing drift until a trace showed a branch on the game's "player 2 connected" byte going the other way. The lists are @@ -385,7 +387,7 @@ When verifying SNES emulator accuracy: - **CRC-based integration tests**: Capture frame CRCs at known stable points (e.g., frame 600) and use as golden values for regression testing. Update test comments to reference GitHub issues for known differences. - **Screenshot settings for comparable captures**: - Mesen2: `--Video.VideoFilter=None --Video.AspectRatio=NoStretching --snes.disableFrameSkipping=true - --snes.port1.type=SnesController --snes.port2.type=SnesController` + --snes.port1.type=SnesController --snes.port2.type=SnesController` plus `--snes.RamPowerOnState=AllZeros`; `python -m scripts.reference_capture.compare_mesen2` passes all of them (the frame-skip switch is mandatory for animated content; see step 9 of the Instructions; the port flags match NESER's default controllers, see "Plug in the same controllers") - Since the BG vertical-scroll display-line fix (issue #2945, PR #2981), NESER and diff --git a/.github/skills/snes-hardware-research/references/source-priority.md b/.github/skills/snes-hardware-research/references/source-priority.md index 15be28a46..e463733ee 100644 --- a/.github/skills/snes-hardware-research/references/source-priority.md +++ b/.github/skills/snes-hardware-research/references/source-priority.md @@ -122,14 +122,10 @@ specification authority, one or two implementation references, one screenshot re 7. **Mesen2** (navigator decision in #3000) - The only emulator whose captures approve a NESER golden frame. - - Headless test mode: `Mesen --testRunner --enableStdout --timeout=N ` - - Screenshot settings: `--Video.VideoFilter=None --Video.AspectRatio=NoStretching - --snes.disableFrameSkipping=true` (the frame-skip switch is mandatory for animated - content, see SKILL.md), plus `--snes.RamPowerOnState=AllZeros` for any ROM that can - display uninitialised WRAM; pin NESER's side with `--ram-init-mode zero` too. - Always add `--snes.port1.type=SnesController --snes.port2.type=SnesController`, the - standard pads NESER has in each port by default, and pin NESER with - `--snes-controller-port1 standard --snes-controller-port2 standard` (nr-0an). + - Compare with one command, which owns the flags (frame skip off, zero RAM, a standard pad + in both ports, NESER on an empty `--config`) and isolates battery saves per run: + `python -m scripts.reference_capture.compare_mesen2 --frames --out-dir ` + (`scripts/reference_capture/README.md` says why each flag is there). - Capture twice before trusting any non-zero diff; a capture that changes between identical runs means the reference is not pinned. - Diff the captures with `python -m scripts.diff_screenshots --shift-search 1` diff --git a/README-SNES.md b/README-SNES.md index d8a4ce56e..5c5f3c10a 100644 --- a/README-SNES.md +++ b/README-SNES.md @@ -568,8 +568,10 @@ directory and are never committed. To approve a new or changed golden: 1. Run the test with `NESER_CAPTURE_SCREEN=1` to write a PNG per test under `target/snes_test_captures//` (each suite's source file documents its specific recording steps). -2. Capture the Mesen2 ground truth for the same ROM/frame (headless - `--testRunner` with a Lua screenshot script). Always pass +2. Capture the Mesen2 ground truth for the same ROM/frame with + `python -m scripts.reference_capture.compare_mesen2 --frames ` + (see `scripts/reference_capture/README.md`), which passes every flag below + and this step's NESER side; the flags are listed here for why. It passes `--Video.VideoFilter=None --Video.AspectRatio=NoStretching --snes.disableFrameSkipping=true` — without the frame-skip switch Mesen2's testRunner renders only every other frame and screenshots of @@ -579,8 +581,8 @@ directory and are never committed. To approve a new or changed golden: Mesen2 otherwise takes its controller ports from its own settings, which may leave port 2 empty, and games that check which pads are connected then play differently from NESER, which has a standard pad in each port (nr-0an). -3. If the ROM can display uninitialised RAM, add - `--snes.RamPowerOnState=AllZeros` to the Mesen2 command line. Mesen2's SNES +3. If the ROM can display uninitialised RAM, Mesen2 needs + `--snes.RamPowerOnState=AllZeros` (`compare_mesen2` passes it). Mesen2's SNES default is `RamState::Random`, and without the flag the ground truth is not even self-consistent: for `test_dmatiming/demo.smc` two default Mesen2 captures differ from *each other* by 1.06%, which is what produced #3063's diff --git a/architecture.md b/architecture.md index fee8a4fc7..7e767e78c 100644 --- a/architecture.md +++ b/architecture.md @@ -106,7 +106,7 @@ The `src/bin/roms.rs` file is a library binary (accessed via `cargo run --bin ro | `scripts/diff_bus_traces.py` | Diffs two SNES CPU bus traces (NESER `--trace-cpu=2` and a reference emulator's) by event **ordinal** rather than by clock, so a constant clock origin difference is ignored and the ordinal where the per-cycle offset *steps* names the exact cycle whose accounting diverges. Auto-aligns the leading edge (a clock window never opens on the same cycle in both emulators) and reports a clock-offset histogram plus side-by-side context. This is what localised #3050 to a single DMA end pad. | | `scripts/diff_timing_traces.py` | Diffs a NESER timing trace (`timing_trace`) against Mesen2's (`mesen2_nmi_clock.lua`, `mesen2_exec_trace.lua`) by line ordinal: the first pair's clock offset is the baseline, and the first line whose PC or NMI number differs, or whose offset leaves the baseline and stays off it, is the divergence. A one-line excursion is listed as a stamp difference (a stall charged on the other side of an instruction boundary), not failed. Exit 0 match, 1 divergence, 2 no trace lines. | | `scripts/diff_screenshots.py` | Pixel-diffs two emulator screenshots for the Mesen2 golden-approval workflow. Reports the per-pixel difference the 0-pixel approval rule is written against, an optional `--shift-search N` row/column offset search (a non-zero best shift is evidence of a bug, not a capture convention, so the exit code stays non-zero regardless), and `--rows`, the per-row mean-luminance vector of each image plus the lag that best aligns them -- the diagnostic for scanline-banding ROMs, whose whole signal a 2D shift search cannot express. | -| `scripts/reference_capture/` | Verified headless screenshot recipes for the screenshot-reference emulators (`README.md`), the Lua capture scripts for Mesen2 (`mesen2_capture.lua`, NES and SNES), the Mesen2 timing-trace scripts (`mesen2_nmi_clock.lua`, `mesen2_exec_trace.lua`) with the Lua API behaviours they rely on, and `mgba-headless` (`mgba_capture.lua`), and the one-hunk patch that gives stock `mgba-headless` a framebuffer. Every hardware research skill's Tier 3 points here. | +| `scripts/reference_capture/` | Verified headless screenshot recipes for the screenshot-reference emulators (`README.md`), the NESER-against-Mesen2 comparison command (`compare_mesen2.py`, which owns the Mesen2 flags and isolates battery saves per run), the Lua capture scripts for Mesen2 (`mesen2_capture.lua`, NES and SNES), the Mesen2 timing-trace scripts (`mesen2_nmi_clock.lua`, `mesen2_exec_trace.lua`) with the Lua API behaviours they rely on, and `mgba-headless` (`mgba_capture.lua`), and the one-hunk patch that gives stock `mgba-headless` a framebuffer. Every hardware research skill's Tier 3 points here. | | `scripts/sort_roms.py` | Sorts ROM files into mapper-numbered subdirectories based on their iNES header. | | `scripts/prepare_release.py` | Applies a release to the tree for the release skill (`.claude/skills/release/SKILL.md`): computes the next version for a maintenance, minor or major release, sets it in `Cargo.toml` and `Cargo.lock`, replaces the web frontend's idle `SCROLLER_TEXT` sentence with the release date, version and highlights, and copies the navigator-approved notes to `docs/releases/v.md`. `--print-version` only reports the next version. | | `scripts/package_release.py` | Builds per-target release archives with a top-level `neser/` directory. Includes the binary, runtime resources, README, LICENSE, config example, fonts, and only shader files reachable from configured shader presets. Excludes development scripts. | diff --git a/docs/retrospectives/nr-ocx.md b/docs/retrospectives/nr-ocx.md new file mode 100644 index 000000000..ea2ed64b0 --- /dev/null +++ b/docs/retrospectives/nr-ocx.md @@ -0,0 +1,27 @@ +# nr-ocx — retrospective + +- **Implementer:** Storm +- **Date:** 2026-10-04 +- **PR:** #3335 + +## A rebase brought in a test that imported constants this PR had moved, and only CI saw it + +**What happened.** This PR moved the Mesen2 flag lists `NES_FLAGS`/`SNES_FLAGS` out of +`scripts/test_mesen2_capture.py` into `scripts/reference_capture/compare_mesen2.py`. While it was +in review, nr-ggx (#3333) merged `scripts/test_mesen2_traces.py`, which imports those two names +from `test_mesen2_capture`. The rebase applied without a conflict in any Python file. After it I +ran only the two Mesen2 test modules, ruff and mypy, all green. CI's `python-tests` then failed +with `ImportError: cannot import name 'NES_FLAGS' from 'scripts.test_mesen2_capture'`. + +**Why.** A semantic conflict: textually separate files, so git had nothing to merge. mypy did not +catch it either, because `scripts/pyproject.toml` sets `ignore_errors = true` for every module +except a short list, and the test modules are not on it. + +**Cost.** One CI round (about five minutes) and one of the three fix attempts. + +**Prevent by.** After a rebase onto a main that changed anything under `scripts/`, rerun the full +`python -m unittest discover -s scripts -t . -p "test_*.py"` (about a minute) before pushing, +not only the modules the PR touches. A step for that belongs in `produce-bead`'s *Merging*, +where a `422` conflict sends the producer to rebase. + +**Seen before.** none found diff --git a/scripts/reference_capture/README.md b/scripts/reference_capture/README.md index b3e0d66de..327067cac 100644 --- a/scripts/reference_capture/README.md +++ b/scripts/reference_capture/README.md @@ -13,7 +13,7 @@ Mesen2 recipe was re-verified on animated NES and SNES content on 2026-09-27 (nr | System | Reference | Tool | Frame-exact | |---|---|---|---| -| NES, SNES | Mesen2 2.1.1 | `Mesen --testRunner` + `mesen2_capture.lua` | yes | +| NES, SNES | Mesen2 2.1.1 | `compare_mesen2` (`Mesen --testRunner` + `mesen2_capture.lua`) | yes | | GB / CGB | SameBoy 1.0.3 | `sameboy_tester` (built from source, patched) | no, time-based | | GBA | mGBA 0.11 (git) | `mgba-headless` (built from source, patched) + `mgba_capture.lua` | yes | @@ -54,27 +54,70 @@ checks for that before anything else: it prints one `ERROR: Lua file access is o "AllowIoOsAccess": true ...` line and stops Mesen2 with exit code 1 at once. Without that check, the run would sit silently until `--timeout` (nr-hg7). +### Comparing a ROM: `compare_mesen2` + +Compare a NES or SNES ROM with this one command; sweep beads name it rather than copying flags, +so a confound found later is fixed in the command once: + ```bash -CAPTURE_FRAME=120 CAPTURE_OUT="$PWD/mesen.png" \ - /Applications/Mesen.app/Contents/MacOS/Mesen --testRunner --enableStdout --timeout=30 \ - --Video.VideoFilter=None --Video.AspectRatio=NoStretching \ - --nes.DisableFrameSkipping=true --nes.RamPowerOnState=AllZeros \ - scripts/reference_capture/mesen2_capture.lua +cargo build --release --bin neser +python -m scripts.reference_capture.compare_mesen2 --frames 120 600 --out-dir out/ +``` + +It prints what each emulator decided about the ROM, then one line per frame, and writes +`mesen2-.png` and `neser-.png` into `--out-dir` (a new temporary directory when it is +left out) for `python -m scripts.diff_screenshots`. Exit status 0 means every frame matched, 1 a +frame differs, 2 a capture failed. + +```text +Mesen2: [iNes] Mapper: 0 Sub: 0 +Mesen2: [DB] Game not found in database +NESER: Loaded rom with CRC32: 5CE951EA, mapper=0, submapper=0, PRG-ROM=32KB, CHR-ROM=8KB +NESER: Hardware: NES (NTSC) | Port 1: Joypad | Port 2: Joypad +frame 120: 0 differing pixels ``` -For the SNES replace the two `--nes.*` flags with `--snes.disableFrameSkipping=true ---snes.RamPowerOnState=AllZeros --snes.port1.type=SnesController ---snes.port2.type=SnesController`. The port flags plug in the standard pad NESER has in each -port by default; without them Mesen2 takes the ports from `settings.json`, where port 2 may -be empty. Pin NESER's side too with `--snes-controller-port1 standard --snes-controller-port2 -standard`, since a `neser.conf` port line would otherwise apply; for a game NESER recognises as -a Mouse or Super Scope game, give Mesen2 that type (`SnesMouse`, `SuperScope`) instead. Games that read which pads are connected then play differently: Super Bomberman 3's -attract demo lags a frame at frame 2917 with port 2 empty, which showed as a 0.9% difference -at frame 3000 with no emulation difference behind it (nr-0an). The frame-skip flag is mandatory for animated content: +The command owns the flags and removes the confounds earlier comparisons paid for: + +- **Battery saves** (nr-kds, nr-nuf, nr-7v3). Both emulators load a save by ROM file name, and + Mesen2 keeps its saves in `~/Library/Application Support/Mesen2/Saves/`, not beside the ROM. + A name shared between runs (one `fresh.nes` for every game, or one copy per game reused across + checkpoints) boots later runs on an earlier run's SRAM. Every invocation of either emulator + runs on its own copy (`--.`), and Mesen2's `Saves`, + `RecentGames` and `SaveStates` entries for that copy are deleted afterwards. +- **Mesen2's game database** (nr-nwy, nr-sjt, nr-1le). Mesen2 replaces the iNES header with its + database entry without saying so on screen, and misses the database for a file with trailing + data (it hashes everything after the header). The `[iNes]` and `[DB]` lines show which it used: + a `[DB] Mapper:` that differs from `[iNes] Mapper:`, `Game not found in database` or `File is + larger than expected` means the two emulators may not be running the same board. Compare + NESER's `mapper=` before filing a difference; `--no-game-database` makes Mesen2 use the header + (`--nes.DisableGameDatabase=true`). Mesen2's database is also wrong for some Vs. System PPUs + (nr-1le): it is not the reference for a Vs. game's palette. +- **Controller ports** (nr-0an). Without port flags Mesen2 takes its ports from + `settings.json`, where port 2 may be empty, and games that read which pads are connected play + differently (Super Bomberman 3's attract demo lagged a frame at 2917). Both systems get a + standard pad in each port (`--nes.portN.type=NesController`, + `--snes.portN.type=SnesController`), and NESER runs with an empty `--config` file, so a + `neser.conf` port or palette line cannot apply; NESER's defaults are the same pads. For a + Mouse or Super Scope game (NESER picks the device itself from + `src/snes/input/mouse_games.rs` and `super_scope_games.rs`) pass the Mesen2 type with + `--mesen2-arg`, which replaces the pinned flag with the same key, and NESER's port with + `--neser-arg` (for example `--mesen2-arg=--snes.port2.type=SnesMouse --neser-arg=--snes-controller-port2 + --neser-arg=mouse`). +- **Region** (nr-f6o). NESER's `Hardware:` line names the region it picked; NESER's ROM database + makes some "NTSC" files PAL or Dendy. Record it per ROM with a sweep's results, so a + region-gated change is read per region. NESER prints no cartridge lines for SNES ROMs. + +What it runs, for reading only: Mesen2 `--testRunner --enableStdout --timeout=30 +--Video.VideoFilter=None --Video.AspectRatio=NoStretching`, frame skipping off, zero RAM and the +port flags above, with `CAPTURE_FRAME=N CAPTURE_OUT=` and `mesen2_capture.lua`; +NESER `--config --headless --frames N` (`--nes-palette mesen` for the NES), whose +`--headless` forces zero RAM. The frame-skip flag is mandatory for animated content: testRunner emulation runs far faster than real time and otherwise renders only every other -frame. `CAPTURE_OUT` must be absolute. The script prints `SAVED ` and stops the -emulator; the whole run takes about a second. The stdout log also lists the mapper, -CRCs and any uninitialised-memory reads, which is useful in itself. +frame. The capture script prints `SAVED ` and stops the emulator; one run takes about a +second. Do not run two Mesen2 testRunners at once: the second exits 0 with no output, so the +command waits for any other `Mesen --testRunner` to finish first. `MESEN2_BIN`, `NESER_BIN` and +`MESEN2_HOME` (or `--mesen2-bin`, `--neser-bin`, `--mesen2-home`) point it elsewhere. Frame numbering: the script counts `startFrame` events from power-on and, at the N-th, reads the pixels with `emu.getScreenBuffer()`, so `CAPTURE_FRAME=N` is the N-th emulated @@ -115,6 +158,9 @@ gate builds none). It leaves `AllowIoOsAccess` alone. The same module's `TestMesen2CaptureWithoutFileAccess` (nr-hg7) checks the file-access error above. It simulates the setting being off with a shim that sets `io` and `os` to nil, and needs Mesen2 but no NESER build. +`scripts/test_compare_mesen2.py` pins `compare_mesen2` itself (a ROM copy per Mesen2 +invocation, its save removed afterwards, the flags per system and the printed cartridge lines) +against fake emulators, so it runs in the gate with no Mesen2 installed. ## Tracing against Mesen2 (NES and SNES) @@ -138,7 +184,7 @@ patching. `timing_trace` reads the PC, the clock and two trace counters the core cargo build --release --features native --bin timing_trace ROM=; T="$PWD/trace"; mkdir -p "$T" MESEN=/Applications/Mesen.app/Contents/MacOS/Mesen -FLAGS="--nes.RamPowerOnState=AllZeros" # SNES: the four --snes.* flags from "Mesen2" above +FLAGS="--nes.RamPowerOnState=AllZeros" # SNES: SNES_MESEN2_FLAGS in compare_mesen2.py # 1. The first NMI entry whose clock differs. target/release/timing_trace nmi "$ROM" --nmis 3600 --out "$T/neser_nmi.txt" diff --git a/scripts/reference_capture/__init__.py b/scripts/reference_capture/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/scripts/reference_capture/compare_mesen2.py b/scripts/reference_capture/compare_mesen2.py new file mode 100644 index 000000000..4c7a83b63 --- /dev/null +++ b/scripts/reference_capture/compare_mesen2.py @@ -0,0 +1,273 @@ +"""Compare a NES or SNES ROM between NESER and Mesen2 at given frames (nr-ocx). + + python -m scripts.reference_capture.compare_mesen2 --frames 120 600 [--out-dir DIR] + +This command is the Mesen2 comparison recipe; sweep beads and the README name it rather than +copying its flags, so a confound is fixed here once. For each frame it runs Mesen2's +testRunner with ``mesen2_capture.lua`` and ``neser --headless``, writes ``mesen2-.png`` and +``neser-.png`` into the output directory and prints the number of differing pixels. Before +the frames it prints what each emulator decided about the ROM: Mesen2's ``[iNes]``/``[DB]`` +lines (or its SNES header block) beside NESER's ``Loaded rom ... mapper=`` and ``Hardware:`` +lines. Exit status: 0 every frame matches, 1 a frame differs, 2 a capture failed. + +The confounds it removes, each paid for by an earlier comparison: + +* Battery saves (nr-kds, nr-nuf, nr-7v3). Both emulators load a save by ROM file name, and + Mesen2 keeps its saves in ``/Saves``, not beside the ROM. Every invocation of + either emulator runs on its own copy of the ROM, named uniquely per invocation, and Mesen2's + ``Saves``/``RecentGames``/``SaveStates`` entries for that copy are removed afterwards. +* The game database (nr-nwy, nr-sjt, nr-1le). Mesen2 silently replaces the header with its + database entry, and misses the database for files with trailing data. Its decision is + printed; ``--no-game-database`` makes it use the header (NES only). +* Controller ports (nr-0an). Mesen2 otherwise takes them from its ``settings.json``; both + ports are pinned to a standard pad, and NESER runs with an empty ``--config`` so a local + ``neser.conf`` cannot change its ports or palette. +* Region (nr-f6o). NESER's ``Hardware:`` line, printed with every comparison, names the region + it picked. + +Mesen2 needs ``"AllowIoOsAccess": true`` in its ``settings.json`` (see README.md); without it +the capture script prints one ``ERROR:`` line, which this command passes on. +""" + +import argparse +import os +import re +import shutil +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +from scripts.diff_screenshots import diff_pixels, load_screenshot + +REPO = Path(__file__).resolve().parent.parent.parent +CAPTURE_SCRIPT = Path(__file__).resolve().parent / "mesen2_capture.lua" +DEFAULT_MESEN2 = "/Applications/Mesen.app/Contents/MacOS/Mesen" +DEFAULT_NESER = str(REPO / "target" / "release" / "neser") +DEFAULT_MESEN2_HOME = str(Path.home() / "Library" / "Application Support" / "Mesen2") + +COMMON_MESEN2_FLAGS = [ + "--testRunner", + "--enableStdout", + "--timeout=30", + "--Video.VideoFilter=None", + "--Video.AspectRatio=NoStretching", +] +# Frame skipping off is mandatory for animated content (testRunner otherwise renders every +# other frame); zero RAM matches NESER's --headless; a standard pad in each port matches +# NESER's defaults (nr-0an). +NES_MESEN2_FLAGS = [ + "--nes.DisableFrameSkipping=true", + "--nes.RamPowerOnState=AllZeros", + "--nes.port1.type=NesController", + "--nes.port2.type=NesController", +] +SNES_MESEN2_FLAGS = [ + "--snes.disableFrameSkipping=true", + "--snes.RamPowerOnState=AllZeros", + "--snes.port1.type=SnesController", + "--snes.port2.type=SnesController", +] +NO_GAME_DATABASE_FLAG = "--nes.DisableGameDatabase=true" +# Mesen2's colours; without them NES whites differ. +NESER_FLAGS = {"nes": ["--nes-palette", "mesen"], "snes": []} +SYSTEMS = {".nes": "nes", ".sfc": "snes", ".smc": "snes"} +# Mesen2 keeps per-game files under its home, named after the ROM file. +MESEN2_GAME_DIRS = ("Saves", "RecentGames", "SaveStates") + + +class CaptureFailed(Exception): + """One emulator produced no screenshot; the message says what it printed.""" + + +def system_for(rom: Path) -> str: + try: + return SYSTEMS[rom.suffix.lower()] + except KeyError: + raise ValueError(f"{rom.name}: only .nes, .sfc and .smc ROMs are compared against Mesen2") from None + + +def mesen2_flags(system: str, no_game_database: bool) -> list[str]: + if system == "snes": + return list(SNES_MESEN2_FLAGS) + return [*NES_MESEN2_FLAGS, NO_GAME_DATABASE_FLAG] if no_game_database else list(NES_MESEN2_FLAGS) + + +def merge_flags(pinned: list[str], extra: list[str]) -> list[str]: + """``pinned`` with ``extra`` appended; an extra ``--key=value`` replaces the pinned flag with that key. + + Keys compare without case: the pinned flags themselves mix it (``--nes.DisableFrameSkipping``, + ``--snes.disableFrameSkipping``), and Mesen2 accepts either. + """ + keys = {flag.split("=", 1)[0].lower() for flag in extra if "=" in flag} + return [*(flag for flag in pinned if flag.split("=", 1)[0].lower() not in keys), *extra] + + +def fresh_copy(rom: Path, directory: Path) -> Path: + """Copy ``rom`` into ``directory`` under a name no other invocation uses.""" + copy = directory / f"{rom.stem}-{os.getpid()}-{time.time_ns()}{rom.suffix}" + shutil.copyfile(rom, copy) + return copy + + +def remove_mesen2_leftovers(mesen2_home: Path, copy: Path) -> None: + """Delete what Mesen2 stored under its home for this copy's name, and nothing else.""" + for sub in MESEN2_GAME_DIRS: + directory = mesen2_home / sub + if not directory.is_dir(): + continue + for path in directory.iterdir(): + if path.stem == copy.stem or path.name.startswith(copy.stem + "_"): + path.unlink() + + +def mesen2_cartridge_lines(stdout: str) -> list[str]: + """Mesen2's account of the ROM: header and database lines (NES), header block (SNES).""" + lines = stdout.splitlines() + separators = [i for i, line in enumerate(lines) if line.startswith("---")] + if len(separators) >= 2 and any(line.startswith("Game: ") for line in lines[separators[0] : separators[1]]): + block = lines[separators[0] + 1 : separators[1]] + return [line for line in block if not line.startswith("File: ")] + return [ + line + for line in lines + if line.startswith("PRG+CHR CRC32") + or (line.startswith("[DB]") and not line.startswith("[DB] Initialized")) + or line.startswith("[iNes]") + or line.startswith("[NES") + ] + + +def neser_cartridge_lines(output: str) -> list[str]: + """NESER's account of the ROM: the mapper it loaded and the hardware (region) it picked.""" + return [line for line in output.splitlines() if line.startswith(("Loaded rom with", "Hardware:"))] + + +def testrunner_pattern(mesen2: Path) -> str: + """The ``pgrep -f`` pattern for any testRunner of this binary, however it was started.""" + return f"{re.escape(mesen2.name)} --testRunner" + + +def wait_for_other_mesen2(mesen2: Path) -> None: + """A concurrent testRunner makes Mesen2 exit 0 with no output (snes-hardware-research).""" + deadline = time.monotonic() + 300 + pattern = testrunner_pattern(mesen2) + while subprocess.run(["pgrep", "-f", pattern], capture_output=True).returncode == 0: + if time.monotonic() > deadline: + raise TimeoutError(f"another {pattern} is still running") + time.sleep(2) + + +def capture_mesen2( + rom: Path, frame: int, out: Path, mesen2: Path, mesen2_home: Path, flags: list[str], work: Path +) -> str: + """Run Mesen2 once on a fresh copy of ``rom``, saving frame ``frame`` to ``out``; return stdout.""" + out.unlink(missing_ok=True) # a rerun into the same --out-dir must not diff the last run's file + copy = fresh_copy(rom, work) + env = {**os.environ, "CAPTURE_FRAME": str(frame), "CAPTURE_OUT": str(out.resolve())} + try: + if shutil.which("pgrep"): + wait_for_other_mesen2(mesen2) + cmd = [str(mesen2), *flags, str(copy), str(CAPTURE_SCRIPT)] + run = subprocess.run(cmd, capture_output=True, text=True, env=env, timeout=120) + finally: + remove_mesen2_leftovers(mesen2_home, copy) + copy.unlink(missing_ok=True) + if not out.is_file(): + errors = [line for line in run.stdout.splitlines() if line.startswith("ERROR")] + raise CaptureFailed("Mesen2 saved no screenshot: " + ("\n".join(errors) or run.stdout[-2000:])) + return run.stdout + + +def capture_neser(rom: Path, frame: int, out: Path, neser: Path, flags: list[str], work: Path) -> str: + """Run NESER once on a fresh copy of ``rom`` with an empty config; return its output.""" + out.unlink(missing_ok=True) + copy = fresh_copy(rom, work) + config = work / f"empty-{copy.stem}.conf" + config.write_text("") + try: + cmd = [str(neser), "--config", str(config), "--headless", "--frames", str(frame)] + cmd += ["--output", str(out), *flags, str(copy)] + run = subprocess.run(cmd, capture_output=True, text=True, timeout=600) + finally: + copy.unlink(missing_ok=True) + copy.with_suffix(".sav").unlink(missing_ok=True) + config.unlink(missing_ok=True) + if run.returncode != 0 or not out.is_file(): + raise CaptureFailed(f"NESER saved no screenshot (exit {run.returncode}): {(run.stdout + run.stderr)[-2000:]}") + return run.stdout + run.stderr + + +def compare(rom: Path, frames: list[int], out_dir: Path, args: argparse.Namespace) -> int: + system = system_for(rom) + flags_m = merge_flags([*COMMON_MESEN2_FLAGS, *mesen2_flags(system, args.no_game_database)], args.mesen2_arg) + flags_n = [*NESER_FLAGS[system], *args.neser_arg] + print(f"ROM: {rom}") + print(f"Output: {out_dir}") + worst = 0 + with tempfile.TemporaryDirectory(prefix="compare-mesen2-") as tmp: + work = Path(tmp) + for index, frame in enumerate(frames): + mesen2_png, neser_png = out_dir / f"mesen2-{frame}.png", out_dir / f"neser-{frame}.png" + try: + m_out = capture_mesen2(rom, frame, mesen2_png, args.mesen2_bin, args.mesen2_home, flags_m, work) + n_out = capture_neser(rom, frame, neser_png, args.neser_bin, flags_n, work) + except (CaptureFailed, OSError, subprocess.TimeoutExpired, TimeoutError) as failure: + # Exit 1 means "a frame differs"; a run that never captured must not read as one. + print(f"frame {frame}: capture failed: {failure}") + return 2 + if index == 0: + m_lines = mesen2_cartridge_lines(m_out) or ["(no cartridge lines printed)"] + n_lines = neser_cartridge_lines(n_out) or ["(no cartridge lines printed)"] + for line in m_lines: + print(f"Mesen2: {line}") + for line in n_lines: + print(f"NESER: {line}") + neser_shot, mesen2_shot = load_screenshot(neser_png), load_screenshot(mesen2_png) + if (neser_shot.width, neser_shot.height) != (mesen2_shot.width, mesen2_shot.height): + sizes = f"NESER {neser_shot.width}x{neser_shot.height}, Mesen2 {mesen2_shot.width}x{mesen2_shot.height}" + print(f"frame {frame}: sizes differ ({sizes})") + worst = 1 + continue + differing = diff_pixels(neser_shot, mesen2_shot) + print(f"frame {frame}: {differing} differing pixels") + worst = max(worst, 1 if differing else 0) + return worst + + +def parse_args(argv: list[str] | None) -> argparse.Namespace: + parser = argparse.ArgumentParser( + prog="python -m scripts.reference_capture.compare_mesen2", + description="Compare a NES or SNES ROM between NESER and Mesen2 at given frames.", + ) + parser.add_argument("rom", type=Path) + parser.add_argument("--frames", type=int, nargs="+", required=True, help="frames to compare, from power-on") + parser.add_argument("--out-dir", type=Path, help="where the PNGs go (default: a new temporary directory)") + parser.add_argument( + "--no-game-database", + action="store_true", + help="make Mesen2 use the iNES header instead of its game database (NES)", + ) + parser.add_argument("--neser-arg", action="append", default=[], help="extra NESER argument (repeatable)") + parser.add_argument("--mesen2-arg", action="append", default=[], help="extra Mesen2 argument (repeatable)") + parser.add_argument("--neser-bin", type=Path, default=Path(os.environ.get("NESER_BIN", DEFAULT_NESER))) + parser.add_argument("--mesen2-bin", type=Path, default=Path(os.environ.get("MESEN2_BIN", DEFAULT_MESEN2))) + parser.add_argument("--mesen2-home", type=Path, default=Path(os.environ.get("MESEN2_HOME", DEFAULT_MESEN2_HOME))) + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + try: + system_for(args.rom) + except ValueError as error: + print(error) + return 2 + out_dir = args.out_dir or Path(tempfile.mkdtemp(prefix="mesen2-comparison-")) + out_dir.mkdir(parents=True, exist_ok=True) + return compare(args.rom, args.frames, out_dir, args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/test_compare_mesen2.py b/scripts/test_compare_mesen2.py new file mode 100644 index 000000000..d562c3707 --- /dev/null +++ b/scripts/test_compare_mesen2.py @@ -0,0 +1,366 @@ +"""Unit tests for scripts/reference_capture/compare_mesen2.py (nr-ocx). + +The command owns the Mesen2 comparison recipe, so these tests pin the confounds earlier +comparisons paid for: battery saves leaking between Mesen2 runs of one ROM name (nr-nuf, +nr-7v3), Mesen2's game database silently overriding the header (nr-nwy, nr-sjt), controller +ports taken from Mesen2's local settings (nr-0an) and the region NESER picked going unrecorded +(nr-f6o). The end-to-end cases run the command against fake ``Mesen`` and ``neser`` +executables, so no emulator is needed. +""" + +import contextlib +import io +import os +import stat +import sys +import tempfile +import textwrap +import unittest +from pathlib import Path + +from scripts.reference_capture.compare_mesen2 import ( + NES_MESEN2_FLAGS, + SNES_MESEN2_FLAGS, + fresh_copy, + main, + merge_flags, + mesen2_cartridge_lines, + mesen2_flags, + neser_cartridge_lines, + remove_mesen2_leftovers, + system_for, + testrunner_pattern, +) + +# Real Mesen2 2.1.1 --testRunner --enableStdout output, captured 2026-10-04. +MESEN2_NES_STDOUT = """------------------------------------------------------ +Loading rom: demo_ntsc.nes +File CRC32: 0xA2347476 +------------------------------------------------------ +[DB] Initialized - 10655 games in DB +PRG CRC32: 0xD76717F5 +PRG+CHR CRC32: 0x5CE951EA +[iNes] Mapper: 0 Sub: 0 +[iNes] PRG ROM: 32 KB +[iNes] CHR ROM: 8 KB +[iNes] Mirroring: Horizontal +[iNes] Battery: No +[DB] Game not found in database +[CPU] Uninitialized memory read: $0300 +SAVED /tmp/out.png +""" + +MESEN2_SNES_STDOUT = """----------------------------- +File: window-precalculated-single.sfc +Game: PRECALCULATED WINDOW +Type: LoROM +FastROM +Map Mode: $30 +Rom Type: $00 +File size: 128 KB +ROM size: 128 KB +----------------------------- +SAVED /tmp/out.png +""" + +# Real NESER --headless output, captured 2026-10-04. +NESER_NES_OUTPUT = """Loaded rom with CRC32: 5CE951EA, mapper=0, submapper=0, PRG-ROM=32KB, CHR-ROM=8KB +Hardware: NES (NTSC) | Port 1: Joypad | Port 2: Joypad +Saved screenshot to out.png +""" + + +class TestMesen2Flags(unittest.TestCase): + def test_nes_flags_pin_ports_and_zero_ram(self) -> None: + for flag in ( + "--nes.DisableFrameSkipping=true", + "--nes.RamPowerOnState=AllZeros", + "--nes.port1.type=NesController", + "--nes.port2.type=NesController", + ): + self.assertIn(flag, NES_MESEN2_FLAGS) + + def test_snes_flags_pin_standard_pads(self) -> None: + for flag in ( + "--snes.disableFrameSkipping=true", + "--snes.RamPowerOnState=AllZeros", + "--snes.port1.type=SnesController", + "--snes.port2.type=SnesController", + ): + self.assertIn(flag, SNES_MESEN2_FLAGS) + + def test_flags_follow_the_system(self) -> None: + self.assertEqual(mesen2_flags("nes", no_game_database=False), NES_MESEN2_FLAGS) + self.assertEqual(mesen2_flags("snes", no_game_database=False), SNES_MESEN2_FLAGS) + + def test_no_game_database_only_for_nes(self) -> None: + self.assertIn("--nes.DisableGameDatabase=true", mesen2_flags("nes", no_game_database=True)) + self.assertNotIn("--nes.DisableGameDatabase=true", mesen2_flags("nes", no_game_database=False)) + self.assertEqual(mesen2_flags("snes", no_game_database=True), SNES_MESEN2_FLAGS) + + def test_an_extra_flag_replaces_the_pinned_flag_with_its_key(self) -> None: + merged = merge_flags(SNES_MESEN2_FLAGS, ["--snes.port2.type=SnesMouse", "--snes.Overclock=1"]) + self.assertIn("--snes.port2.type=SnesMouse", merged) + self.assertNotIn("--snes.port2.type=SnesController", merged) + self.assertIn("--snes.port1.type=SnesController", merged) + self.assertIn("--snes.Overclock=1", merged) + + def test_extra_flag_keys_match_regardless_of_case(self) -> None: + merged = merge_flags(SNES_MESEN2_FLAGS, ["--snes.Port2.Type=SnesMouse"]) + self.assertNotIn("--snes.port2.type=SnesController", merged) + + def test_concurrent_testrunner_is_matched_by_basename(self) -> None: + # Another session may start Mesen2 through PATH or a symlink (nr-ocx review). + self.assertEqual(testrunner_pattern(Path("/Applications/Mesen.app/Contents/MacOS/Mesen")), "Mesen --testRunner") + + def test_system_for_extension(self) -> None: + self.assertEqual(system_for(Path("a/game.nes")), "nes") + self.assertEqual(system_for(Path("a/GAME.NES")), "nes") + self.assertEqual(system_for(Path("a/game.sfc")), "snes") + self.assertEqual(system_for(Path("a/game.smc")), "snes") + with self.assertRaises(ValueError): + system_for(Path("a/game.gb")) + + +class TestFreshCopies(unittest.TestCase): + def test_each_copy_has_its_own_name(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + rom = Path(tmp) / "zelda.nes" + rom.write_bytes(b"NES\x1a rom") + first, second = fresh_copy(rom, Path(tmp)), fresh_copy(rom, Path(tmp)) + self.assertNotEqual(first.name, second.name) + for copy in (first, second): + self.assertEqual(copy.suffix, ".nes") + self.assertEqual(copy.read_bytes(), rom.read_bytes()) + + def test_leftovers_removed_only_for_that_copy(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + home = Path(tmp) + copy = home / "zelda-1-2.nes" + for sub, name in ( + ("Saves", "zelda-1-2.sav"), + ("Saves", "zelda-1-2.srm"), + ("RecentGames", "zelda-1-2.rgd"), + ("SaveStates", "zelda-1-2_1.mss"), + ("Saves", "zelda.sav"), + ("Saves", "zelda-1-23.sav"), + ): + (home / sub).mkdir(exist_ok=True) + (home / sub / name).write_bytes(b"x") + remove_mesen2_leftovers(home, copy) + left = sorted(p.name for p in home.rglob("*") if p.is_file()) + self.assertEqual(left, ["zelda-1-23.sav", "zelda.sav"]) + + def test_leftovers_tolerates_missing_directories(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + remove_mesen2_leftovers(Path(tmp), Path(tmp) / "x-1-2.nes") + + +class TestCartridgeLines(unittest.TestCase): + def test_mesen2_nes_lines(self) -> None: + self.assertEqual( + mesen2_cartridge_lines(MESEN2_NES_STDOUT), + [ + "PRG+CHR CRC32: 0x5CE951EA", + "[iNes] Mapper: 0 Sub: 0", + "[iNes] PRG ROM: 32 KB", + "[iNes] CHR ROM: 8 KB", + "[iNes] Mirroring: Horizontal", + "[iNes] Battery: No", + "[DB] Game not found in database", + ], + ) + + def test_mesen2_snes_header_block(self) -> None: + lines = mesen2_cartridge_lines(MESEN2_SNES_STDOUT) + self.assertEqual(lines[0], "Game: PRECALCULATED WINDOW") + self.assertIn("Type: LoROM", lines) + self.assertNotIn("File: window-precalculated-single.sfc", lines) + self.assertFalse(any(line.startswith("---") or line.startswith("SAVED") for line in lines)) + + def test_neser_lines(self) -> None: + self.assertEqual( + neser_cartridge_lines(NESER_NES_OUTPUT), + [ + "Loaded rom with CRC32: 5CE951EA, mapper=0, submapper=0, PRG-ROM=32KB, CHR-ROM=8KB", + "Hardware: NES (NTSC) | Port 1: Joypad | Port 2: Joypad", + ], + ) + + +# A fake Mesen2: logs its arguments, writes a battery save named after the ROM into the +# fake Mesen2 home (as the real one does on exit), prints its cartridge lines and writes a +# 2x1 PNG to CAPTURE_OUT. +FAKE_MESEN2 = textwrap.dedent( + """\ + #!{python} + import os, sys + from pathlib import Path + from PIL import Image + log, home = Path(os.environ["FAKE_LOG"]), Path(os.environ["FAKE_MESEN2_HOME"]) + rom = Path(next(a for a in sys.argv[1:] if not a.startswith("--") and not a.endswith(".lua"))) + with log.open("a") as f: + f.write("mesen2 " + " ".join(sys.argv[1:]) + "\\n") + f.write("saves-before " + " ".join(sorted(p.name for p in (home / "Saves").glob("*"))) + "\\n") + if os.environ.get("FAKE_LUA_ERROR"): + print('ERROR: Lua file access is off; set "AllowIoOsAccess": true') + sys.exit(1) + (home / "Saves").mkdir(exist_ok=True) + (home / "Saves" / (rom.stem + ".sav")).write_bytes(b"sram") + print("[DB] Initialized - 1 games in DB") + print("[iNes] Mapper: 4 Sub: 0") + print("[DB] Game found in database") + print("[DB] Mapper: 118 Sub: 0") + Image.new("RGB", (2, 1), (1, 2, 3)).save(os.environ["CAPTURE_OUT"]) + print("SAVED " + os.environ["CAPTURE_OUT"]) + """ +) + +# A fake NESER: logs its arguments, prints its cartridge lines and writes a 2x1 PNG whose +# second pixel differs when FAKE_NESER_DIFFERS is set. +FAKE_NESER = textwrap.dedent( + """\ + #!{python} + import os, sys + from PIL import Image + args = sys.argv[1:] + with open(os.environ["FAKE_LOG"], "a") as f: + f.write("neser " + " ".join(args) + "\\n") + print("Loaded rom with CRC32: 12345678, mapper=4, submapper=0, PRG-ROM=128KB, CHR-ROM=128KB") + print("Hardware: NES (PAL) | Port 1: Joypad | Port 2: Joypad") + img = Image.new("RGB", (2, 1), (1, 2, 3)) + if os.environ.get("FAKE_NESER_DIFFERS"): + img.putpixel((1, 0), (9, 9, 9)) + img.save(args[args.index("--output") + 1]) + """ +) + + +class TestCompareMesen2EndToEnd(unittest.TestCase): + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + root = Path(self.tmp.name) + self.home = root / "mesen2-home" + (self.home / "Saves").mkdir(parents=True) + (self.home / "Saves" / "other-game.sav").write_bytes(b"keep") + self.log = root / "log.txt" + self.out = root / "out" + self.rom = root / "roms" / "kirby.nes" + self.rom.parent.mkdir() + self.rom.write_bytes(b"NES\x1a") + self.mesen2 = self._executable(root / "Mesen", FAKE_MESEN2) + self.neser = self._executable(root / "neser", FAKE_NESER) + self.env = {"FAKE_LOG": str(self.log), "FAKE_MESEN2_HOME": str(self.home)} + + def _executable(self, path: Path, source: str) -> Path: + path.write_text(source.replace("{python}", sys.executable)) + path.chmod(path.stat().st_mode | stat.S_IXUSR) + return path + + def run_main(self, *extra: str, env: dict[str, str] | None = None) -> tuple[int, str]: + argv = [str(self.rom), "--frames", "60", "120", "--out-dir", str(self.out)] + argv += ["--mesen2-bin", str(self.mesen2), "--neser-bin", str(self.neser)] + argv += ["--mesen2-home", str(self.home), *extra] + saved = dict(os.environ) + os.environ.update({**self.env, **(env or {})}) + stdout = io.StringIO() + try: + with contextlib.redirect_stdout(stdout): + code = main(argv) + finally: + os.environ.clear() + os.environ.update(saved) + return code, stdout.getvalue() + + def logged(self, program: str) -> list[str]: + return [line for line in self.log.read_text().splitlines() if line.startswith(program + " ")] + + def test_each_mesen2_invocation_gets_its_own_rom_and_save_is_removed(self) -> None: + self.run_main() + runs = self.logged("mesen2") + self.assertEqual(len(runs), 2) + roms = [next(a for a in run.split()[1:] if a.endswith(".nes")) for run in runs] + self.assertEqual(len({Path(r).name for r in roms}), 2, roms) + self.assertTrue(all(Path(r).name != "kirby.nes" for r in roms), roms) + # The second run started on no save from the first, and only the user's own save is left. + befores = [line for line in self.log.read_text().splitlines() if line.startswith("saves-before")] + self.assertEqual(befores, ["saves-before other-game.sav"] * 2) + self.assertEqual(sorted(p.name for p in (self.home / "Saves").iterdir()), ["other-game.sav"]) + + def test_neser_runs_on_its_own_copy_with_an_empty_config(self) -> None: + self.run_main() + runs = self.logged("neser") + self.assertEqual(len(runs), 2) + for run in runs: + args = run.split()[1:] + self.assertIn("--headless", args) + self.assertEqual(args[args.index("--nes-palette") + 1], "mesen") + config = Path(args[args.index("--config") + 1]) + self.assertNotEqual(Path(args[-1]).name, "kirby.nes") + self.assertEqual(config.read_text() if config.exists() else "", "") + + def test_mesen2_gets_the_system_flags_and_capture_script(self) -> None: + self.run_main("--no-game-database") + args = self.logged("mesen2")[0].split()[1:] + for flag in [*NES_MESEN2_FLAGS, "--testRunner", "--enableStdout", "--nes.DisableGameDatabase=true"]: + self.assertIn(flag, args) + self.assertTrue(args[-1].endswith("mesen2_capture.lua")) + + def test_report_prints_both_emulators_decisions(self) -> None: + _, out = self.run_main() + self.assertIn("Mesen2: [iNes] Mapper: 4 Sub: 0", out) + self.assertIn("Mesen2: [DB] Mapper: 118 Sub: 0", out) + self.assertIn("NESER: Loaded rom with CRC32: 12345678, mapper=4", out) + self.assertIn("NESER: Hardware: NES (PAL)", out) + self.assertNotIn("[DB] Initialized", out) + # The cartridge report is printed once, not once per frame. + self.assertEqual(out.count("[iNes] Mapper"), 1) + + def test_identical_frames_exit_zero_and_keep_the_pngs(self) -> None: + code, out = self.run_main() + self.assertEqual(code, 0) + self.assertIn("frame 60: 0 differing pixels", out) + self.assertIn("frame 120: 0 differing pixels", out) + for name in ("neser-60.png", "mesen2-60.png", "neser-120.png", "mesen2-120.png"): + self.assertTrue((self.out / name).is_file(), name) + + def test_differing_frames_exit_one(self) -> None: + code, out = self.run_main(env={"FAKE_NESER_DIFFERS": "1"}) + self.assertEqual(code, 1) + self.assertIn("frame 60: 1 differing pixels", out) + + def test_lua_error_is_reported(self) -> None: + code, out = self.run_main(env={"FAKE_LUA_ERROR": "1"}) + self.assertEqual(code, 2) + self.assertIn("AllowIoOsAccess", out) + + def test_failed_capture_into_a_reused_out_dir_is_not_a_match(self) -> None: + # A rerun must not diff the previous run's PNG (nr-ocx review). + self.assertEqual(self.run_main()[0], 0) + code, out = self.run_main(env={"FAKE_LUA_ERROR": "1"}) + self.assertEqual(code, 2) + self.assertNotIn("differing pixels", out) + + def test_missing_binary_is_a_capture_failure(self) -> None: + for flag in ("--mesen2-bin", "--neser-bin"): + with self.subTest(flag=flag): + code, out = self.run_main(flag, str(Path(self.tmp.name) / "nonexistent")) + self.assertEqual(code, 2) + self.assertNotIn("differing pixels", out) + + def test_extra_mesen2_arg_replaces_a_common_flag(self) -> None: + self.run_main("--mesen2-arg=--timeout=60") + args = self.logged("mesen2")[0].split()[1:] + self.assertIn("--timeout=60", args) + self.assertNotIn("--timeout=30", args) + + def test_extra_mesen2_arg_replaces_a_pinned_port(self) -> None: + self.run_main("--mesen2-arg=--nes.port2.type=Zapper") + args = self.logged("mesen2")[0].split()[1:] + self.assertIn("--nes.port2.type=Zapper", args) + self.assertNotIn("--nes.port2.type=NesController", args) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_mesen2_capture.py b/scripts/test_mesen2_capture.py index 1eae6a52e..99a1414ad 100644 --- a/scripts/test_mesen2_capture.py +++ b/scripts/test_mesen2_capture.py @@ -34,6 +34,12 @@ from pathlib import Path from scripts.diff_screenshots import Screenshot, diff_pixels, load_screenshot +from scripts.reference_capture.compare_mesen2 import ( + COMMON_MESEN2_FLAGS, + NES_MESEN2_FLAGS, + SNES_MESEN2_FLAGS, + wait_for_other_mesen2, +) REPO = Path(__file__).resolve().parent.parent SCRIPT = REPO / "scripts" / "reference_capture" / "mesen2_capture.lua" @@ -57,26 +63,19 @@ # (rom, NESER flags, Mesen2 flags, frames), each frame one where the content differs # from both neighbouring frames. -NES_FLAGS = ["--nes.DisableFrameSkipping=true", "--nes.RamPowerOnState=AllZeros"] -SNES_FLAGS = [ - "--snes.disableFrameSkipping=true", - "--snes.RamPowerOnState=AllZeros", - "--snes.port1.type=SnesController", - "--snes.port2.type=SnesController", -] CASES = [ ( # Alternates two images every frame, so it pins the parity of the frame; no NES # ROM in roms/ changes monotonically and also matches Mesen2 at 0 px. "roms/nes/automated_tests/nmi_sync/demo_ntsc.nes", ["--nes-palette", "mesen"], - NES_FLAGS, + NES_MESEN2_FLAGS, (61, 64, 67, 120, 300), ), ( "roms/snes/automated_tests/snes_test_roms/undisbeliever-ppu-window/window-precalculated-single.sfc", [], - SNES_FLAGS, + SNES_MESEN2_FLAGS, (61, 72, 73, 120, 300), ), ] @@ -89,21 +88,11 @@ def neser_frame(rom: Path, flags: list[str], frame: int, out_dir: Path) -> Scree return load_screenshot(out) -def wait_for_other_mesen2() -> None: - """A concurrent testRunner makes Mesen2 exit 0 with no output (snes-hardware-research).""" - deadline = time.monotonic() + 300 - while subprocess.run(["pgrep", "-f", "Mesen --testRunner"], capture_output=True).returncode == 0: - if time.monotonic() > deadline: - raise TimeoutError("another Mesen --testRunner is still running") - time.sleep(2) - - def mesen2_capture(rom: Path, flags: list[str], frame: int, out_dir: Path) -> Screenshot: script = out_dir / "capture.lua" script.write_text(SHIM.replace("__FRAME__", str(frame)) + SCRIPT.read_text()) - wait_for_other_mesen2() - cmd = [str(MESEN2), "--testRunner", "--enableStdout", "--timeout=30"] - cmd += ["--Video.VideoFilter=None", "--Video.AspectRatio=NoStretching", *flags] + wait_for_other_mesen2(MESEN2) + cmd = [str(MESEN2), *COMMON_MESEN2_FLAGS, *flags] run = subprocess.run([*cmd, str(rom), str(script)], capture_output=True, text=True, timeout=120) hex_lines = [line for line in run.stdout.splitlines() if line.startswith("PNG_HEX ")] if len(hex_lines) != 1: @@ -143,12 +132,13 @@ class TestMesen2SnesFlags(unittest.TestCase): attract demo differently for an empty port 2, so a comparison that leaves the ports to the settings file differs by a lag frame at frame 3000 with no emulation difference. NESER's defaults are a standard pad in each port (``SnesConfig::default``, pinned by - ``controller_ports_default_to_standard``). + ``controller_ports_default_to_standard``). The flags live in + ``scripts/reference_capture/compare_mesen2.py``, the comparison command (nr-ocx). """ def test_both_ports_hold_a_standard_controller(self) -> None: - self.assertIn("--snes.port1.type=SnesController", SNES_FLAGS) - self.assertIn("--snes.port2.type=SnesController", SNES_FLAGS) + self.assertIn("--snes.port1.type=SnesController", SNES_MESEN2_FLAGS) + self.assertIn("--snes.port2.type=SnesController", SNES_MESEN2_FLAGS) # Mesen2 with "AllowIoOsAccess": false leaves the globals io and os nil (measured on 2.1.1, @@ -164,7 +154,7 @@ def test_stops_at_load_naming_the_setting(self) -> None: with tempfile.TemporaryDirectory() as tmp: script = Path(tmp) / "capture.lua" script.write_text(NO_FILE_ACCESS_SHIM + SCRIPT.read_text()) - wait_for_other_mesen2() + wait_for_other_mesen2(MESEN2) cmd = [str(MESEN2), "--testRunner", "--enableStdout", "--timeout=30", str(rom), str(script)] start = time.monotonic() run = subprocess.run(cmd, capture_output=True, text=True, timeout=120) diff --git a/scripts/test_mesen2_traces.py b/scripts/test_mesen2_traces.py index e817c3d74..01fa1242d 100644 --- a/scripts/test_mesen2_traces.py +++ b/scripts/test_mesen2_traces.py @@ -22,7 +22,8 @@ from pathlib import Path from scripts.diff_timing_traces import diff_traces, format_report, parse_lines -from scripts.test_mesen2_capture import MESEN2, NES_FLAGS, NO_FILE_ACCESS_SHIM, SNES_FLAGS, wait_for_other_mesen2 +from scripts.reference_capture.compare_mesen2 import NES_MESEN2_FLAGS, SNES_MESEN2_FLAGS, wait_for_other_mesen2 +from scripts.test_mesen2_capture import MESEN2, NO_FILE_ACCESS_SHIM REPO = Path(__file__).resolve().parent.parent LUA = REPO / "scripts" / "reference_capture" @@ -42,7 +43,7 @@ NES_ROM = "roms/nes/automated_tests/nmi_sync/demo_ntsc.nes" SNES_ROM = "roms/snes/automated_tests/snes_test_roms/undisbeliever-ppu-window/window-precalculated-single.sfc" -ROMS = [(NES_ROM, NES_FLAGS), (SNES_ROM, SNES_FLAGS)] +ROMS = [(NES_ROM, NES_MESEN2_FLAGS), (SNES_ROM, SNES_MESEN2_FLAGS)] def mesen2_trace(script: str, rom: Path, flags: list[str], env: dict[str, str], tmp: Path) -> list[str]: @@ -50,7 +51,7 @@ def mesen2_trace(script: str, rom: Path, flags: list[str], env: dict[str, str], lua = tmp / script entries = ", ".join(f'{key} = "{value}"' for key, value in values.items()) lua.write_text(SHIM.replace("__ENV__", entries) + (LUA / script).read_text()) - wait_for_other_mesen2() + wait_for_other_mesen2(MESEN2) cmd = [str(MESEN2), "--testRunner", "--enableStdout", "--timeout=60", *flags, str(rom), str(lua)] run = subprocess.run(cmd, capture_output=True, text=True, timeout=180) lines = [line.removeprefix("TRACE ") for line in run.stdout.splitlines() if line.startswith("TRACE ")] @@ -107,7 +108,7 @@ def test_each_script_stops_at_load_naming_the_setting(self) -> None: with self.subTest(script=script), tempfile.TemporaryDirectory() as tmp: lua = Path(tmp) / script lua.write_text(NO_FILE_ACCESS_SHIM + (LUA / script).read_text()) - wait_for_other_mesen2() + wait_for_other_mesen2(MESEN2) cmd = [str(MESEN2), "--testRunner", "--enableStdout", "--timeout=30", str(REPO / NES_ROM), str(lua)] start = time.monotonic() run = subprocess.run(cmd, capture_output=True, text=True, timeout=120)