Step c

Burnout3_recomp/.gitignore

@@ -0,0 +1,42 @@
+# Burnout 3 (SLUS_210.50) workspace: keep game files and game-derived data out of git.
+# Tracked here: TOML configs (repo-relative paths), tools/, COMPLETED_WORK.md, this file.
+
+# Game executables and disc images (also renamed copies such as SLUS_210.50.bak)
+/SLUS_*
+*.elf
+*.irx
+*.iso
+*.bin
+*.img
+*.cue
+*.chd
+*.cso
+
+# Extracted disc contents and assets
+/disc*/
+/assets/
+/game_data/
+
+# Ghidra project database and lock files
+/ghidra_project/
+*.gpr
+*.rep/
+*.lock
+*.lock~
+
+# Logs
+/logs/
+*.log
+
+# Generated C++ from every recompiler run (output/, output_bounded/, ...)
+/output*/
+
+# Earlier export backups
+/backups/
+
+# Function maps, instruction ranges and reports derived from the ELF
+/*.csv
+/reports/
+
+# Machine-specific values: ELF SHA-256, original ELF location
+/local/

Burnout3_recomp/COMPLETED_WORK.md

@@ -0,0 +1,142 @@
+# Burnout 3 USA (SLUS_210.50): recompilation workspace
+
+All paths below are relative to the PS2Recomp repo root. Run every command from there.
+
+## Git policy
+
+- Only the TOML configs, `tools/`, this file and `.gitignore` are tracked.
+- Never commit game files or anything derived from the game: the ELF, disc images,
+  IRX modules, extracted assets, the Ghidra project, function maps, reports, logs,
+  backups, or generated C++ (`output*/`). `Burnout3_recomp/.gitignore` enforces this.
+- Machine-specific values (ELF SHA-256, original ELF location) live in the ignored
+  `Burnout3_recomp/local/burnout3_local.json`; the tools read them from there.
+- Generated C++ is reproducible from the ELF, TOML, CSV and the recompiler commit,
+  so it is never tracked.
+
+## Workspace layout
+
+| Path | Tracked | Contents |
+|---|---|---|
+| `SLUS_210.50` | no | Working copy of the ELF, byte-identical to the original |
+| `ghidra_project/` | no | Ghidra 12.1.4 project `Burnout3_USA_Recomp` (r5900:LE:32:default) |
+| `burnout3_usa_ghidra_functions.csv` | no | Original exporter map (71,108 entries) |
+| `burnout3_usa_ghidra_functions_instruction_bounded.csv` | no | Repaired map (71,112 entries) |
+| `burnout3_usa_recomp.toml` | yes | Original exporter config, writes `output/` |
+| `burnout3_usa_recomp_instruction_bounded.toml` | yes | Repaired-map config, writes `output/` |
+| `burnout3_usa_recomp_instruction_bounded_output_bounded.toml` | yes | Same, writes `output_bounded/` |
+| `tools/` | yes | Ghidra scripts and the repair/verify scripts |
+| `reports/`, `logs/`, `backups/` | no | Analysis/validation reports, console logs, pre-repair exports |
+| `output/`, `output_bounded/` | no | Generated C++ |
+| `disc_image/` | no | Read-only mount point for your own Burnout 3 ISO (empty when detached) |
+| `disc/` | no | Runtime launch folder: relative symlinks into `disc_image/`, an APFS clone of `SLUS_210.50`, and a writable `mc0/` for saves |
+
+Mount before running the game, and detach afterwards. The runtime maps `cdrom0:` to the folder containing the launched ELF.
+
+```bash
+hdiutil attach -readonly -nobrowse -mountpoint "$PWD/Burnout3_recomp/disc_image" "<path to your Burnout 3 ISO>"
+```
+
+```bash
+hdiutil detach Burnout3_recomp/disc_image
+```
+
+## Completed steps
+
+1. **Ghidra import and analysis.** 7,776 functions, 880,944 instructions. Single
+   executable section `0x00100000–0x004E2680` (code and data mixed), entry `0x00100008`.
+   12 error bookmarks and 1 warning bookmark (see "Known issues").
+2. **Export.** The unchanged repository script `ps2xRecomp/tools/ghidra/ExportPS2Functions.java`
+   produced `burnout3_usa_recomp.toml` and `burnout3_usa_ghidra_functions.csv`
+   (7,776 functions + 63,332 code labels, 58 runtime stubs).
+3. **Baseline recompilation** with `burnout3_usa_recomp.toml` into `output/`.
+   It completed, but the original map has two defects:
+   - **Data treated as code.** 4,270 unique `Unhandled …` instructions (emitted as
+     `throw std::runtime_error`) in four files: `entry_004b9238`, `entry_004ca498`,
+     `entry_004dd7e0` and `entry_004e2118`. All lie in `0x4B9440–0x4E2664`, and none
+     are inside Ghidra's instruction ranges.
+   - **Noncontiguous function bodies.** 453 rows have `End - Start != Size`, because
+     the exporter sets End to the body's highest address. That produced 887 MB of
+     output, including `FUN_0014d410` at 139 MB.
+4. **Range repair.** `tools/ExportBurnout3InstructionRanges.java` exported Ghidra's
+   contiguous instruction runs to `reports/burnout3_usa_ghidra_instruction_ranges.csv`.
+   `tools/repair_burnout3_export_ranges.py` bounded every range to those runs. It kept
+   all 71,108 original entries, added 4 fragments, and excluded 193,076 non-instruction
+   bytes. `tools/verify_burnout3_exports.py` passes all 30 checks.
+5. **Repaired-map recompilation** into `output_bounded/`, logged to
+   `logs/08_ps2_recomp_instruction_bounded.log`. It ran in 4m50s with exit code 0.
+
+   | | Baseline `output/` | Repaired `output_bounded/` |
+   |---|---|---|
+   | Functions in map | 71,108 | 71,112 (all recompiled or stubbed, 0 decode failures) |
+   | Unhandled instructions | 4,270 unique | **0** |
+   | `// Unimplemented` register markers | 61 | 0 |
+   | Total size | 887 MB | 517 MB |
+   | Largest function file | 139 MB (`FUN_0014d410`) | 3.6 MB (`FUN_00186850`, a genuinely 77 KB function) |
+   | Errors | not captured | 0 |
+   | Warnings | not captured | 5,636, all `control-flow` |
+
+   Each warning is "unresolved JR/JALR; promoted N fallback entries". These are
+   indirect jumps, mostly `switch` jump tables (`caseD_*`, `FUN_*`, `entry_*`), that the
+   recompiler could not resolve statically. It registered 413,683 fallback entry points
+   instead, so these jumps go through the runtime dispatch table (`register_functions.cpp`
+   is 21.8 MB). They are not build blockers, but they are slower and are the next thing
+   to triage. Whether the 4 split fragments are always reached correctly can only be
+   confirmed at runtime.
+
+   `output_bounded/` is the output to integrate next. Keep `output/` only for comparison.
+
+## Known issues
+
+- Ghidra error bookmarks: 11 flows from `0x484414–0x485654` into `0x74000–0x76680`,
+  which is memory not in the ELF (probably code copied to low RAM at runtime; this
+  is inferred), and conflicting data at `0x200024`.
+- Ghidra decompiler `VarnodeContext` errors for `vf1`, `vf8` and `vf22` at `0x1700B0`,
+  `0x17C01C` and `0x18EF0C`. These affect the decompiler only, not `ps2_recomp`.
+
+## Commands
+
+```bash
+python3 Burnout3_recomp/tools/verify_burnout3_exports.py
+```
+
+```bash
+./out/build/ps2xRecomp/ps2_recomp Burnout3_recomp/burnout3_usa_recomp_instruction_bounded_output_bounded.toml > Burnout3_recomp/logs/08_ps2_recomp_instruction_bounded.log 2>&1
+```
+
+```bash
+sed -n '/PS2Recomp report/,/Events:/p' Burnout3_recomp/logs/08_ps2_recomp_instruction_bounded.log
+```
+
+## Not done yet
+
+- Triage the 5,636 control-flow warnings by resolving jump tables in Ghidra.
+- Build, link, then handle runtime work (syscalls, GS/VU1, IOP/IRX), using your own disc for assets.
+
+## Runtime integration (done)
+
+`output_bounded/` is integrated into `ps2xRuntime`:
+- **Code:** all 71,113 `.cpp` files, cloned with `cp -c` (no extra disk space), are in
+  `ps2xRuntime/src/runner/`.
+- **Headers:** `ps2_recompiled_functions.h` and `ps2_recompiled_stubs.h` are in
+  `ps2xRuntime/include/`.
+- **Git:** upstream's `.gitignore` already ignores both locations. The tracked placeholder
+  `ps2xRuntime/src/runner/register_functions.cpp` is marked `--skip-worktree`, so the
+  generated version does not show as modified.
+
+To undo (restores the upstream placeholder):
+
+```bash
+find ps2xRuntime/src/runner -name '*.cpp' ! -name register_functions.cpp -delete
+```
+
+```bash
+rm ps2xRuntime/include/ps2_recompiled_functions.h ps2xRuntime/include/ps2_recompiled_stubs.h
+```
+
+```bash
+git update-index --no-skip-worktree ps2xRuntime/src/runner/register_functions.cpp
+```
+
+```bash
+git checkout -- ps2xRuntime/src/runner/register_functions.cpp
+```

Burnout3_recomp/README.md

@@ -0,0 +1,102 @@
+# Burnout 3 USA (SLUS_210.50) recompilation workspace
+
+Workspace for statically recompiling Burnout 3 (USA) with PS2Recomp. Game files and
+game-derived data are never tracked (see `.gitignore`); use your own disc only.
+
+- `ROADMAP.md`: the overall plan, steps A–D.
+- `COMPLETED_WORK.md`: what has been done so far.
+
+All commands below run from the PS2Recomp repo root.
+
+## Step C: build `ps2EntryRunner`, in 4 milestones
+
+Step C turns the integrated generated code (step B) into one executable,
+`out/build/ps2xRuntime/ps2EntryRunner`.
+
+In unity mode, CMake joins the 71,113 generated files in `ps2xRuntime/src/runner/` into
+about 2,200 batches of 32. A precompiled header parses the 6 MB
+`ps2_recompiled_functions.h` once instead of 71,113 times. The linker then combines the
+batches with `libps2_runtime.a` and the dependencies already in `out/build/_deps`.
+
+**Done when:** the executable links and its log is saved. Booting the game is step D.
+
+| # | Milestone | Duration | Main risk |
+|---|---|---|---|
+| 1 | Pre-flight and reconfigure | minutes | Stale or missing runner files; an unexpected download |
+| 2 | Runtime library at the new flags | minutes | Toolchain or flag errors |
+| 3 | Compile the generated code | hours | Running out of memory on the heaviest batches |
+| 4 | Link, triage, record | minutes to hours | Missing symbols |
+
+### Milestone 1: pre-flight and reconfigure
+
+Confirm the inputs, then switch `out/build` to the step C settings.
+
+1. Check the runner files and headers match the integrated output:
+   - 71,113 `.cpp` files in `ps2xRuntime/src/runner/`, identical to `Burnout3_recomp/output_bounded/`
+   - both `ps2_recompiled_*.h` headers in `ps2xRuntime/include/`
+   - `register_functions.cpp` marked `S` (skip-worktree) in `git ls-files -v`
+2. Check the dependencies are present in `out/build/_deps` (raylib, SDL2, imgui, rlImGui and
+   the rest), so no network is needed.
+3. Check free disk space: expect several GB of object files, plus up to 10 GB of sccache cache.
+4. Reconfigure (aggressive logs off, `-O1`, asserts kept, so no `-DNDEBUG`):
+
+```bash
+cmake -S . -B out/build -DPS2X_ENABLE_AGRESSIVE_LOGS=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS_RELEASE="-O1"
+```
+
+**Done when:** configure finishes without downloading anything, and `out/build/CMakeCache.txt`
+shows `PS2X_ENABLE_AGRESSIVE_LOGS=OFF`, `CMAKE_BUILD_TYPE=Release` and `CMAKE_CXX_FLAGS_RELEASE=-O1`.
+
+### Milestone 2: runtime library at the new flags
+
+Rebuild the runtime and its dependencies alone, before the hours-long part. This catches
+toolchain or flag problems in minutes.
+
+```bash
+cmake --build out/build --target ps2_runtime -j8 2>&1 | tee Burnout3_recomp/logs/09a_build_ps2_runtime.log
+```
+
+**Done when:** `out/build/ps2xRuntime/libps2_runtime.a` is rebuilt (newer timestamp) with no errors.
+
+### Milestone 3: compile the generated code
+
+The long step. Run it in the background and watch memory use.
+
+```bash
+cmake --build out/build --target ps2EntryRunner -j8 2>&1 | tee Burnout3_recomp/logs/09_build_ps2EntryRunner.log
+```
+
+- **Heaviest batches:** the one containing `register_functions.cpp` (21.8 MB, a lookup table
+  for about 4 million guest addresses) and those with the largest functions, such as
+  `FUN_00186850` (3.6 MB of source).
+- **If memory runs out or the compiler crashes:** rerun the same command with `-j6`, then
+  `-j4`. Make resumes where it stopped.
+- **If a single file is the problem:** shrinking the unity batch size or excluding that file
+  from unity mode means editing upstream CMake, so it needs review first.
+- **Possible name collisions:** two files in one batch could define file-local helpers with the
+  same name. This is unlikely, because generated names include addresses.
+
+**Done when:** every unity batch compiles. The log shows no `error:` lines before the link step.
+
+### Milestone 4: link, triage, record
+
+- **Missing symbols:** link errors usually mean declared functions have no implementation, such
+  as stubs the generated code references that the runtime lacks. Group them by name in the log.
+- **Fix them, preferring the runtime side:**
+  - Burnout 3-specific fixes go in a game override, kept in `Burnout3_recomp/overrides/` and
+    cloned into `ps2xRuntime/src/runner/`. That rebuilds one batch.
+  - Generic gaps go in `ps2xRuntime/src/lib`, which is upstream source, after review.
+  - TOML changes regenerate code and force a full rebuild, so batch them.
+  - Never hand-edit generated files.
+- **Record the result:** note the outcome in `COMPLETED_WORK.md` and update the status table
+  in `ROADMAP.md`.
+
+**Done when:** `out/build/ps2xRuntime/ps2EntryRunner` links cleanly and is newer than the old
+placeholder build, and the log is saved at `Burnout3_recomp/logs/09_build_ps2EntryRunner.log`.
+This proves the code compiles. Missing syscalls, IOP modules and graphics gaps show up in step D.
+
+### Reverting the build settings
+
+```bash
+cmake -S . -B out/build -DCMAKE_BUILD_TYPE= -DPS2X_ENABLE_AGRESSIVE_LOGS=ON
+```