nix: only install the Unity stdio hooks in a Unity player

[colefuerth]

Fixes #88.

Problem

LD_PRELOAD is inherited by every child process, so Doorstop is also loaded into the launcher scripts that start the game. When no UnityPlayer module is present, doorstop_ctor() falls back to hooking the main executable's PLT and installs fclose_hook and dup2_hook there.

Both hooks deliberately suppress redirection of stdout/stderr — which is what a shell uses for $(...) and > — so any /bin/sh in the launch chain silently breaks:

$ LD_PRELOAD=./libdoorstop.so DOORSTOP_ENABLED=1 DOORSTOP_TARGET_ASSEMBLY=... \
    sh -c 'x="$(echo hello)"; echo "got=[$x]"'
hello
got=[]

That is the cause behind the SteamLaunch re-exec dance in assets/nix/run.sh, which exists specifically to delay adding Doorstop to LD_PRELOAD until after the Steam Linux Runtime's shell scripts have parsed their arguments. SteamLinuxRuntime_soldier/_v2-entry-point does getopt_temp="$(getopt ...)", gets an empty string back, and exits with Error: A command to run is required.

Change

Only install the stdio hooks when the PLT we opened actually belongs to a Unity player:

• UnityPlayer is mapped, or
• we fell back to the main executable and that executable ships a Unity data folder beside it (<name>_Data).

The check fails open: if the executable path cannot be read at all (no /proc, non-dumpable process) the hooks stay installed, because a false negative silently costs a real player its console output while a false positive is loud and easy to diagnose. It also probes the full file name before the extension-stripped one, so a player called Game.Client finds Game.Client_Data.

macOS is left exactly as it is today: bundles lay the data folder out differently and I have no Mac to verify against, so main_module_is_unity_player() returns TRUE there and behaviour is unchanged. Happy to extend it if someone can test.

dlsym stays hooked, since it is how injection works for older Unity games where the player is the main executable. Note that it is not inert in a non-Unity process that resolves one of the redirected symbol names — REDIRECT_INIT matches on the symbol name alone — but that is existing behaviour this PR does not change; gating it is a separate discussion. The boot_config_override fopen/fopen64 hooks are likewise left in place; they only gained a NULL filename guard, since they are installed process-wide and the real fopen merely fails on a NULL path rather than crashing.

Testing

Built from source and verified on Linux x64 (Unity 2021.3.45f2 Mono, BepInEx 5.4.23.5, STRAFTAT), against master (8e66ca0) as the control:

1. The shell bug is fixed — same command as above:

	command substitution	file redirection
master	got=[]	got=[]
this PR	got=[hello]	got=[redir]

2. Detection behaves in both directions. A test binary placed in various layouts, reporting whether dup2(fd, 1) was suppressed:

layout	hooks
Game.x86_64 + Game_Data/	installed
Game.Client + Game.Client_Data/	installed
Mr.Prepper.x86_64 + Mr.Prepper_Data/	installed
executable with no _Data folder	skipped

3. The end-to-end case works. Running the Steam Linux Runtime chain with the Doorstop environment exported around it, which is what a mod manager wants to do instead of the run.sh re-exec:

# master
[198008]: Error: A command to run is required

# this PR
pressure-vessel-wrap[197598]: ...        # reaches the container, same as with no Doorstop at all

4. fopen(NULL, "r") with DOORSTOP_BOOT_CONFIG_OVERRIDE set: segfault on master, returns NULL with this PR.

There is no test harness in the repo, so the above is manual; the one-line sh -c repro from #88 is the cheapest regression check if you ever add one.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TTHrLRGBdwxEMboXdCRmdj

[arrowmaster]

I tested on native linux Valheim which uses Unity v6000.0.75f1 and this appears to be working good. I completely removed the workaround from the run.sh during testing.