Summary
On Unix, Doorstop installs dup2_hook and fclose_hook into the main executable's PLT whenever no UnityPlayer module is present. Since LD_PRELOAD is inherited by every child process, that includes the launcher scripts that start the game — and both hooks deliberately suppress redirection of stdout/stderr, which is exactly what a shell uses to implement $(...) and >.
The result is that any /bin/sh in the launch chain silently stops working.
Reproduction
No Unity or Steam needed — any shell will do:
$ LD_PRELOAD=./libdoorstop.so DOORSTOP_ENABLED=1 DOORSTOP_TARGET_ASSEMBLY=/path/to/BepInEx.Preloader.dll \
sh -c 'x="$(echo hello)"; echo "substitution: [$x]"; echo redir > /tmp/t; echo "redirect: [$(cat /tmp/t)]"'
hello
substitution: []
redirect: []Expected:
substitution: [hello]
redirect: [redir]
Reproduced against master (8e66ca0) built from source, and against the shipped 4.5.0 release.
Real-world impact: BepInEx is broken on every Steam Linux Runtime game
Steam launches native Linux games through the Steam Linux Runtime:
_v2-entry-point --verb=waitforexitandrun -- scout-on-soldier-entry-point-v2 -- <game>
Both entry points are shell scripts. SteamLinuxRuntime_soldier/_v2-entry-point parses its arguments with:
getopt_temp="$(getopt -o '' --long "$getopt_temp" -n "$me" -- "$@")"
eval "set -- $getopt_temp"
With Doorstop preloaded that command substitution returns an empty string, so $@ is destroyed and the script exits at its own guard:
if [ "$#" -eq 0 ] || [ "$1" = -- ]; then
log "Error: A command to run is required"
usage 125
fiThe game never starts. Demonstrated with STRAFTAT (native Linux, Unity 2021.3.45f2, Mono):
# Doorstop environment set around the runtime chain
$ LD_PRELOAD=libdoorstop.so DOORSTOP_ENABLED=1 DOORSTOP_TARGET_ASSEMBLY=... \
_v2-entry-point --verb=waitforexitandrun -- scout-on-soldier-entry-point-v2 -- STRAFTAT.x86_64
[27623]: Error: A command to run is requiredThis is, I believe, why Linux mod managers have to work around Doorstop rather than simply exporting its environment — e.g. ebkr/r2modmanPlus#1848.
Cause
doorstop_ctor() in src/nix/entrypoint.c:
void *unity_player = plthook_handle_by_name("UnityPlayer");
if (unity_player && PLTHOOK_OPEN_BY_HANDLE_OR_ADDRESS(&hook, unity_player) == 0) {
LOG("Found UnityPlayer, hooking into it instead");
} else if (plthook_open(&hook, NULL) != 0) { // <-- the main executable
...
}
...
plthook_replace(hook, "fclose", &fclose_hook, NULL);
plthook_replace(hook, "dup2", &dup2_hook, NULL);In a shell there is no UnityPlayer, so the fallback patches the shell's own dup2/fclose. Both hooks are written for Unity's behaviour specifically and are unconditionally destructive elsewhere:
int dup2_hook(int od, int nd) {
// Newer versions of Unity redirect stdout to player.log, we don't want that
if (nd == fileno(stdout) || nd == fileno(stderr))
return F_OK; // silently refuses the redirection
return dup2(od, nd);
}Note the fallback itself is legitimate — older Unity games have no separate UnityPlayer.so and the player is the main executable — so it can't simply be removed; it needs to distinguish a player from a launcher.
Related
#110 proposed this same scoping ("scope Unix stdout/fclose hooks to UnityPlayer so inherited LD_PRELOAD no longer breaks shell command substitution or file redirection") but bundled a large set of unrelated cross-platform changes and was closed with a request to split it up. This issue covers only the stdio scoping; I have a focused PR for it.
Summary
On Unix, Doorstop installs
dup2_hookandfclose_hookinto the main executable's PLT whenever noUnityPlayermodule is present. SinceLD_PRELOADis inherited by every child process, that includes the launcher scripts that start the game — and both hooks deliberately suppress redirection ofstdout/stderr, which is exactly what a shell uses to implement$(...)and>.The result is that any
/bin/shin the launch chain silently stops working.Reproduction
No Unity or Steam needed — any shell will do:
Expected:
Reproduced against
master(8e66ca0) built from source, and against the shipped 4.5.0 release.Real-world impact: BepInEx is broken on every Steam Linux Runtime game
Steam launches native Linux games through the Steam Linux Runtime:
Both entry points are shell scripts.
SteamLinuxRuntime_soldier/_v2-entry-pointparses its arguments with:With Doorstop preloaded that command substitution returns an empty string, so
$@is destroyed and the script exits at its own guard:The game never starts. Demonstrated with STRAFTAT (native Linux, Unity 2021.3.45f2, Mono):
This is, I believe, why Linux mod managers have to work around Doorstop rather than simply exporting its environment — e.g. ebkr/r2modmanPlus#1848.
Cause
doorstop_ctor()insrc/nix/entrypoint.c:In a shell there is no
UnityPlayer, so the fallback patches the shell's owndup2/fclose. Both hooks are written for Unity's behaviour specifically and are unconditionally destructive elsewhere:Note the fallback itself is legitimate — older Unity games have no separate
UnityPlayer.soand the player is the main executable — so it can't simply be removed; it needs to distinguish a player from a launcher.Related
#110 proposed this same scoping ("scope Unix stdout/fclose hooks to UnityPlayer so inherited
LD_PRELOADno longer breaks shell command substitution or file redirection") but bundled a large set of unrelated cross-platform changes and was closed with a request to split it up. This issue covers only the stdio scoping; I have a focused PR for it.