Repository navigation
Fix nine bugs across the executor, agents, and SQL layer - #85
Merged
Merged
Conversation
`run_subprocess` refused only DATABASE_URL and PG_CONN from a command's
`env_snapshot`, then merged the rest of the snapshot into the child
environment. Because `env_snapshot` is untrusted submitter state, a snapshot
carrying `LD_PRELOAD` made the dynamic loader run an attacker-supplied shared
object inside an allowlisted binary:
run_subprocess("/usr/bin/printf hi", cwd, {"LD_PRELOAD": "/tmp/evil.so"})
-> (42, 'PWNED: arbitrary code ran inside an allowlisted binary\n')
That defeats EXECUTOR_ALLOWED_COMMANDS, which the README documents as the
required process isolation boundary in the default deployment: no new
executable is ever launched, so the allowlist check never sees the payload.
Reject the whole class instead of a two-name denylist. `LD_*`, `DYLD_*` and
`BASH_FUNC_*` are matched by prefix, and the glibc path overrides that have
the same effect (GCONV_PATH, HOSTALIASES, LOCPATH, MALLOC_TRACE, NLSPATH,
RESOLV_HOST_CONF) join the exact-name set. Matching is case-insensitive
because the loader itself is not case-sensitive about finding these names in
the environment block. Unrelated names that merely share a prefix boundary
(LDAP_URL) are still passed through.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
latest_output had two branches that disagreed on row order. The p_since_id=0 backfill returned `ORDER BY c.id DESC LIMIT 20`; every later poll returned `ORDER BY c.id ASC`. Both callers render rows in the order they arrive, so both rendered a scrambled transcript: - The htmx UI polls /rpc/latest_output_html with no p_since_id and swaps the result into #output with hx-swap="innerHTML", while submit_command_html appends with hx-swap="beforeend". History loaded newest-at-top and new commands landed at the bottom. - `shell_cli tail` without --since omits p_since_id, so it printed the backlog in reverse before switching to chronological order. Keep the "last N commands" semantics from SPEC.md by confining the DESC and LIMIT to an inner query, then re-sorting ascending. The incremental branch is unchanged. latest_output_html sequenced fragments by `row_number() OVER ()` with no ORDER BY, which is unspecified: it inherited whatever order the planner chose to emit latest_output's rows in. Now that the function's order is well defined, order by the command id and drop the wrapper subquery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
fetch_pending opened its transaction with an explicit `cur.execute("BEGIN;")`.
On a non-autocommit psycopg2 connection the driver has already issued BEGIN
implicitly by the time that statement is sent, so PostgreSQL answered it with
WARNING: there is already a transaction in progress
on every call. fetch_pending is the executor's poll, so a worker at the
default POLL_INTERVAL of one second wrote that warning to the server log
86,400 times a day per worker, drowning out real messages.
Removing the statement does not change the transaction boundary: psycopg2
still opens the transaction implicitly on the requeue UPDATE and the existing
conn.commit() still closes it. Document the non-autocommit requirement that
this now relies on -- test_executor_integration.py already sets
autocommit = False around every fetch_pending call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
cleanup_agent's loop ended the process on any database trouble: - A failure to connect logged an error and `break`, then fell off the end of main() returning None -- so the process exited 0. A momentary restart of PostgreSQL permanently stopped retention, and because the status was 0 a supervisor configured with Restart=on-failure would not bring it back. - An error raised by cleanup_once was not caught at all, so a dropped connection mid-run propagated out and killed the daemon the same way. Retry instead, matching what monitor_agent already does: log, back off by RETRY_DELAY_SECONDS (capped at --interval), and continue. The connection is closed before the backoff sleep rather than being held across it. --once keeps one-shot semantics but now returns 1 on failure, so cron and systemd see that the run did not happen. main() returns its status and __main__ raises SystemExit with it, as replay_agent and monitor_agent do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
cleanup_once reset every stale environment to the literal '/home/sandbox'
while the executor resolves the sandbox root from `SHELL_ROOT`. A deployment
that configures SHELL_ROOT -- as the README's own worked example does with
/home/pg-shell-command -- had its sessions reset to a directory outside that
root, which usually does not exist. The executor passes environments.cwd
straight to the subprocess, so every later command in a reset session failed:
run_subprocess("/usr/bin/printf hi", "/home/sandbox", None)
-> FileNotFoundError: [Errno 2] No such file or directory: '/home/sandbox'
Read SHELL_ROOT in cleanup_agent too, keeping /home/sandbox as the default so
existing deployments are unaffected, and note in the README that both agents
need the same value.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
fetch_pending selected a single candidate command and returned None when its
per-session advisory lock was already held. Sessions are independent, so
giving up there starved every session queued behind the locked one:
session A (submitted first): claim lock held by a peer worker
session B (submitted second): pending, ready to run
worker B poll 1: None
worker B poll 2: None
worker B poll 3: None
Worker B spun at POLL_INTERVAL and never ran session B's command for as long
as the peer held session A's lock -- reachable whenever a worker's lease
expires while it is still alive and still holding the lock, which is exactly
the case the requeue at the top of fetch_pending exists to handle.
SPEC.md section 3 asks for per-user serialization "while still allowing
different users to execute concurrently". Fetch up to CLAIM_CANDIDATES rows
instead of one and take the first session whose lock can be acquired. The
NOT EXISTS predicate already admits at most one command per session, so the
candidates are distinct sessions in submission order and the serialization
guarantee is unchanged: a session is still claimed only via its own lock, and
a session that is locked is left untouched for the worker that owns it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
ensure_monitor_state_table issued CREATE TABLE IF NOT EXISTS but never
committed, leaving it to whatever transaction the rest of the run happened to
commit. save_monitor_state is the only commit in the path and it is skipped
whenever there is no watermark to store -- always under --since-hours and
--since-days, and on any run with nothing newly completed. In those cases
conn.close() rolled the table back:
$ dropdb-side: DROP TABLE monitor_state
$ python -m workers.monitor_agent --once --since-days 1
$ SELECT to_regclass('monitor_state') IS NOT NULL -> f
So the agent re-ran CREATE TABLE on every poll and, on a deployment that had
not installed init_schema.sql, could never keep an incremental watermark --
load_monitor_state always read back an empty table and collect_metrics
re-aggregated the full history each time.
Commit right after creating it. Its own transaction is the right boundary
anyway: the table is setup, not part of the metrics snapshot that follows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
`from __future__ import annotations` sat above the module docstring, so the string was parsed as an ordinary expression statement rather than a docstring and monitor_agent.__doc__ was None -- `help(workers.monitor_agent)` and `pydoc` showed nothing. A __future__ import is only allowed to precede the docstring, not required to, so move it below. Also annotate main() as returning int: it already returns 0 or 1 and __main__ raises SystemExit with the result, so the -> None was wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
psql's \i takes the rest of the line as arguments, so the trailing comment on
\i sql/latest_output.sql -- updated latest_output with optional since_id filter
produced seven "warning: \i: extra argument ... ignored" lines on every
install. The file still loaded, but the noise buries real diagnostics in
install output. Move the comment onto its own line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine independent issues found by auditing the codebase against SPEC.md and by exercising each component against a live PostgreSQL 16 instance. Each is one self-contained commit with its own regression test, so any of them can be reverted or cherry-picked alone.
Every test added here was confirmed to fail against the unpatched code and pass after the fix. Full suite: 102 passed, 1 skipped (was 76 passed, 1 skipped).
Findings
1.
env_snapshotbypasses the executable allowlist (6c87561)run_subprocessrejected onlyDATABASE_URLandPG_CONN, then merged the rest of the untrusted snapshot into the child environment. A snapshot carryingLD_PRELOADmakes the dynamic loader run attacker-supplied code inside an allowlisted binary:No new executable is ever launched, so the allowlist check never sees the payload — defeating what the README documents as the required process isolation boundary. Now rejects
LD_*,DYLD_*,BASH_FUNC_*by prefix plus the glibc path overrides with the same effect, case-insensitively.LDAP_URLand friends still pass through.2.
latest_outputreturns the two branches in opposite orders (27d0cdd)The
p_since_id = 0backfill returnedORDER BY c.id DESC LIMIT 20; every later poll returnedORDER BY c.id ASC. Both callers render rows in arrival order, so both rendered a scrambled transcript: the htmx UI loaded history newest-at-top whilehx-swap="beforeend"appended new commands at the bottom, andshell_cli tailwithout--sinceprinted the backlog in reverse before switching to chronological. The DESC + LIMIT now lives in an inner query so "last N commands" is preserved while the result is oldest-first.latest_output_htmlalso sequenced fragments byrow_number() OVER ()with noORDER BY, which is unspecified — it inherited whatever order the planner chose. Now ordered by command id.3. Executor warns on every poll (
2ea2663)fetch_pendingopened its transaction with an explicitBEGINthat psycopg2 had already issued implicitly, so PostgreSQL answered every poll withWARNING: there is already a transaction in progress— 86,400 log lines a day per worker at the defaultPOLL_INTERVAL. Verified: 18 warnings before the fix, 0 across an 8-second polling run after.4. Cleanup daemon dies on any transient database error (
c70701b)A failed connection logged and
break, then fell off the end ofmain()returningNone— so the process exited 0. A momentary PostgreSQL restart permanently stopped retention, and the zero status meantRestart=on-failurewould not bring it back. Errors fromcleanup_oncewere not caught at all. Now retries with backoff likemonitor_agentalready does;--oncereturns 1 on failure so cron and systemd notice.5. Cleanup resets sessions outside
SHELL_ROOT(aee3145)cleanup_oncereset stale environments to a literal/home/sandboxwhile the executor resolves the root fromSHELL_ROOT. A deployment following the README's own worked example (SHELL_ROOT=/home/pg-shell-command) had sessions reset to a directory that usually does not exist, and the executor passesenvironments.cwdstraight to the subprocess:Every later command in a reset session failed. Default is unchanged, so existing deployments are unaffected.
6. One busy session starves every other session (
2c3f728)fetch_pendingselected a single candidate and returnedNonewhen its per-session advisory lock was held, so sessions queued behind it never ran:Reachable whenever a worker's lease expires while it is still alive and holding the lock — exactly the case the requeue at the top of
fetch_pendingexists to handle. SPEC.md §3 asks for per-user serialization "while still allowing different users to execute concurrently". Now fetches up toCLAIM_CANDIDATESrows and takes the first session whose lock is free. TheNOT EXISTSpredicate already admits at most one command per session, so candidates are distinct sessions and the serialization guarantee is unchanged.7.
monitor_stateis never committed (ef846b0)ensure_monitor_state_tablecreated the table but left committing tosave_monitor_state, which is skipped on every--since-hours/--since-daysrun and any run with nothing newly completed.conn.close()then rolled it back:The agent re-ran
CREATE TABLEevery poll and, withoutinit_schema.sqlinstalled, could never keep a watermark — socollect_metricsre-aggregated the full history each time.8.
monitor_agent.__doc__isNone(5903fcb)from __future__ import annotationssat above the module docstring, demoting it to a dead expression statement;help()andpydocshowed nothing. Also correctedmain() -> None, which already returned exit codes.9.
install.sqlwarns on every run (5c766a2)psql's
\itakes the rest of the line as arguments, so a trailing comment produced sevenwarning: \i: extra argument ... ignoredlines per install, burying real diagnostics.Verification
sql/install.sqlapplies cleanly to a virgin database and is idempotent on re-run, with zero psql warnings.cdupdates the session environment, and the following command picks up the new cwd at claim time.Note on PR granularity
You asked for one PR per issue. This session is pinned to a single designated branch, so all nine landed here as separate commits instead — each is self-contained with its own test and can be reverted independently. Happy to split them into separate PRs if you'd prefer.
🤖 Generated with Claude Code
https://claude.ai/code/session_017jvrX4SVJvSJKVsSHKCNEe
Generated by Claude Code