Skip to content

THI-450: build_state forks less than half as many processes per refresh - #55

Merged
tdody merged 46 commits into
mainfrom
thibaultdody/thi-450-build_state-runs-too-many-subprocesses-per-tick-so-the
Nov 6, 2026
Merged

tdody merged 46 commits into
mainfrom
thibaultdody/thi-450-build_state-runs-too-many-subprocesses-per-tick-so-the

Conversation

@tdody

@tdody tdody commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes THI-450. Also fixes the main case of THI-453: Codex panes were missed because ps cut the process name to 16 characters. Executables are now read from argv[0].

build_state runs about once a second. On a loaded machine each poll started hundreds of processes per minute, and the dashboard fell 3–13 s behind. This PR makes each poll read the system once and share the result:

  • One ps table per poll. agent_processes.process_snapshot() is the only process table. It runs ps -axo pid=,ppid=,lstart=,state=,args= under LC_ALL=C/TZ=UTC.
    • It is shared by session state, the pane → claude map, codex detection and resources.
    • agent_processes.is_claude_executable is the one claude test.
    • If ps fails or times out (2 s), the last table stands in while it is under 15 s old.
  • One tmux list-panes -a (tmux.list_panes) replaces list-windows. A newline inside a field is rejoined, not dropped. list_windows still reads fresh, so a window that was just stamped or created reads back at once.
  • Snapshots last for the whole poll. build_state pins the list-panes rows and the ps table it took (tmux.pinned_snapshot, agent_processes.pinned_snapshot, both ContextVars). Every reader in that poll, the fan-out threads included, gets them however long the poll runs. Callers outside a poll get a 1 s cache (tmux.panes_snapshot).
  • Git state in two commands, refreshed in the background.
    • git_state_for runs status --porcelain=v2 --branch, then diff-index -M --shortstat HEAD. Both run with --no-optional-locks and under LC_ALL=C, and neither writes the index.
    • cached_git_state serves the cached value and refreshes it in the background every 15 s ± 5 s. A cwd seen for the first time is still read on the spot, because pid rebinding needs its branch.
    • Repo identity is cached for 10 minutes once a GitHub slug is found, the rule reviews already uses.
  • periscope/gh.py. Every gh call gets a GH_TOKEN read every 10 minutes, so gh's keyring no longer runs /usr/bin/security on each call.
    • Telemetry and the update check are off.
    • A 401 with the token retries once without it (*.ghe.com), and that cwd is remembered until the next token read.
    • gh.AVAILABLE is the one check for whether gh is installed.
  • Measuring it. periscope/forks.py counts processes through a Python audit hook.
    • /api/state gains a build_stats block: build time and the gap between polls, both per poll; processes per minute and top commands, both server-wide over 60 s.
    • /api/healthz has the full map. A poll slower than 3 s logs "slow poll", at most once a minute.
    • The resource popover ends with poll 0.4 s · every 1.1 s · server 140 forks/min, then the slowest build and the top commands.
  • Notification cleanup (_channel_gc) keeps a pane's notification if it arrived during the current poll, since the poll's pane list can be as old as the poll.

Before / after

Dev server on :17375, 10 panes, 90 s per run. "Before" is acd98472: it already reports the poll's cost but still uses the old commands. A dev server skips the narrator, the session poke and the MCP listener, so compare these runs with each other, not with prod.

Under the ticket's loaded subprocess timings. Wrapper scripts on the server's PATH delay ps 1.0 s, tmux 0.2 s and git 1.5 s. The ticket measured ps at 0.5–1.7 s, list-panes at 0.15–0.26 s and git status at 1.4–3.3 s.

run frames in 90 s median frame gap p90 gap build median / max forks/min
before-s1 1 — (one poll took 173 s) — 172.7 s 225
after-s1 27 2.76 s 3.94 s 1.75 / 4.49 s 121
before-s2 1 — (one poll took 174 s) — 173.5 s 186
after-s2 20 2.78 s 3.90 s 1.77 / 8.44 s 115

In the slow "after" runs, ps and tmux list-panes came to about one per poll (19–20 per minute at ~20 polls per minute). That held even with builds over 1 s.

No added delay:

run frames median frame gap p90 gap build median / max forks/min
before-i1 58 1.30 s 1.97 s 0.34 / 3.05 s 548
after-i1 72 1.15 s 1.54 s 0.15 / 1.42 s 213

Processes per minute by command, idle runs:

before after
ps 117 46
tmux list-panes 78 46
tmux list-windows 38 —
tmux capture-pane 59 46
git rev-parse / rev-list / remote / ls-files 104 / 34 / 34 / 33 —
git diff → git status + diff-index 33 31 + 31
gh pr 8 8

I checked gh's security child directly: 10 gh api rate_limit calls without a token showed one, and with GH_TOKEN and the quiet flags they showed none.

Assumptions a reviewer may want to challenge

  • Untracked count from a subdirectory: status counts untracked files across the whole repo, so a cwd in a subdirectory can go from clean to ?3.
  • Detached HEAD: a pane shows @ plus 7 characters of the commit. Before, git's abbreviation chose the length, which here was 8, so a detached pane's branch changes once on deploy.
  • Repos with a clean filter (e.g. git-crypt): nothing writes a refreshed index back, so files that look changed but aren't are re-hashed on every git refresh.
  • No GitHub remote: a repo without one re-reads its identity on every git refresh, about two forks per 15 s, the same as before.
  • The branch shown can lag a checkout by up to ~20 s, and pid rebinding uses that same possibly stale branch. A background read that returns nothing keeps the last state while the cwd is still a directory.
  • The executable is argv[0]. claude and codex don't rewrite their process title. On this machine the old and new claude tests found the same 12 panes.
  • A \x1f in a window name or path still drops that pane's row. The full-roster passes then treat the window as gone: its track row is pruned, and boot prunes its session rows. A newline in those fields is rejoined.
  • Snapshot age inside a poll: the poll's pane rows are as old as the poll, up to 3–13 s under the ticket's load.
    • A pane created during the poll reads "no claude yet" and fails open.
    • Its notification is kept: the cleanup spares anything that arrived during the poll.
  • A slow ps: if the 2 s ps times out, the last table (under 15 s old) stands in. A claude that exited or started in that time reads as it did then; session ids still come from the fresh 1 s index. Past 15 s, the four readers blank for one poll.
  • gh tokens:
    • A token revoked within the 10 minutes fails until the 401 re-read, at most a minute later.
    • A token every host rejects makes each gh call fork twice until the user logs in again.
  • Fork counts are server-wide over 60 s, not per poll. THI-450 asked for them per poll, by caller. Per-poll attribution across the fan-out threads would need context tracking in the audit hook. Build time and the gap are per poll.

Checks

  • uv run pytest -q: 2168 passed
  • npm test: 1321 passed
  • bin/check: exit 0
  • static/dist up to date

Spec: docs/superpowers/specs/2026-11-02-fewer-forks-per-refresh-design.md · plan: docs/superpowers/plans/2026-11-02-fewer-forks-per-refresh.md · review: two rounds, every finding fixed, the second round found nothing blocking

🤖 Generated with Claude Code

tdody and others added 19 commits November 2, 2026 14:41
…napshot, git in the background, gh without keychain reads (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…utables and C locale for ps, fresh list_windows over a shared snapshot, git fallbacks and keep-on-failure, gh token TTLs and 401 shapes, interleaved measurement (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…over a rolling minute (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…; proc_table and live claudes read it (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nd the pane pid map (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nt says one row per window; the spec states the snapshot's staleness (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… session_status's ps eww can't drift from the snapshot's locale; lstart parses as UTC (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d for 10 minutes (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… s, first sight still synchronous (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d update check off (THI-450)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t runs under LC_ALL=C; isdir stat moves outside the refresh lock; _GIT_TTL renamed to _GIT_REFRESH_S (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…he short sha's real growth rule, the identity-TTL and clean-filter assumptions (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… gh in the module table (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…igration notes; comments without history; two snapshot assumptions (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@tdody tdody changed the title THI-450: build_state forks a third as many processes per refresh THI-450: build_state forks less than half as many processes per refresh Nov 3, 2026
tdody and others added 10 commits November 3, 2026 08:13
…v (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… without it (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d reviews' own (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ound (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ows, not dropped (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… guard; _git_fetching/_fetch_git_state_into_cache match the PR cache's names (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oes, not an invented history (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts own global (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…sets LC_ALL (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tdody and others added 17 commits November 3, 2026 09:06
…a ps slower than the TTL still serves the callers after it (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…the ps table it took, so a build slower than the 1 s TTLs still forks each once (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… it is under 15 s old, with a warning at most once a minute, instead of blanking the refresh (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…process snapshot changes, not on a clock of its own (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…de test; the pane walk uses it over descendants (now nearest first), and proc_table becomes command_lines (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… and kept 15 s on failure, one claude test, the loaded ps timings (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…not a refresh; its fork counts are server-wide (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s as visible rows, not a hover-only tooltip (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ter the poll's start (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…OKEN forever (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h, no can't-start guard, four assumptions (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ach twice (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ended (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…(THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…es (THI-450)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…build_state-runs-too-many-subprocesses-per-tick-so-the

# Conflicts:
#	static/dist/app.js
…build_state-runs-too-many-subprocesses-per-tick-so-the

# Conflicts:
#	static/dist/app.js
@tdody
tdody merged commit 914c009 into main Nov 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant