[danshapiro]

Fixes a group of server and WebSocket tests that failed now and then in pre-push hooks and GitHub CI, mostly when the machine was busy. No test was skipped, no assertion was loosened, and no sleep was made longer. Each flake was reproduced first, and each fix was checked by repeated runs under load. "Load" below means a load average of roughly 100 to 900 on the 96-core development server, from parallel copies of the test suite plus other agents' work.

The shared cause

Most of these flakes come from one fact. When any thread starts a child process (Git, a shell, a terminal, a sidecar), the child briefly holds a copy of every file and socket the server has open, until it starts its own program. So a lock or listening socket that the server "closed" can stay alive for that moment. A test that started /bin/true in a tight loop from 8 threads made this happen on demand.

Each test

1. session_names::tests::cancelled_wait_retains_executing_lock ("lock releases when the transaction completes")

• Cause (product): the session-names store released its file lock only by closing its own handle, so a child started during a naming transaction kept the lock held after the transaction ended. With the child-starting loop running, the lock was still held after 156 of 176 commits.
• Fix: dropping the document lock or the worker lock now unlocks it explicitly, which releases it for every copy. Two new tests hold a child process open mid-start and fail without the fix.
• After: 0 of 1,014 in the commit-and-check loop; 36 of 36 with the child-starting loop; 40 of 40 in group runs under load.

2. The four "document lock stayed busy" tests (a_capability_paused_head_does_not_starve_the_ready_series_behind_it, an_opencode_native_series_resumes_when_the_shared_serve_arrives_after_boot, the_worker_loop_converges_an_armed_series, capability_pause_consumes_nothing_and_resumes_on_the_wake)

• Not reproduced on current main in 120 full-suite runs under load; PR Fix three stale or load-sensitive tests and queue same-process name-store transactions #858 covers what it changed. The remaining way in is the same held lock as item 1 (one of these failed once in 24 runs with the child-starting loop before the fix, never in 36 after), so item 1's fix covers it.

3. checkpoints::tests::cooldown_sends_immediately_without_capture

• Cause (test): the 1 s limit also covered finding the workspace, which runs Git, and saving the outcome, which syncs files to disk. Under load the whole send took 1.5 to 4.9 s (48 of 48 full-suite runs at load ~800).
• Fix: the test holds the workspace's capture lock for the whole send, so a cooldown that waited for it would fail. Test builds now record when the workspace was found, and the 1 s limit applies from that moment to the cooldown's answer. A cooldown that stalled for 5 s fails the test.
• After: 40 of 40 checkpoint-module runs at load 160 to 240, plus the earlier 120 + 60 module runs and 32 full-suite runs.

4. checkpoints::tests::review_r6_automatic_checkpoint_cooldown_is_logged and 5. extensions::tests::scan_warns_with_legacys_two_log_lines_for_the_two_failure_classes

• Cause (test setup): the tracing logging library remembers, for each log line, whether anyone is listening, the first time that line runs. If another test's thread hit the line first, or hit it while the shared log capture was being installed, the line was marked "nobody listens" for the whole test process. That is how the cooldown log came back empty (19 of 120 module runs). The extension test did not fail in 1,500 module runs, but a standalone program showed the install-time window in 5 of 2,000 runs, and other extension tests hit the same two warning lines while this test installs the capture.
• Fix: the shared log capture is now installed before any test starts (new dev-dependency ctor, already in Cargo.lock), and it keeps only the events it captures. A new test fails without this.
• After: 0 failures in 120, 60 and 200-run checks; extensions 300 of 300.

6. net_bind::tests::rebind_serves_a_connection_queued_on_the_retiring_listener (connection reset by peer)

• Cause (product and test): a child process kept a retired listener open, so new connections queued on a listener nobody served and were reset. The same thing happened in production after a rebind. Separately, the test picked its port on 127.0.0.1 only, so a later rebind to 0.0.0.0 could fail with "address in use".
• Fix: a retired listener is now shut down explicitly, which stops it on Linux for every copy. The test fixture picks its port on 0.0.0.0. A new test, Linux-only because only Linux stops a listening socket this way, fails without the fix.
• After: the rebind tests with the child-starting loop went from 21 of 24 failing to 24 of 24 passing.

7. term09_output_queue::incident_backlog_spills_instead_of_disconnecting and slow_but_progressing_client_survives_over_threshold_backlog

• Cause (test): both read the terminal flood at a fixed ~1 MiB/s and assumed the flood would get far enough ahead to overflow the queue. On a busy machine the flood is produced about that slowly, so the queue never overflowed and the "must produce a gap" check failed (the incident test 1 of 18 at load ~470; the slow-client test 1 of 4 at load ~100).
• Fix: the reading client keeps reading, but only one frame per pause, until the server's own "queue overflowed" log event appears; then it reads at ~1 MiB/s. Also, the shared reading helper cut the end-of-flood marker out of its buffer before looking for it, so both tests always ran until their 60 s and 75 s deadlines. It now looks first, so both tests end when the flood ends. All assertions are unchanged.
• After: 32 of 32 runs of both tests at load 150 to 210. The two tests together now take about 50 s instead of about 135 s.

8. Environment-variable collisions (including real_gateway_outage_preserves_exact_cold_native_history_without_live_authority)

• Cause (test): tests that change HOME, CLAUDE_CONFIG_DIR and similar settings each held a different lock, so they changed the same settings at the same time. That test plus the Claude route-discovery tests failed 76 of 120 runs. Some session-directory tests also read CLAUDE_HOME without any lock while two session-name tests set it; run together, they failed 60 of 60.
• Fix: one shared guard (TestEnv) now owns the environment for the whole test binary and restores everything it changed, even when a test fails. Tests that can name their inputs directly now do, instead of reading the environment: the session-directory tests pass their test home's Claude folder explicitly.
• After: 120 of 120 for the first combination; 60 of 60 for the session-directory combination.

Overall

Full freshell-server unit suite, 30 runs at load 77 to 148: the only failures were in settings_store::tests::override_reload_is_throttled_to_a_configurable_window (17 runs) and concurrent_persists_across_two_store_instances_serialize_without_lost_updates (1 run). Neither is touched here; in an earlier side-by-side check the first failed 8 of 16 runs both with and without this branch. They are left for a separate fix. cargo fmt --check and clippy -D warnings (all targets) are clean for freshell-server and freshell-ws.

🤖 Generated with Claude Code