Skip to content

Fix flaky agent-crate tests and five cloud e2e tests - #869

Merged
danshapiro merged 15 commits into
mainfrom
fix/flaky-agent-tests
Oct 23, 2026
Merged

danshapiro merged 15 commits into
mainfrom
fix/flaky-agent-tests

Conversation

@danshapiro

Copy link
Copy Markdown
Owner

Fixes intermittent failures in the agent crates and five cloud e2e tests. Each was reproduced first (or, where it would not reproduce, the race was shown by forcing the timing), then fixed at the cause. No test was skipped or loosened, and no fixed sleep was made longer. "Under load" below means the host load average was about 100–900 on 96 cores (busy-loop burners plus other agents' work).

Rust tests

  1. session_watcher metadata_late_subagent_named_mkdir_arms_via_deferred_recheck_without_a_discover (freshell-sessions)

    • Cause (test): the watcher marks the recheck as armed before it applies the index entry, and a background refresh publishes the new row. The test read the index once, right after it saw the arm.
    • Fix: wait for the row within the same 5 s budget, then check that no full discover ran.
    • Before: 2/20 failed quiet, 13/30 under load. After: 140/140 under load.
  2. session_handoff opencode_handoff_stop_waits_for_a_mid_dispatch_rest_drive_under_the_gate (freshell-freshagent)

    • Cause (test): the release used notify_waiters(), which is lost if the drive has not reached its pause yet. The drive then holds the gate and the handoff times out. The "prompt POST" check also matched the first drive's POST, so it proved nothing about the second.
    • Fix: a test-only handshake (stored permits) in lib.rs / opencode_ws.rs. Assertions now use the second drive's POST.
    • Did not fail naturally in about 90 runs. A 400 ms delay injected before the pause failed it every time; with the fix it passes. 90/90 under load. Removing the product gate now fails the test at once.
  3. session_handoff an_opencode_placeholder_handoff_resolves_the_durable_owner_and_completes (freshell-freshagent)

    • Cause (test): a claude_snapshot test removed HOME for the whole process while this test was reading it.
    • Fix: that test no longer removes HOME (its own result never depended on it).
    • Before: 20/30 failed when run with that test. After: 40/40 for the pair, 30/30 with all 67 claude_snapshot tests under load.
  4. claude a_sidecar_exit_after_the_live_verdict_never_gets_its_stamp_restored and seven sibling tests (freshell-freshagent, test-only changes in claude.rs)

    • Cause (test): a new session first publishes its alias, then writes its identity record, then records ownership together with the release stamp. These tests waited for the alias or the identity record and then read or took the stamp, which might not exist yet. One test also set a pause that the first session could get caught in.
    • Fix: shared helpers wait for the stamp itself, which is the real condition.
    • A 60 ms delay injected before the ownership step failed seven of the eight old tests. With it injected, all 201 claude unit tests pass. Without it, all eight passed 40/40 alongside 200 busy loops.
  5. freshagent_session_lease expired_lease_kill_sweeps_the_sidecar_tree_before_release (freshell-ws)

    • Product fix: the expired-holder kill reads /proc files of processes it is killing. A process reaped between open and read returns ESRCH, which was treated as "can't tell", so the kill counted as unconfirmed and the second client was refused. ESRCH now means "process gone". A regression test reaps a real child mid-read and fails under the old handling.
    • Test fixes:
      • The test's decoy sleep 300 is now killed on cleanup. Before, it was left running after every run.
      • The second client's sidecar no longer inherits the decoy setting or the first client's 5 s create delay.
      • The test fails at once on create.failed.
      • The decoy now ignores SIGTERM. Before, the claude lane's own SIGTERM of its tagged processes also killed the decoy, so the test kept passing with the lease sweep removed. With the sweep removed, the test now fails.
    • The created timeout did not reproduce in about 110 sandbox runs. After the fix: 30/30 full-file runs under load, with no processes left behind.

E2E tests (cloud)

  1. unified-agent-names-restart "[freshopencode] pending rename survives a zero-turn restart"

    • Cause (product, found from two retained cloud traces of failures): after a server restart, the browser's reconnect attach names the pane's not-yet-created freshopencode-* placeholder. The server tried to resume it. It took the ownership claim for that id and held it while it started the shared opencode daemon, only to get "not found". The user's first message landed inside that window. Its claim on the same id was refused with SESSION_RESERVED. The browser then retried the attach, not the message, so the message was silently lost. The traces show this: the message was echoed, status went "running" then "idle" within 300 ms, and there was no error banner and no new session.
    • Fix: resuming a placeholder id now answers "not found" at once, with no claim and no daemon start. A unit test that holds the daemon start open reproduces the exact failure on the old code.
    • Before: about 1 failure in 30–90 runs. After: 100/100 on cloud (3 workers), and the unit test passed 40/40 under load.
  2. handoff-two-device-rust "codex: reopen-as-CLI on desktop converges the phone pane"

    • Cause (product, browser client): the handoff's HTTP answer can switch the pane to the new terminal before the WebSocket owner update arrives. The attach is refused while the old owner is still recorded, and nothing retried it.
    • Fix: TerminalView retries the attach once the owner record names this pane's terminal. A unit test fails before the fix.
    • Before: 8/20 failed. After: 90/90 across three cloud runs.
  3. fresh-agent-rollback-rust "codex: undo-to-here … /redo is refused with the codex copy"

    • Cause (test): the pane's own undo confirmation could arrive after /redo was typed. It refilled the composer and replaced the redo notice.
    • Fix: wait for the refill before typing.
    • Before: 4/20 failed. After: 177/180 across five cloud runs. The 3 failures came back-to-back inside one 40-second window of a single cloud task, in a run without retries, so no trace was kept. The 160 runs after that (with retries on, so a failure would have kept its trace) all passed.
  4. layout-sync-authoritative "REST snapshot exactly mirrors a visible-UI-driven layout"

    • Cause (test): the drag-and-drop library ignores clicks for 50 ms after a drag ends. A click right after the drop was lost.
    • Fix: wait on a page timer that fires after the library's timer.
    • After: 90/90 on cloud.
  5. unified-agent-names-cross-mode "tab source survives focus swap move and close"

    • Cause (test): the rename helper resubmitted while the first rename was still in flight, which closed the editor under the second Enter.
    • Fix: resubmit only when the rename error is shown, as the pane-header helper already does.
    • After: 90/90 on cloud.

Not fixed here (noted for follow-up)

  • In the browser, a send that the server refuses with a session-wide SESSION_RESERVED is never resent. The message stays on screen as sent. Item 6 removes the cause seen in these tests, but any other short-lived claim would lose a message the same way.
  • The OpenCode test server picks a free port, releases it, and then uses it. With 30 test processes running at once, one test's health check was answered by another test's server.
  • Several OpenCode test processes using /tmp as their working directory compete for the /tmp/.opencode lock.

All changed crates pass cargo fmt --check and clippy -D warnings, and the branch passed the pre-push hook. The branch merges the current origin/main.

🤖 Generated with Claude Code

danshapiro and others added 15 commits October 22, 2026 21:18
… after the arm

The deferred-recheck flush releases the book lock (making the arm
visible) before its scoped metadata.json mark drains into mark_dirty,
and the mark's re-parse then publishes from a background refresh; a
snapshot taken while that sweep holds the refresh lock serves the
previous generation. The test read the index exactly once right after
observing the arm, so it failed whenever it won that window (2/20 on a
quiet machine, 13/30 under ~800 load).

Await the row under the same 5 s budget as the arm, and check the
no-discover condition after the row is visible, so it now also proves
the row itself arrived without a full provider discover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…50 ms waits

opencode_handoff_stop_waits_for_a_mid_dispatch_rest_drive_under_the_gate
released the parked REST drive with Notify::notify_waiters() after the
retained-turn abort row plus a 100 ms sleep and a 250 ms "not done yet"
window. notify_waiters() wakes only already-registered waiters, so a
drive that had not reached the park within ~350 ms (the abort answer is
held >= 25 ms by the fake serve, and the box was loaded) missed the
release and parked forever holding the dispatch gate; the handoff's stop
then queued behind the gate until the runner reported REAP_TIMEOUT.
Injecting a 400 ms delay before the park reproduces it every time.

The R4-2 seam is now a RestTurnTestPause handshake with stored-permit
notifies: the drive signals `parked`, waits for `release`; the lifecycle
quiesce paths signal `quiesce_blocked` when they find the gate held. The
test waits for the drive to park, then for the stop to queue on the gate
(failing at once if the handoff completes first), then releases. With
the quiesce gate removed the test now fails immediately with the R4-2
message instead of depending on a 250 ms window.

The prompt-POST await and the ordering proof also matched drive #1's
earlier POST row, so they proved nothing about drive #2; they now count
prompt rows and use drive #2's (the second). Failure messages carry the
drive's response body and the audit rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ss HOME

locate_transcript_selected_answers_root_path_and_cwd_under_config_dir
removed HOME (with CLAUDE_CONFIG_DIR and CLAUDE_HOME) under the claude
env lock. Since the store root moved to std::env::home_dir() (passwd
fallback on POSIX, USERPROFILE on Windows) that removal no longer changes
the candidates the test exercises, but HOME is process-global and the
OpenCode home harness (run_with_opencode_home) reads it without that lock
before and after its child run and asserts it is unchanged. Running
an_opencode_placeholder_handoff_resolves_the_durable_owner_and_completes
next to this test failed 20/30 (HOME None at start, Some at end).

Only the claude-specific roots are cleared now; HOME is never touched.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er binding write

Five fresh-create kill tests (a_sidecar_exit_after_the_live_verdict_
never_gets_its_stamp_restored among them) waited for
sink.was_recorded(durable id) and then read or took the retained
ownership stamp. The adoption writes the identity binding BEFORE it
commits ownership (ext r27 F4), and the commit plus the stamp insertion
are one critical section after it, so a poll landing between the two
saw the binding with no stamp yet: "the create's stamp" /
"the create's commit retained the release stamp" panics (1/20 on a quiet
machine), and the no-stamp test silently ran its kill while the create's
claim still held the key. A 60 ms delay injected before the commit fails
four of them every time.

A shared await_create_adoption_committed helper now waits for the stamp
(the real condition, which implies Live) and then pins the binding-first
order. Test-only change in claude.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…to the lease kill sweep

The expired-lease tree kill (kill_and_confirm_tree_dead) and its
ownership scan read /proc/<pid>/{stat,status,environ} for exactly the
processes they are killing. When a member exits and is reaped between
the open and the read, procfs answers ESRCH on the open descriptor
(verified: stat and status reads both return errno 3). Only NotFound was
treated as gone, so ESRCH became "unavailable evidence": the sweep
reported the kill unconfirmed, the lease stayed held closed and the
contending create was refused SESSION_RESERVED although the tree was
dead. In production the tokio reaper runs on another worker thread, so
the sidecar's reap can land inside that window.

proc_entry_gone() now classifies NotFound and ESRCH alike for the
start-time read, the scan's status/environ reads and the ancestry walk.
The start-time read is split at the open so the regression test can
reap a real child between the open and the read; it fails with
Err(()) under the old classification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s fast on a refusal

expired_lease_kill_sweeps_the_sidecar_tree_before_release left a
detached `sleep 300` running after every run: FAKE_SIDECAR_SPAWN_GRANDCHILD
was still set when client B's replacement sidecar spawned, so B's sidecar
started its own decoy that nothing ever kills (confirmed in the sandbox:
one orphaned `sleep 300` reparented to pid 1 after a passing run). B's
sidecar also inherited the 5 s create delay meant only for A, spending a
quarter of the 20 s budget for B's `created` on an artificial wait.

The knobs are now removed once A's sidecar has spawned with them, and the
env guard's Drop SIGKILLs any logged decoy still alive (pidfd-pinned and
matched on this env's unique request-log path in /proc/<pid>/environ), so
a failing run cleans up too. The test now waits for B's create to settle
either way and fails at once with the create.failed frame instead of
timing out on a created that never comes, then asserts the decoy is
already dead when B's created lands (the release follows the sweep's
confirmation) instead of polling for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…clined

After "Reopen as Codex CLI" the desktop pane is switched onto the
handoff's new terminal by the handoff's HTTP answer, while the session's
runtime-owner record is updated by a separate WS frame. When the answer
is folded first, TerminalView mounts while the record still names the
fresh-agent runtime, the attach gate declines ("owned by a fresh-agent
runtime"), and nothing re-fires the lifecycle effect when the committed
frame then names this terminal: the pane never attaches. Cloud
diagnostics of handoff-two-device-rust "codex: reopen-as-CLI ..." (8 of
20 runs failing) showed exactly that: the server terminal printed the
resume marker but reported hasClients:false, and the client had the
buffer registered with nothing in it.

TerminalView now remembers the terminal id the gate declined and, once
the committed owner record names that same terminal, bumps the pane's
reconcileEpoch (the lifecycle effect's re-fire signal) so the attach
runs. A vacant record or another terminal never re-attaches the declined
handle, so the delta F1 dead-handle relaunch hazard stays closed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e typing /redo

"codex: undo-to-here reverts in place; /redo is refused with the codex
copy" failed 5 of 20 cloud runs (and in other agents' runs) with the
"Redo is not available for Codex sessions" notice never visible. The test
polled the REST snapshot for the reverted thread and then typed /redo,
but the pane processes its own rolledBack ack later: the ack refills the
composer with the removed prompt and posts the undo-refill notice, which
replaces the redo notice when it lands after the typed /redo.

The test now waits for the refill ('codex turn one' back in the
composer) — the same gate the claude /undo test uses — before typing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fore selecting a tab

"REST snapshot exactly mirrors a visible-UI-driven layout" failed in
cloud runs with activeTabId never becoming the clicked "Alpha" tab. The
test clicks Alpha as soon as the drag reorder shows in the store, but
dnd-kit's pointer sensor keeps a capture-phase click blocker on the
document until a 50 ms timer after the drag ends; a click inside that
window is swallowed. Clicking right after the drop reproduces it locally
(1 of 3 runs, same symptom).

The test now resolves a page timer of that delay, scheduled after the
drop, before clicking: it fires after dnd-kit's own timer, so the blocker
is gone (8 of 8 local runs pass even when clicking right after the drop).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-flight

"tab source survives focus swap move and close" failed in cloud runs
with locator.press timing out on a detached rename input. The
session-owned tab rename keeps its editor open while the request runs;
renameThroughTabEdit resubmitted whenever the input was still visible
after a backoff, so the healthy first request could close the editor
between the visibility check and the resubmit's Enter, leaving the press
waiting 30 s on an input that never came back.

It now resubmits only when the tab shows its rename error (the role=alert
sibling a 409 conflict or 429 refusal renders while keeping the editor
open), the discipline commitPaneRenameEditor already follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…not the alias

an_adopt_over_a_live_runtime_commits_the_owner and
a_delayed_kill_after_a_handoff_never_fabricates_a_foreign_owner_claim
waited for the create's cli_index alias and then took the retained
ownership stamp. The adoption publishes that alias even earlier than the
binding write the earlier fix (958cf92) moved off, so a poll landing
before the commit saw the alias with the claim still Starting: a panic
on "the create's stamp", or the delayed kill answering STALE_CLAIM
instead of FOREIGN_OWNER. a_stale_commit_tears_down_and_abandons_after_
the_in_operation_binding armed its pre-commit park without waiting for
the first create's commit at all, so that adoption could park there too
and recover_stale_starts saw two claims instead of one.

A 60 ms delay injected before the adoption's commit failed all three
every time. The stamp wait is now its own helper
(await_create_ownership_committed) and all three use it;
await_create_adoption_committed keeps the binding-first check on top of
it. With the delay injected, all 201 claude unit tests pass; without it,
all eight tests that use the two helpers passed 40/40 alongside 200 busy
loops (load average about 190). Test-only change in claude.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ep can end it

expired_lease_kill_sweeps_the_sidecar_tree_before_release passed with
the expired-lease sweep removed (kill_and_confirm_tree_dead's step 3
replaced by `true`), on the host and in the sandbox. When the sweep
kills A's sidecar mid-create, the claude lane's own failure arm sends
SIGTERM to every process carrying the sidecar's ownership tag, and the
decoy `sleep 300` died from that instead. So the assertion that the
decoy is already dead at B's created did not show that the release
waited for the sweep.

The decoy now ignores SIGTERM (`trap "" TERM; exec sleep 300`; an
ignored signal stays ignored across exec), and the fake sidecar logs it
only after the shell has set that up. The lane's one SIGTERM no longer
ends it; only the sweep's SIGKILL escalation does. With the sweep
removed the test now fails ("grandchild survived B's convergence"), and
it still leaves no process behind either way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arts no daemon

The e2e "[freshopencode] pending rename survives a zero-turn restart"
failed now and then because the first message sent after the server
restart was lost. After a restart the client's reconnect attach names
the pane's never-materialized freshopencode-* placeholder, which the new
process has never seen. The attach went to resume_durable_session, which
claimed the placeholder's ownership key and then called get_session,
and get_session cold-starts the shared opencode daemon only to get a 404
(opencode never knows a placeholder id). The pane's reconcile re-create
and the user's first send landed inside that cold start: the send's
materialize claim on the same key was refused Blocked and answered
SESSION_RESERVED, the client's reservation re-drive re-sent the attach
rather than the message, and the pane stayed on its placeholder. Two
retained cloud traces of the failure show exactly that: the send's
checkpoint ran, the status went running then idle within 300 ms, the
local echo stayed, and no materialization or error banner followed.

resume_durable_session now answers NotFound for a placeholder id before
any claim, lease or daemon start (its doc already said it only ever
resumes durable ses_* ids). The attach still answers lost, now at once.

Regression test an_untracked_placeholder_attach_never_holds_the_key_
across_a_daemon_cold_start holds the daemon's health probe closed, runs
the reconnect attach, the re-create and the first send inside it, and
fails on the old code with exactly the e2e symptom (SESSION_RESERVED,
then INVALID_SESSION_ID, no materialization).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@danshapiro
danshapiro merged commit 4e6fda4 into main Oct 23, 2026
7 checks passed
danshapiro added a commit that referenced this pull request Oct 23, 2026
Brings in #864 (Windows/Linux tree-guard fixes and the shared
process_probe test support), #866/#868/#869 (flaky-test fixes) and
#867 (managed helper create retry) on top of core PR-C.

No textual conflicts. Seven files changed on both sides and merged
cleanly; each was checked by hand:
- claude.rs: main's await_create_ownership_committed /
  await_create_adoption_committed helpers and their call sites land in
  PR-C's slimmed test module unchanged (12 uses, same as main).
- claude_snapshot.rs: main's "never clear HOME" fix applies to the
  locate_transcript test; PR-C's genesis tests do not touch HOME.
- claude/teardown_tests.rs, lib.rs: main's zombie-aware ProcessProbe
  module and the REST turn test-pause handshake sit beside PR-C's
  module reshuffle.
- opencode_ws.rs, session_handoff/tests.rs: independent hunks.
- fresh-agent-rollback-rust.spec.ts: main's wait for the codex undo
  refill before typing /redo applies inside PR-C's reworked spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant