Skip to content

test(sessions): drive run_with_semaphore in the detach test - #2971

Merged
ScriptedAlchemy merged 2 commits into
masterfrom
bug/flakes-b
Oct 2, 2026
Merged

ScriptedAlchemy merged 2 commits into
masterfrom
bug/flakes-b

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

This PR fixes one test that never exercised its subject. It also closes three load-flake issues whose listed tests are already fixed on master, with evidence for each.

#2809 deadline_detaches_without_leaking_the_permit

Root cause. BoundedGitControl uses a wall-clock Instant deadline (25 ms in the test). On a loaded host, run_with_semaphore can hit that deadline at its pre-spawn control.check() before it spawns the closure. The closure's started_tx was then dropped, and started_rx.await.unwrap() panicked with RecvError. #2852 removed that panic by reimplementing the semaphore and poll loop inline in the test. After that change the test never called run_with_semaphore, so it passed even when the product released the permit early.

Fix. The test calls run_with_semaphore again. If the deadline expires before the closure starts, the test asserts that the attempt returned CommandTimedOut and leaked no permit, then retries. Once the closure is running, the test asserts that the detached closure holds the permit until it is released.

Regression test. deadline_detaches_without_leaking_the_permit.

  • I made the product release the permit early by changing let _permit = permit; to drop(permit); in blocking.rs. Master's test still passed. This test fails with left: 1 right: 0 at tests.rs:168.
  • With a 50 ms delay injected before started_rx.await, the pre-fix(hosts,sessions): Hermes pycache cleanup and session test flakes #2852 test fails with RecvError(()). This test passes, taking the retry path once.
  • It passed in 48 of 48 runs of the tracedecay-sessions lib suite at load 36 to 678.

Fixes #2809

#2811 prepared_generation_uses_bounded_parallelism_and_retained_bytes

Already fixed on master. 0f4bd88 (#2852) replaced the peak() > 8 assertion, which depended on scheduling, with a path_count-wide barrier that every page build must pass before it builds. fa6678c (#2929) made those gates one-shot so a rebuild cannot park in a barrier with no partners. The build guard is entered before the barrier, so a passing barrier already proves that all path_count builds overlapped.

Evidence. The test passed in 48 of 48 runs of the sessions lib suite at load 36 to 678, and passes again on this branch.

Fixes #2811

#2667 six tests that fail under host load

All six are fixed on master. The evidence comes from debug builds of master at 08f8fac.

The 64 runs of each mcp_suite test ran with 8 to 24 copies in parallel, some of them alongside full lib-suite copies.

Fixes #2667

#2650 lib tests that fail under full-suite load

All ten are fixed on master or no longer reproduce. Each passed in all 11 full lib-suite runs (2 or 3 concurrent copies, load 50 to 355).

Fixes #2650

Related

Looping the lib suite surfaced four other load flakes that neither umbrella lists. They are tracked in #2969. The most frequent is cancelled_large_candidate_search_stops_before_the_next_batch, which failed in 5 of 11 runs.

Landing verification

Rebased on 05fe6b9.

  • Mutation: let _permit = permit; changed to drop(permit); in blocking.rs. deadline_detaches_without_leaking_the_permit fails with left: 1 right: 0 at tests.rs:168. So does detached_task_holds_capacity_until_its_closure_finishes.
  • cargo test -p tracedecay-sessions --lib: 542 passed.
  • clippy -D warnings --all-targets, Windows gnu cargo check --all-targets on tracedecay-sessions, and cargo fmt --check are clean.

The test still loops when the 25 ms wall-clock deadline expires before the closure spawns. Each pass of the loop asserts the outcome of the branch it took: CommandTimedOut with the permit returned, or a detached closure that holds the permit until it is released. So the loop cannot hide a leak. BoundedGitControl reads std::time::Instant, and no paused clock can pin that branch.

The test reimplemented the semaphore and poll loop inline, so it never exercised run_with_semaphore and passed with the permit dropped early. It now calls run_with_semaphore and retries when the wall-clock deadline expires before the closure is spawned, asserting that such an attempt leaks no permit.
@changeset-bot

changeset-bot Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e4dbb0c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@ScriptedAlchemy ScriptedAlchemy changed the title test(sessions): drive run_with_semaphore in the deadline detach test test(sessions): drive run_with_semaphore in the detach test Oct 2, 2026
@ScriptedAlchemy
ScriptedAlchemy merged commit b01c817 into master Oct 2, 2026
19 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant