test(sessions): drive run_with_semaphore in the detach test - #2971
Merged
Merged
Conversation
The test reimplemented the semaphore and poll loop inline, so it never exercised run_with_semaphore and passed with the permit dropped early. It now calls run_with_semaphore and retries when the wall-clock deadline expires before the closure is spawned, asserting that such an attempt leaks no permit.
|
Contributor
There was a problem hiding this comment.
🔍 Devin Review: 1 flag
Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes one test that never exercised its subject. It also closes three load-flake issues whose listed tests are already fixed on master, with evidence for each.
#2809
deadline_detaches_without_leaking_the_permitRoot cause.
BoundedGitControluses a wall-clockInstantdeadline (25 ms in the test). On a loaded host,run_with_semaphorecan hit that deadline at its pre-spawncontrol.check()before it spawns the closure. The closure'sstarted_txwas then dropped, andstarted_rx.await.unwrap()panicked withRecvError. #2852 removed that panic by reimplementing the semaphore and poll loop inline in the test. After that change the test never calledrun_with_semaphore, so it passed even when the product released the permit early.Fix. The test calls
run_with_semaphoreagain. If the deadline expires before the closure starts, the test asserts that the attempt returnedCommandTimedOutand leaked no permit, then retries. Once the closure is running, the test asserts that the detached closure holds the permit until it is released.Regression test.
deadline_detaches_without_leaking_the_permit.let _permit = permit;todrop(permit);inblocking.rs. Master's test still passed. This test fails withleft: 1 right: 0attests.rs:168.started_rx.await, the pre-fix(hosts,sessions): Hermes pycache cleanup and session test flakes #2852 test fails withRecvError(()). This test passes, taking the retry path once.tracedecay-sessionslib suite at load 36 to 678.Fixes #2809
#2811
prepared_generation_uses_bounded_parallelism_and_retained_bytesAlready fixed on master. 0f4bd88 (#2852) replaced the
peak() > 8assertion, which depended on scheduling, with apath_count-wide barrier that every page build must pass before it builds. fa6678c (#2929) made those gates one-shot so a rebuild cannot park in a barrier with no partners. The build guard is entered before the barrier, so a passing barrier already proves that allpath_countbuilds overlapped.Evidence. The test passed in 48 of 48 runs of the sessions lib suite at load 36 to 678, and passes again on this branch.
Fixes #2811
#2667 six tests that fail under host load
All six are fixed on master. The evidence comes from debug builds of master at 08f8fac.
production_hook_ingest_reads_only_the_pinned_transcript_homewas fixed by fix: unwedge refresh joins and scope rollback under load #2677. It passed 64 of 64 runs.maintenance_reclaims_a_removed_linked_worktree_and_the_text_artifact_only_it_namedwas fixed by fix: unwedge refresh joins and scope rollback under load #2677. It passed in 11 of 11 full lib-suite runs at load 50 to 355.an_identifier_named_in_the_task_leads_plan_and_explore_contextwas fixed by test: wait on readiness instead of racing host load #2678. It passed 64 of 64 runs.unenrolled_directory_session_serves_user_settingswas fixed by test: wait on readiness instead of racing host load #2678. It passed 16 of 16 runs with 8 copies in parallel.failed_cold_mount_graph_replay_preserves_retained_text_generationwas fixed by test: wait on readiness instead of racing host load #2678. It passed 11 of 11 full-suite runs.resume_attempts_refuses_a_live_holder_and_reports_the_lost_attemptwas fixed by test: wait on readiness instead of racing host load #2678 and test(work): resume only after the restart's startup fence lands #2688. It passed 64 of 64 runs.The 64 runs of each mcp_suite test ran with 8 to 24 copies in parallel, some of them alongside full lib-suite copies.
Fixes #2667
#2650 lib tests that fail under full-suite load
All ten are fixed on master or no longer reproduce. Each passed in all 11 full lib-suite runs (2 or 3 concurrent copies, load 50 to 355).
many_slow_initialize_roots_share_one_discovery_budget,client_identity_startup_replays_retained_profile_receipts,owned_project_replay_worker_continues_past_one_bounded_batch,one_shot_tool_call_allows_long_response_while_daemon_stays_liveandmounted_code_generation_retention_continues_capped_segment_reclamationwere fixed by test(daemon): wait on readiness in load-sensitive lib tests #2674.first_advisory_cycle_after_a_reopen_answers_without_a_retrywas fixed by fix(feedback): wait for a reopened generation's source proof #2668.session_temporal_benchmarktests were fixed by test(sessions): drop invented wall clocks from session tests #2676.preserved_profile_lcm_discovery_converges_without_blocking_retrievalwas fixed by c0fe494 (feat(sessions): publish typed LCM convergence in status #2717). The fleet analysis on this issue left it open because it needed a truthful quiescence signal in place of the test's 2-minuteCONVERGENCE_WAIT. feat(sessions): publish typed LCM convergence in status #2717 published a typedconvergedstate, and the test now waits on it with no deadline. It passed 12 of 12 targeted runs alongside two full-suite copies at load 50 to 88.a_deferred_discovery_converges_on_the_next_requestno longer reproduces. It failed once in the original 50 runs. It passed 12 of 12 targeted runs under full-suite load, plus 6 runs pinned to one or two cores shared with 6 to 12 busy-loop processes.Fixes #2650
Related
Looping the lib suite surfaced four other load flakes that neither umbrella lists. They are tracked in #2969. The most frequent is
cancelled_large_candidate_search_stops_before_the_next_batch, which failed in 5 of 11 runs.Landing verification
Rebased on 05fe6b9.
let _permit = permit;changed todrop(permit);inblocking.rs.deadline_detaches_without_leaking_the_permitfails withleft: 1 right: 0attests.rs:168. So doesdetached_task_holds_capacity_until_its_closure_finishes.cargo test -p tracedecay-sessions --lib: 542 passed.-D warnings --all-targets, Windows gnucargo check --all-targetson tracedecay-sessions, andcargo fmt --checkare clean.The test still loops when the 25 ms wall-clock deadline expires before the closure spawns. Each pass of the loop asserts the outcome of the branch it took:
CommandTimedOutwith the permit returned, or a detached closure that holds the permit until it is released. So the loop cannot hide a leak.BoundedGitControlreadsstd::time::Instant, and no paused clock can pin that branch.