test: tracedecay lib tests that fail only under full-suite load

[ScriptedAlchemy]

What happens

These tracedecay lib tests fail only under load. Each one passes when run alone. I found them while looping the lib suite for #2513 and did not investigate them.

How it was reproduced

• Binary: master d842ec3, cargo test -p tracedecay --features test-transport --lib --no-run.
• Load: the whole lib suite, 772 tests, run with --test-threads=64. Two to four copies ran at once on a 96-core host that was shared with other agents' builds.
• 50 runs in total.

Test	Failed runs (of 50)
mcp::server::routing::tests::many_slow_initialize_roots_share_one_discovery_budget	14
daemon::production_harness::advisory_cycle_language_journey_test::first_advisory_cycle_after_a_reopen_answers_without_a_retry	13
daemon::tests::replay::client_identity_startup_replays_retained_profile_receipts	10
mcp::server::host_admission_tests::owned_project_replay_worker_continues_past_one_bounded_batch	7
daemon::production_harness::lcm_preserved_profile_journey_test::preserved_profile_lcm_discovery_converges_without_blocking_retrieval	7
daemon::tests::lifecycle::one_shot_tool_call_allows_long_response_while_daemon_stays_live	3
daemon::production_harness::generation_retention_test::mounted_code_generation_retention_continues_capped_segment_reclamation	3
session_temporal_benchmark::tests::fresh_benchmark_db_provisions_key_for_rank_and_hydration	2
session_temporal_benchmark::tests::fixture_refresh_persists_progress_before_measurement	2
daemon::tests::route_discovery::a_deferred_discovery_converges_on_the_next_request	1

corrupt_graph_restart_rebuilds_from_the_sealed_segments_without_decoding also failed in 10 of the 50 runs. #2633 already tracks it, so it is not listed here.

Known cause for one test

many_slow_initialize_roots_share_one_discovery_budget compares wall-clock time: it asserts elapsed < 3s across three 1.5s injected probes (crates/tracedecay/src/mcp/server/routing.rs). Under CPU contention the budget check fails before the deadline logic it is meant to test gets a chance to run.

I did not capture the panic text for the other tests. The run above reproduces them.

Repro

B=$(cargo test -p tracedecay --features test-transport --lib --no-run 2>&1 | sed -n 's/.*(\(target[^)]*\)).*/\1/p')
cd crates/tracedecay
for i in 1 2 3; do "../../$B" --test-threads=64 & done; wait

[ScriptedAlchemy]

Claimed by fleet slug load-flake-sweep (looping lib, mcp_suite, session_suite, daemon_suite, application_suite under fleet load; fixes grouped by area).

[ScriptedAlchemy]

Also seen under the same load: two full lib-suite runs hung for more than an hour (one on d842ec3 plus test-only changes, one on 8d4fd5e plus test-only changes). In both, the single remaining libtest thread was named daemon::store_r…, so the hang is in one of the daemon::store_runtime_tests::* tests. The thread name is truncated, so I can't tell which one. Hauler lanes should wrap each loop iteration in timeout.

[ScriptedAlchemy]

Findings from load-flake-sweep-harness (the three lib-suite flakes delegated from this sweep). Whole tracedecay --lib suite, default parallelism, debug build, 96 cores.

test	root cause	change	before (master)	after	PR / merge
advisory_cycle_language_journey_test::first_advisory_cycle_after_a_reopen_answers_without_a_retry	Production. A reopen serves its restored text generation before the worker's first pass proves the checkout matches it; in that window latest_feedback_generation_for_scope is None, so the placeholder advisory owner's mount() answered Answers and the request got feedback.advisory-cycle.unavailable immediately. Holding that proof with the existing retained-recovery gate reproduced it 10/10 alone (refusal in 1.6 ms while the harness already reported ready).	Placeholder reports Mounting while the scope holds a retained text owner, so the request waits for the deferred mount within its own deadline.	8 / 27 runs (load 56-233)	0 / 22 (load 72-159)	#2668, 03ba683fbc
code_index_runtime_graph_activation_tests::corrupt_graph_restart_rebuilds_from_the_sealed_segments_without_decoding	Test. The simulated restart skipped the daemon's terminal order: the seeded publication's retained staging-release sweep still held the project graph store, the restarted runtime's sealed graph publication failed with database file is locked by another process: …/tracedecay.grafeo, and the scheduler took its documented fallback decode (sealed_decode_count 1).	Join shutdown_terminal_tasks() before the reconciliation shutdown.	2 / 27 full-suite; 5 / 30 module-filter runs	0 / 22 full-suite; 0 / 30 module-filter	#2671, 5aa5fd06b8 (closes #2633)
lcm_preserved_profile_journey_test::preserved_profile_lcm_discovery_converges_without_blocking_retrieval	Test-invented bound on serial background work; not fixed here. Every failing payload shows progress, not a stall, at different stages (raw 0 / 160 / 200 / 232 / 240; git generation absent in some, published in others; summaries 0-2 current of 80-120 pending; search stale/partial). Instrumented alone (~25-40 s total): one project refresh worker runs everything serially. Its first history pass takes ~12 s (Codex ~1.5 s, Claude observation admission ~7 s over ~120 submits, Cursor ~2 s), returns Retryable{ingest_pass_backpressured}, then about ten projection-only passes run, then a second history pass installs the 40-span Git evidence (~4.7 s), and only then does summary convergence get Admitted(Full). In a failing loaded run (load ~200), the Codex-to-Git-evidence phase took 77 s against 9-19 s alone. #843 names no convergence deadline, so the 2-minute CONVERGENCE_WAIT is the test's own.	None. A deadline-free wait needs a truthful quiescence signal for the project refresh worker, and SessionTemporalRefreshWakeState::is_idle() is not one: it also reports idle while a pass waits on the historical-work permit (worker.rs ~333-339). The other option is to make the first history pass cheaper (Claude admission is one submit per transcript write group), measured with Hotpath. Both are larger than this lane.	6 / 27	2 / 22 (runs at load 99-152; payloads again show partial progress)	open

Other failures seen in my loops (the parent's, not mine): concurrent_same_identity_worktrees_keep_exact_server_and_scheduler_bindings (every run), daemon_http_shutdown_releases_loopback_listener, client_identity_startup_replays_retained_profile_receipts, session_temporal_benchmark::tests::*, remote_tls_listener_bounds_connections_and_expires_incomplete_headers, rmcp_peer_disconnect_mid_delivery_settles_dropped_rather_than_unknown, owned_project_replay_worker_continues_past_one_bounded_batch, many_slow_initialize_roots_share_one_discovery_budget, maintenance_reclaims_a_removed_linked_worktree_and_the_text_artifact_only_it_named.

[ScriptedAlchemy]

Status from fleet slug load-flake-sweep (whole lib suite at default parallelism on the shared host, load 57-233; baseline master 35fe79e, 20 runs; post-fix 20 runs, load 72-161).

Merged:

• test(daemon): wait on readiness in load-sensitive lib tests #2674 (6aace0f): owned project replay worker (lost wakeup in the test helper), client-identity restart replay (production store shutdown before the in-process restart), liveness fake daemons that stay live, rmcp peer disconnect (explicit socket shutdown: a concurrently forked child inherits the fd), remote TLS egress (sequential peers, paused time at barrier release), empty invocation shutdown (paused clock), cancel-candidate journey (checkpoint waits on the cancellation signal), shared discovery budget (typed outcome instead of wall clock), generation retention (no invented window), plus the fix(mcp)!: carry tool problems only as structured content #2651 fallout in concurrent_same_identity_worktrees_.... All 0/20 post-fix.
• test(sessions): drop invented wall clocks from session tests #2676 (8553947): session_temporal_benchmark pair (per-request contexts instead of one 30 s deadline across 64 refreshes), 7/20 and 6/20 -> 0/20.
• Sibling lanes: fix(feedback): wait for a reopened generation's source proof #2668 (advisory-cycle reopen), test(code-index): join terminal tasks before a simulated restart #2671 (test(code-index): corrupt graph restart decodes under concurrent load #2633 corrupt-graph restart), fix(daemon): join dropped observability drains at shutdown #2665 (daemon_suite).

Still open here: store_runtime_tests::schema_convergence_is_reported_and_completes_after_admission (seed insert hits database is locked while the fixture's dropped runtime closes, 1/20) and runtime_identity::maintenance_reclaims_a_removed_linked_worktree_... (1/20), both assigned to sibling lanes. The product race behind the liveness fake-daemon failures is #2654.

[ScriptedAlchemy]

claimed by fleet slug close-all-issues/daemon-flakes