fix(private-fs): admit a free lock after its admission deadline - #3082
Open
devin-ai-integration[bot] wants to merge 9 commits into
Open
devin-ai-integration[bot] wants to merge 9 commits into
devin-ai-integration[bot] wants to merge 9 commits into
Conversation
Bounded lock admission counted the caller's own latency against the deadline: it refused before its first try_lock once the deadline had passed, and it released a lock it had just taken if the deadline passed during the attempt. A thread descheduled past its budget therefore timed out on a free lock. The hook spool's settlement reacquires its writer lease with the lease duration as its budget, which is 100us in the spool test fixture, so a short stall on a loaded runner failed single-writer spool tests with AdmissionTimedOut (#3081). Admission now always attempts the lock once and times out only when a holder outlasts the deadline. The delivery spool's try_lock-first special case, which existed to get that behavior, is deleted. Closes #3081 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
A waiter that saw WouldBlock slept up to the deadline and then retried unconditionally, so a holder releasing after the deadline still admitted it. Check the deadline after each contention sleep; the first attempt stays unconditional so a free lock is still taken past the deadline. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tracedecay_private_fs::lock_until/lock_shared_until) now always attempts the lock once and times out only when a holder outlasts the deadline. Before this change it also timed out on a free lock whenever the caller was descheduled past its deadline.try_lock-first special case inHookDeliveryReceiptSpoolV1::open. It existed only to get this behavior, whichlock_untilnow provides.spool::tests::a_writer_reusing_its_acquisition_timestamp_never_expires_mid_session(AdmissionTimedOutatspool/tests.rs:1587, ARM core-contracts run 37213198052) and every sibling spool test with the same exposure.Closes #3081
Motivation
admit_untilcounted the caller's own latency against the admission deadline in two places:TimedOutbefore its firsttry_lockifInstant::now() >= deadline.HookSpoolV1::acknowledge→publish_meta→without_leasedrops the writer lease across durability barriers and reacquires it withacquire_lease_bounded(.., Some(config.writer_lease())). The spool test fixture setswriter_lease_micros: 100, so that reacquire has a 100µs budget. A 100µs stall on a loaded runner (CPU-quota throttling, preemption) between computing the deadline and checking it fails a single-writer test that has no contention at all. Every test inspool/tests.rsthat settles or reclaims underconfig()has the same wall-clock dependency, so fixing the test alone would leave its siblings flaky.This is a product bug, not only a test problem.
lease.rsdocuments the budget as bounding "only the lock wait", but the implementation also counted time the caller spent not waiting. In production, a stall longer than the budget refuses a free lock. That stall can be a drain thread descheduled across its 5s reacquire, or aspawn_blockingcaller (pr_tracking/worktrees.rs,source-edit/journal.rs) whose deadline was computed before the blocking pool ran it. After a failed reacquire, the drain's spool handle refuses all further mutations.Decision
The budget means "how long to wait for a holder", not "how much wall-clock the caller may have consumed". I considered and rejected two alternatives:
This intentionally reverses the old "an exhausted budget never admits, even when the lease is free" assertions, one in
spool/tests.rsand one indelivery_spool.rs. No production caller depends on a zero budget refusing a free lock:open(.., ZERO)already meant "try once" through the special case deleted here. The new contract is the same for every caller: a zero or spent budget takes a free lock, and a held lock refuses it without waiting.Changes
crates/tracedecay-private-fs/src/lock_admission.rs:admit_untilloop is nowtry_lock→Okadmits,WouldBlocksleeps until the deadline, thenTimedOut. Added unit tests.crates/tracedecay-hooks/src/delivery_spool.rs:HookDeliveryReceiptSpoolV1::opencallslock_untildirectly. Its test now proves a zero-budget writer takes the free staging lease and stages behind the held publish lock.crates/tracedecay-hooks/src/spool/tests.rs: new regression testa_writer_whose_budget_is_spent_takes_a_free_lease_but_never_a_held_one, which covers open, append, and acknowledge through the settlement lease reacquire. Removed the obsolete zero-budget assertion frombounded_writer_admission_preserves_failfast_and_times_out_without_mutation.Test plan
Red before the fix, green after. Each test was run with the product change reverted and then restored:
Reproduction of the original CI flake. I simulated CPU-quota throttling by sending SIGSTOP/SIGCONT to the test process in a loop (0.5ms paused, 0.3ms running) and ran
spool::tests::a_writer_reusing_its_acquisition_timestamp_never_expires_mid_sessionrepeatedly:called `Result::unwrap()` on an `Err` value: AdmissionTimedOutatspool/tests.rs:1587, the same line and error as CI.Suites and lint:
cargo test -p tracedecay-private-fs -p tracedecay-hooks: all pass (hooks lib 114, private-fs 33, plus integration targets)cargo test -p tracedecay-session-memory -p tracedecay-source-edit(otherlock_untilcallers): 305 + 89 passcargo clippy -p tracedecay-private-fs -p tracedecay-hooks --all-targets -- -D warnings: cleancargo fmt --all -- --check: cleanpython3 scripts/linux-test-partitions.py check: ok (no new test target)node scripts/lint-commit-range.mjs --repository . origin/master HEAD: okChecklist
CHANGELOG.mdupdated: not updated; release commits generate it.envfiles includedLink to Devin session: https://app.devin.ai/sessions/e7930fbb0cc24830a620910172ed8da7
Open in Devin Desktop: https://app.devin.ai/desktop/session/e7930fbb0cc24830a620910172ed8da7?variant=devin
Requested by: @ScriptedAlchemy