Skip to content

fix(private-fs): admit a free lock after its admission deadline - #3082

Open
devin-ai-integration[bot] wants to merge 9 commits into
masterfrom
devin/1791139889-lock-admission-attempt
Open

devin-ai-integration[bot] wants to merge 9 commits into
masterfrom
devin/1791139889-lock-admission-attempt

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Bounded lock admission (tracedecay_private_fs::lock_until / lock_shared_until) now always attempts the lock once and times out only when a holder outlasts the deadline. Before this change it also timed out on a free lock whenever the caller was descheduled past its deadline.
  • Deletes the delivery spool's try_lock-first special case in HookDeliveryReceiptSpoolV1::open. It existed only to get this behavior, which lock_until now provides.
  • Fixes the fail-if-flaky spool::tests::a_writer_reusing_its_acquisition_timestamp_never_expires_mid_session (AdmissionTimedOut at spool/tests.rs:1587, ARM core-contracts run 37213198052) and every sibling spool test with the same exposure.

Closes #3081

Motivation

admit_until counted the caller's own latency against the admission deadline in two places:

  1. It returned TimedOut before its first try_lock if Instant::now() >= deadline.
  2. It released a lock it had just acquired if the deadline passed during the attempt.

HookSpoolV1::acknowledge → publish_meta → without_lease drops the writer lease across durability barriers and reacquires it with acquire_lease_bounded(.., Some(config.writer_lease())). The spool test fixture sets writer_lease_micros: 100, so that reacquire has a 100µs budget. A 100µs stall on a loaded runner (CPU-quota throttling, preemption) between computing the deadline and checking it fails a single-writer test that has no contention at all. Every test in spool/tests.rs that settles or reclaims under config() has the same wall-clock dependency, so fixing the test alone would leave its siblings flaky.

This is a product bug, not only a test problem. lease.rs documents the budget as bounding "only the lock wait", but the implementation also counted time the caller spent not waiting. In production, a stall longer than the budget refuses a free lock. That stall can be a drain thread descheduled across its 5s reacquire, or a spawn_blocking caller (pr_tracking/worktrees.rs, source-edit/journal.rs) whose deadline was computed before the blocking pool ran it. After a failed reacquire, the drain's spool handle refuses all further mutations.

Decision

The budget means "how long to wait for a holder", not "how much wall-clock the caller may have consumed". I considered and rejected two alternatives:

  • Raise the fixture lease or the reacquire budget. This hides the defect, and any stall longer than the new budget brings it back.
  • Keep refusing after the deadline and only drop the post-acquire release. This still refuses a free lock without trying it, so the same flake remains with a smaller window.

This intentionally reverses the old "an exhausted budget never admits, even when the lease is free" assertions, one in spool/tests.rs and one in delivery_spool.rs. No production caller depends on a zero budget refusing a free lock: open(.., ZERO) already meant "try once" through the special case deleted here. The new contract is the same for every caller: a zero or spent budget takes a free lock, and a held lock refuses it without waiting.

Changes

  • crates/tracedecay-private-fs/src/lock_admission.rs: admit_until loop is now try_lock → Ok admits, WouldBlock sleeps until the deadline, then TimedOut. Added unit tests.
  • crates/tracedecay-hooks/src/delivery_spool.rs: HookDeliveryReceiptSpoolV1::open calls lock_until directly. Its test now proves a zero-budget writer takes the free staging lease and stages behind the held publish lock.
  • crates/tracedecay-hooks/src/spool/tests.rs: new regression test a_writer_whose_budget_is_spent_takes_a_free_lease_but_never_a_held_one, which covers open, append, and acknowledge through the settlement lease reacquire. Removed the obsolete zero-budget assertion from bounded_writer_admission_preserves_failfast_and_times_out_without_mutation.

Test plan

Red before the fix, green after. Each test was run with the product change reverted and then restored:

bash scripts/require-exact-test.sh cargo test -p tracedecay-private-fs --lib lock_admission::tests::a_free_lock_is_admitted_after_the_deadline_has_passed -- --exact   # FAILED -> ok
bash scripts/require-exact-test.sh cargo test -p tracedecay-private-fs --lib lock_admission::tests::a_held_lock_times_out_once_the_deadline_passes -- --exact            # FAILED (free-after-unlock assertion) -> ok
bash scripts/require-exact-test.sh cargo test -p tracedecay-hooks --lib spool::tests::a_writer_whose_budget_is_spent_takes_a_free_lease_but_never_a_held_one -- --exact # FAILED -> ok
bash scripts/require-exact-test.sh cargo test -p tracedecay-hooks --lib delivery_spool::tests::a_writer_behind_a_held_owner_stages_its_receipt_for_adoption -- --exact  # FAILED -> ok

Reproduction of the original CI flake. I simulated CPU-quota throttling by sending SIGSTOP/SIGCONT to the test process in a loop (0.5ms paused, 0.3ms running) and ran spool::tests::a_writer_reusing_its_acquisition_timestamp_never_expires_mid_session repeatedly:

  • master binary: 3/300 failed with called `Result::unwrap()` on an `Err` value: AdmissionTimedOut at spool/tests.rs:1587, the same line and error as CI.
  • this branch: 0/600 failed.

Suites and lint:

  • cargo test -p tracedecay-private-fs -p tracedecay-hooks: all pass (hooks lib 114, private-fs 33, plus integration targets)
  • cargo test -p tracedecay-session-memory -p tracedecay-source-edit (other lock_until callers): 305 + 89 pass
  • cargo clippy -p tracedecay-private-fs -p tracedecay-hooks --all-targets -- -D warnings: clean
  • cargo fmt --all -- --check: clean
  • python3 scripts/linux-test-partitions.py check: ok (no new test target)
  • node scripts/lint-commit-range.mjs --repository . origin/master HEAD: ok

Checklist

  • CHANGELOG.md updated: not updated; release commits generate it
  • No secrets, credentials, or .env files included
  • Breaking changes documented: the zero/spent-budget admission contract change is described above

Link to Devin session: https://app.devin.ai/sessions/e7930fbb0cc24830a620910172ed8da7
Open in Devin Desktop: https://app.devin.ai/desktop/session/e7930fbb0cc24830a620910172ed8da7?variant=devin
Requested by: @ScriptedAlchemy


Devin Review

Bounded lock admission counted the caller's own latency against the
deadline: it refused before its first try_lock once the deadline had
passed, and it released a lock it had just taken if the deadline passed
during the attempt. A thread descheduled past its budget therefore
timed out on a free lock.

The hook spool's settlement reacquires its writer lease with the lease
duration as its budget, which is 100us in the spool test fixture, so a
short stall on a loaded runner failed single-writer spool tests with
AdmissionTimedOut (#3081).

Admission now always attempts the lock once and times out only when a
holder outlasts the deadline. The delivery spool's try_lock-first
special case, which existed to get that behavior, is deleted.

Closes #3081

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@changeset-bot

changeset-bot Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 927c392

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

devin-ai-integration[bot]

This comment was marked as resolved.

ScriptedAlchemy and others added 8 commits October 4, 2026 19:05
A waiter that saw WouldBlock slept up to the deadline and then retried
unconditionally, so a holder releasing after the deadline still admitted
it. Check the deadline after each contention sleep; the first attempt
stays unconditional so a free lock is still taken past the deadline.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: spool tests time out uncontended lock admission under load

1 participant