Skip to content

fix(code-index): stop reading a held projection slot as work - #3077

Open
devin-ai-integration[bot] wants to merge 12 commits into
masterfrom
devin/1791129523-master-flakes-3041
Open

devin-ai-integration[bot] wants to merge 12 commits into
masterfrom
devin/1791129523-master-flakes-3041

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • text_projection_needs_work treated any contended slot lock as work in progress. Another probe or a no-op advance over ready owners can also hold that lock. When these operations collided, the published text projection recorded a worker continuation that did not exist. Freshness then remained Verifying and Pending at the serving swap. This caused the reconcile failure in test: four intermittent master failures (reachability, reconcile, git_watch, dashboard) #3041.
  • This change also fixes the git-watch linked-worktree debounce failure, which was a test-clock race.

Fixes #3041

Cause

#3041 lists four intermittent failures on master:

  • Reconcile freshness, (Some(Verifying), Pending). This failed 59 of 3,000 local stress runs. Instrumentation of every pending-wake writer found two late wake sources:
    1. The test's request_complete_generation demand sometimes arrived after the first pass claimed its wake. This queues a real follow-up pass, so Verifying is correct. The test now holds admission until it posts the demand.
    2. Every other failure came from the published projection task's note_worker_continuation. The task called it because the slot's try_lock returned WouldBlock while the slot was idle. This production bug is fixed here.
  • Git-watch debounce (tests.rs:643). The test used start_paused time. The paused clock advanced beyond the test's deadlines while a live notify wake remained in flight. The watcher's copies of the marker writes could also produce a second drain.
  • Reachability. PR fix(code-index): serve a complete seat that lands after the read misses #3044 already addresses this failure. With fix(code-index): serve a complete seat that lands after the read misses #3044 and this branch, the test passed all 80 stress runs. The issue reported 5 failures in 40 runs on the earlier master.
  • Dashboard ChunkLoadError. RouteChunkBoundary.dom.test.tsx deliberately rejects with this error. The message appears whenever that suite runs, so it does not identify the failing test. The suite passed all 15 local runs. This PR makes no change for it.

Changes

  • code_index_scheduler/serving.rs: CodeTextProjectionStateV1 now stores the slot's idle state in the slot_idle atomic. replace_slot writes the atomic on every slot transition. The probe reads the atomic without taking the lock, so it never queues behind an advance's slice.
  • tests/serving.rs: adds a_held_idle_projection_slot_is_not_text_projection_work. The test holds an idle slot's lock and asserts that no work is reported. It fails before the fix at serving.rs:908 and passes after it.
  • tests/reconcile.rs: ready_wait_ends_only_after_the_graph_tail_seats_the_generation holds admission until the complete-generation demand is posted.
  • git_watch/tests.rs, state.rs, and git_watch.rs: the linked-worktree test now uses real time. It waits for the test-only operation_held signal, which means debounce observed InFlight, and accepts >= 1 drains on release instead of requiring an exact count.

Test plan

  • bash scripts/require-exact-test.sh cargo test -p tracedecay-code-index-runtime --lib <test> -- --exact passes for each of the three tests above after the rebase on origin/master
  • Stress results:
    • reconcile ready-wait: 0 failures in 3,000 runs, compared with 59 before
    • git-watch linked hold: 0 failures in 2,000 runs
    • reachability: 0 failures in 80 runs
  • cargo test -p tracedecay-code-index-runtime --lib -- code_index_scheduler::tests::serving code_index_scheduler::tests::reconcile git_watch: 280 passed
  • cargo fmt --all -- --check and cargo clippy -p tracedecay-code-index-runtime --all-targets -- -D warnings are clean
  • python3 scripts/linux-test-partitions.py check and node scripts/lint-commit-range.mjs --repository . origin/master HEAD pass
  • Full workspace cargo nextest run: left to CI

Checklist

  • CHANGELOG.md updated (not updated; release-please generates it from commits)
  • No secrets, credentials, or .env files included
  • Breaking changes documented (none)

Link to Devin session: https://app.devin.ai/sessions/e8f223a3ec5c476885a5d8903bafa4c2
Open in Devin Desktop: https://app.devin.ai/desktop/session/e8f223a3ec5c476885a5d8903bafa4c2?variant=devin
Requested by: @ScriptedAlchemy

Link to Devin session: https://app.devin.ai/sessions/d1c43c1ea5264a0e9398d1b722b54447
Open in Devin Desktop: https://app.devin.ai/desktop/session/d1c43c1ea5264a0e9398d1b722b54447?variant=devin


Devin Review

ScriptedAlchemy and others added 2 commits October 4, 2026 15:58
The paused clock auto-advanced past the test's deadlines while a live
notify wake was in flight, and the watcher's own copies of the marker
writes could add a second drain. Wait on the debounce observing the
operation instead, and assert the release rather than an exact count.

Refs #3041

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
text_projection_needs_work treated any contended slot lock as work in
progress. Another probe or a no-op advance over ready owners holds the
lock too, so the published projection stamped a phantom continuation
and freshness stayed Verifying at the serving swap. Mirror the slot's
idle state in an atomic written on every transition and read that.

The graph-tail ready test also holds admission until the
complete-generation demand is posted, so the demand joins the first
pass instead of queueing a real follow-up.

Fixes #3041

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@changeset-bot

changeset-bot Bot commented Oct 4, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 4b550af

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: four intermittent master failures (reachability, reconcile, git_watch, dashboard)

1 participant