Skip to content

fix: o!rdr render poll deadlocks on finished renders - #1138

Merged
MaxOhn merged 1 commit into
MaxOhn:mainfrom
Philenst:fix/ordr-poll-deadlock
Sep 26, 2026
Merged

MaxOhn merged 1 commit into
MaxOhn:mainfrom
Philenst:fix/ordr-poll-deadlock

Conversation

@Philenst

@Philenst Philenst commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

poll_finished_renders acquires a read lock on the render's shard and only drops it at the end of the loop body, but then calls senders.own(render_id) for the unsubscribe — a write lock on the same shard while still holding the read lock. This self-deadlocks the poll task as soon as it finds a finished render with a live subscriber (~5 minutes after boot).

The frozen read guard then wedges all future subscribers of that shard, whose queued write locks in turn block the websocket event loop's reads (tokio's RwLock is fair), freezing it outside of its select! — so the watchdog can never fire and no further websocket events are dispatched. Symptoms: Subscribing gets logged but no events ever follow, and no warnings appear.

  • drop the read guard before any await by cloning the senders out first (also in the websocket event loop, so a slow subscriber can't stall the loop while holding a shard lock)
  • progress updates now use try_send; they're cosmetic so drop them on a full channel instead of blocking
  • log the poll's previously silent error paths (failed query / missing render)

poll_finished_renders acquires a read lock on the render's shard and
only drops it at the end of the loop body, but then calls
senders.own(render_id) for the unsubscribe - a write lock on the same
shard while still holding the read lock. This self-deadlocks the poll
task as soon as it finds a finished render with a live subscriber.

The frozen read guard then wedges all future subscribers of that shard,
whose queued write locks in turn block the event loop's reads, freezing
it outside of its select! - so the watchdog can never fire and no
further websocket events are dispatched.

- drop the read guard before any await by cloning the senders out first
- same treatment in the websocket event loop so a slow subscriber can't
  stall the loop while holding a shard lock
- progress updates now use try_send; they are cosmetic so drop them on
  a full channel instead of blocking
- log the poll's previously silent error paths
@MaxOhn
MaxOhn merged commit af53854 into MaxOhn:main Sep 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants