Fix: Channel failure on shm resize due to CacheMiss - #254
Conversation
Combines the two independent fixes for the silent channel-death on mid-stream SHM resize (issue #253): - On reattach failure, recover instead of raising. A fast resize storm can dealloc the segment before we attach; broaden the catch to (ValueError, OSError) and drop to the shm-is-None ack path rather than killing the channel task. (from #254) - Swallow CacheMiss in _release_backpressure: the cached entry can be evicted by the reattach path before backpressure clears; a miss just means nothing to free, so keep acking. This is the specific crash #253 reports. (from #254) Retains the UninitializedMemory guards at the msg_id() call sites, which #254 lacks -- msg_id() raises UninitializedMemory on an uninitialized slot (the exact race), which would otherwise escape #254's buffer-id check and kill the task it aims to protect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On top of #254: the two MessageMarshal.msg_id() call sites in the resize path (repopulate loop and the buffer-id check) can hit a slot that is uninitialized after a mid-stream resize. msg_id() calls _assert_initialized() and raises UninitializedMemory there, which would escape the receive loop and kill the channel task -- the exact failure #254 is closing. Guard both sites: treat uninitialized as skip/mismatch (drop + release backpressure). Add tests/test_shm_grow.py: a parametrized real cross-process pub/sub with small buf_size and mid-stream oversized messages (single grow, two successive grows, grow-on-first-message), asserting every message is delivered. Complements the existing shm_resize_race_runner repro. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The attach-recovery + One gap worth closing before merge: the two I pushed a branch on top of this one that wraps both sites (treat uninitialized as skip/mismatch → drop + release) and adds a parametrized cross-process grow test (single grow / two successive grows / grow-on-first-message) alongside your
Both your repro and the new grow tests pass against it (16 passed). Happy to open it as a PR into your branch or you can cherry-pick — whichever you prefer. |
Guard msg_id() against UninitializedMemory + add cross-process grow test
|
Alrighty, after some issues with tests failing due to multiprocessing/pickling issues, I think this one is good to go. Its worth noting the following about this PR somewhere in documentation: In a narrow stale-SHM resize window, ezmsg may currently drop a message rather than risk wedging the channel. This is the only known message-drop condition at present. We have not observed this in production systems, and avoiding it robustly appears to require additional hot-path bookkeeping that we are not yet comfortable landing without stronger evidence and perf validation. Now the data is still present in SHM, and we don't HAVE to drop the message; it'd actually be quite easy to deliver it anyway, but the additional bookkeeping could add overhead that we don't need for such a rare occurrence. |
2 participants
If a
Publisherresizes SHM repeatedly, it can happen that the receivingMessageChanneleither fails to lease the block before it becomes deallocated, or that the associated block is no-longer leased and is deallocated when backpressure is cleared, resulting in an uncaughtCacheMissexception which kills the Task servicing the connection silently.The test case and hardening of message servicing in
MessageChannelin this PR likely closes #253.