We hit a rare hang where a query never comes back. Not slow, not an error - the returned Task just never completes. The 20s read timeout doesn't fire either, so nothing rescues it. Last time it sat there for 17 hours until we restarted the process.
The statement itself is trivial, a prepared single-row UPDATE ... SET is_device_data = ? via ISession.ExecuteAsync. Happens maybe once or twice a day out of a few million queries, always on a different statement, so it doesn't look data-dependent.
Setup: driver 3.22.0, .NET 10.0.11 on Linux, single Cassandra 5.0.5 node, protocol v5. ReadTimeoutMillis 20000, ConnectTimeoutMillis 5000, QueryTimeout 20000. No speculative execution.
I caught one in the act and took a full dump. The task being awaited is a promise that was simply never signalled:
Task<Cassandra.RowSet>
m_stateFlags = 0x2000400 (WAITINGFORACTIVATION | PromiseTask)
m_continuationObject = RequestHandler+<SendAsync>d__51
No result, no exception, not cancelled. The awaiter is still attached. RequestHandler agrees it's still going:
Cassandra.Requests.RequestHandler
_state = 0
_nodeExecutionsCleared = false
_nextExecutionTimeout = null
Here's the part I can't explain. There were 45 OperationState objects on the heap. I dumped all of them, and 44 look like this:
Cassandra.OperationState
_callback = 000073edcad10820 <- same address as the static Noop
_state = 3
_timeoutCallbackSet = 0
_timeout = 000073edd22e3280 (non-null)
<TimeoutMillis> = 20000
Every single one. So there is no operation anywhere that's still waiting for a response, callbacks have all been swapped out for Noop, yet a RequestHandler is parked on a promise that will never be completed.
The _timeoutCallbackSet = 0 bothers me more than the rest. _timeout is there and TimeoutMillis made it onto the request, but if the timeout callback is never armed then a request whose response goes missing has no way out.
Things I checked and can rule out: no thread was blocked anywhere in driver or app code (7 idle pool workers sitting in LowLevelLifoSemaphore.WaitForSignal), so it isn't pool starvation. All five connections to the node were ESTAB with empty send and receive queues, and other queries kept flowing over those same connections the whole time. Host was ~80% idle, server ~94%, iowait basically zero. Nothing on our side holds a lock or a token on that path.
No repro, unfortunately. It only shows up under sustained parallel load, around 60 concurrent queries with a long sequential work list running continuously, and even then rarely. My guess is a race between response dispatch and timeout arming, but that's a guess, and I'd rather not pretend otherwise.
For now we've put our own timeout around each unit of work so a leaked request stops taking the whole batch down with it. Obviously that doesn't cancel the request, it just stops us waiting on it.
Still have the dump if there's anything specific worth pulling out of it.
We hit a rare hang where a query never comes back. Not slow, not an error - the returned Task just never completes. The 20s read timeout doesn't fire either, so nothing rescues it. Last time it sat there for 17 hours until we restarted the process.
The statement itself is trivial, a prepared single-row
UPDATE ... SET is_device_data = ?viaISession.ExecuteAsync. Happens maybe once or twice a day out of a few million queries, always on a different statement, so it doesn't look data-dependent.Setup: driver 3.22.0, .NET 10.0.11 on Linux, single Cassandra 5.0.5 node, protocol v5.
ReadTimeoutMillis20000,ConnectTimeoutMillis5000,QueryTimeout20000. No speculative execution.I caught one in the act and took a full dump. The task being awaited is a promise that was simply never signalled:
No result, no exception, not cancelled. The awaiter is still attached.
RequestHandleragrees it's still going:Here's the part I can't explain. There were 45
OperationStateobjects on the heap. I dumped all of them, and 44 look like this:Every single one. So there is no operation anywhere that's still waiting for a response, callbacks have all been swapped out for
Noop, yet aRequestHandleris parked on a promise that will never be completed.The
_timeoutCallbackSet = 0bothers me more than the rest._timeoutis there andTimeoutMillismade it onto the request, but if the timeout callback is never armed then a request whose response goes missing has no way out.Things I checked and can rule out: no thread was blocked anywhere in driver or app code (7 idle pool workers sitting in
LowLevelLifoSemaphore.WaitForSignal), so it isn't pool starvation. All five connections to the node were ESTAB with empty send and receive queues, and other queries kept flowing over those same connections the whole time. Host was ~80% idle, server ~94%, iowait basically zero. Nothing on our side holds a lock or a token on that path.No repro, unfortunately. It only shows up under sustained parallel load, around 60 concurrent queries with a long sequential work list running continuously, and even then rarely. My guess is a race between response dispatch and timeout arming, but that's a guess, and I'd rather not pretend otherwise.
For now we've put our own timeout around each unit of work so a leaked request stops taking the whole batch down with it. Obviously that doesn't cancel the request, it just stops us waiting on it.
Still have the dump if there's anything specific worth pulling out of it.