MeshGrpcService.WritePumpAsync races the client disconnect and logs a fail-level exception

[meshweaver-cloud]

What is failing

The server-streaming gRPC endpoint MeshGrpcService.Connect runs a background WritePumpAsync that drains a channel into IServerStreamWriter<T>.WriteAsync. When the client disconnects (or the request otherwise completes) with an item still in flight, the pump calls WriteCoreAsync on the now-complete request and Grpc.AspNetCore throws InvalidOperationException: Can't write the message because the request is complete. The exception is not caught inside the pump, so it escapes the service method and is logged by Grpc.AspNetCore.Server.ServerCallHandler at fail: level.

Probable cause — high confidence

Missing cancellation/disconnect guard around the write loop in MeshGrpcService.WritePumpAsync (called from Connect at line 173). The pump should observe ServerCallContext.CancellationToken / HttpContext.RequestAborted and treat InvalidOperationException with the "request is complete" message (and OperationCanceledException) as a normal end-of-stream, not an error. Evidence: the top frame is HttpContextStreamWriter\1.WriteCoreAsync, the message text is verbatim the one Grpc.AspNetCore raises from that method when the request pipeline has already ended, and the caller frames are entirely inside our own WritePumpAsync/Connect`.

Impact

Benign. One occurrence on one pod in a ~0-second window — the disconnecting client has already gone away, no user request fails as a result, and it does not repeat under load. The only real cost is noisy fail: log lines that trip alerting like this one. Low priority.

Where to look

• src/MeshWeaver.Hosting.Grpc/MeshGrpcService.cs, WritePumpAsync around line 195 — wrap the WriteAsync call so OperationCanceledException and InvalidOperationException with HResult/message matching Grpc.AspNetCore's "request is complete" are logged at debug and terminate the pump cleanly.
• Same file, Connect at line 173 — pass the ServerCallContext.CancellationToken into the pump if it isn't already, and stop pumping as soon as it fires.

---

Evidence

Fingerprint	cd98a07bcd38d0f4
Category	Grpc.AspNetCore.Server.ServerCallHandler
Severity	Error
Exception	System.InvalidOperationException
Top frame	Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter1.WriteCoreAsync(TResponse message, CancellationToken cancellationToken)`
Namespace	memex-cloud
Pods	memex-portal-deployment-76cbbdd5f-9mtkj
Occurrences	1
First seen	2026-08-24 11:14:44Z
Last seen	2026-08-24 11:14:44Z

Recent log lines

2026-08-24 11:14:44Z memex-portal-deployment-76cbbdd5f-9mtkj fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
      Error when executing service method 'Connect'.
      System.InvalidOperationException: Can't write the message because the request is complete.
         at Grpc.AspNetCore.Server.Internal.HttpContextStreamWriter`1.WriteCoreAsync(TResponse message, CancellationToken cancellationToken)
         at MeshWeaver.Hosting.Grpc.MeshGrpcService.WritePumpAsync(ChannelReader`1 reader, IServerStreamWriter`1 responseStream) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Grpc/MeshGrpcService.cs:line 195
         at MeshWeaver.Hosting.Grpc.MeshGrpcService.WritePumpAsync(ChannelReader`1 reader, IServerStreamWriter`1 responseStream) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Grpc/MeshGrpcService.cs:line 194
         at MeshWeaver.Hosting.Grpc.MeshGrpcService.Connect(ConnectRequest request, IServerStreamWriter`1 responseStream, ServerCallContext context) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Hosting.Grpc/MeshGrpcService.cs:line 173
         at Grpc.Shared.Server.ServerStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, TRequest request, IServerStreamWriter`1 streamWriter)
         at Grpc.Shared.Server.ServerStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, TRequest request, IServerStreamWriter`1 streamWriter)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

_{Opened automatically from Admin/_LogIncident/cd98a07bcd38d0f4. Recurrences are folded into this issue rather than opening new ones.}