[shreyas-blacksmith]

Summary

On bazel-l1-staging-2 (.17) a systemctl stop hung after Stopping gRPC server: grpcServer.GracefulStop() never returned, systemd SIGKILLed the process at TimeoutStopSec=30, and SaveAccessState (#72) never ran. GracefulStop waits for every in-flight stream to end and the fork sets no server keepalive, so a single stream whose client has gone away without closing it holds the drain forever. Traffic at the time was a trickle (one completed read in the 17 s before the signal), so this is a stuck stream, not load.

stopGrpcServer(s, grpcGracefulStopTimeout) replaces the bare call in the signal handler:

go func() { s.GracefulStop(); close(drained) }()
select {
case <-drained:
case <-time.After(10 * time.Second):
    log.Printf("gRPC server did not drain within %s, closing remaining connections", timeout)
    s.Stop()   // closes remaining transports; the pending GracefulStop returns
    <-drained
}

Idle or quickly-drained servers behave exactly as before (the second staging restart drained in ~1 s and saved/restored 14,901 entries). Only a drain that outlives 10 s is cut short, and the shutdown then still reaches the access-state save and the LRU observation drain well inside TimeoutStopSec.

Tests: TestStopGrpcServerBoundsAHungStream opens a bidi stream whose handler blocks until its context ends and asserts stopGrpcServer returns after the bound; TestStopGrpcServerReturnsEarlyWhenIdle covers the no-op path. Both run under -race.

Link to Devin session: https://app.devin.ai/sessions/df6c6655558146b9bafe936b062e49e7
Open in Devin Desktop: https://app.devin.ai/desktop/session/df6c6655558146b9bafe936b062e49e7?variant=devin
Requested by: @shreyas-blacksmith

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[blacksmith-staging]

Hung gRPC stream no longer blocks shutdown

Before, the server never left GracefulStop and was SIGKILLed at 30 s with no access state saved. After, it closed the hung stream at 10 s, stopped gRPC, saved access state, and exited 0.

_Merge-base

_{This PR ff9311e3}

Also checked · 2 not run

Idle server stops as before · Not run
Drain inside the bound is not cut short · Not run

_{Proof on ff9311e3 · ended early (interrupted)}

View full Proof report