Set the pods' keepalive sysctls so gen_rpc client sockets get a bound (T-614)

[chasers]

TL;DR: Every smolquery pod now sets the keepalive sysctls (5 s / 5 s / 2). A gen_rpc client socket that is idle when its peer dies closes in about 15 s, not 2 hours.

Tracker: T-614. Follows #447.

Why

• gen_rpc 3.6.1 turns SO_KEEPALIVE on for the sockets it dials.
• It does not let an application set the keepalive timers or TCP_USER_TIMEOUT on them.
• So those sockets used Linux's defaults: first probe after 2 hours.
• On 2026-10-02 the quiet buffer-1 -> buffer-0 channel waited the full 15 minutes.

What changed

• deploy/base/api.yaml, buffer.yaml, storage.yaml and deploy/overlays/kind-symmetric/server.yaml set these in the pod securityContext:
  • net.ipv4.tcp_keepalive_time = 5
  • net.ipv4.tcp_keepalive_intvl = 5
  • net.ipv4.tcp_keepalive_probes = 2
• These are Kubernetes safe sysctls from kubelet 1.29.
• They set the defaults for every socket in the pod that has SO_KEEPALIVE.
• A new kind test reads the three values from an api, a buffer and a storage pod.
• docs/deployment.md updates the gen_rpc row and adds an upgrade note.

What it covers, and what it does not

The test ran in a network namespace with the peer blackholed, using the same timers set per socket:

socket after the peer dies	result
idle (nothing sent)	✅ closed after 15.1 s (etimedout)
sent data first	⚠️ still open after 70 s

• Keepalive does not run while sent data is unacknowledged. That socket is on the retransmission path (tcp_retries2, about 15 min).
• So this bounds a gen_rpc channel that was quiet when its peer died.
• gen_rpc's own ping only goes out after 60 s of silence, so the keepalive probes come first.
• A channel that sends to the dead peer first still depends on T-613: the drop on :nodedown, and the timeout probe.
• HotClient sockets set their own timers per socket (Bound sockets to a peer node with keepalive and TCP_USER_TIMEOUT (T-614) #447), so these defaults do not change them.

Tests

• ✅ New kind test: every role's pod runs with the keepalive sysctls (T-614). It passed in the Cluster CI job, so the pods start with the sysctls.
• ✅ mix precommit: 3,078 tests pass
• ✅ mix ci

Review

A Fable review found nothing serious. It confirmed the Kubernetes claims (safe from 1.29), the YAML placement, the overlays, and the Linux semantics. Fixed in Review of T-614 sysctls: ...:

• a broken Markdown link in the upgrade note
• the admission caveats and the affected sockets, now in the docs
• the kind test reads only the last 3 lines of kubectl exec output

Watch out

• ⚠️ kubelet 1.29 or newer is required. An older kubelet rejects the pod with SysctlForbidden, and the pod does not start. Check the sandbox and production node versions before you roll this out. The kind CI cluster (kind v0.32.0) is new enough.
• ⚠️ The sysctls apply to every keepalive socket in the pod:
  • Finch pooled connections to object storage
  • the libpq catalog connections that DuckDB's postgres and ducklake extensions open
  • A peer that answers nothing at the TCP level for 15 s now closes those idle connections too. They reconnect on next use.
  • Erlang distribution and Postgrex set no keepalive, so they keep their behaviour.
• ⚠️ Admission can also refuse the sysctls:
  • Pod Security admission with enforce-version pinned to v1.28 or older
  • a policy engine (Gatekeeper, Kyverno) with its own sysctl allowlist
  • a hostNetwork: true pod (it cannot set net.* sysctls)
• ⚠️ Outside Kubernetes, set the same values with sysctl.

🤖 Generated with Claude Code