[anthr76]

Why

scuttle v0.1.0 (the only release; upstream has only had dependency bumps since) drains with a single pass: it evicts pods one by one, returns on the first error, and never waits for evicted pods to terminate. scuttle.stop then deletes the Node regardless. A PodDisruptionBudget refusal therefore aborts the drain (Oct 22: grafana-pg-1, ~1s in), everything after it is never evicted, and Ceph OSDs/mons, CNPG primaries etc. are killed as if the node crashed. Rook never sees a blocked drain, so it never sets per-host noout or tightens the other hosts' OSD PDBs, and nothing stops a second host from going down.

Changes

• pkgs/scuttle/drain-timeout.patch (applied via patches):
  • New -drain-timeout flag (default 0 = upstream single pass, so the patch is upstreamable as-is).
  • With a timeout, Drain loops every 5s: re-lists pods on the node, evicts each one not already terminating, treats 429 (PDB) and other errors as retryable, and returns only when no evictable pods remain or the timeout expires.
  • Uses the policy/v1 Eviction API (EvictV1) instead of policy/v1beta1.
• services.scuttle.drainTimeout (seconds, default 600) is passed as -drain-timeout, and TimeoutStopSec becomes drainTimeout + 120 (720s by default).

Stacked on #155 (same module).

Verification

• Go unit tests (fake clientset, not included in the patch because they need extra go.sum entries): an eviction blocked by a PDB 3 times succeeds on attempt 4; an always-blocked eviction returns a timeout error; timeout 0 makes one attempt and returns the PDB error. go vet clean.
• Linux build of the patched package (on worker-1, via nixbuild.net): /nix/store/rd613kz54bximam9i40rabwnfjl7pj50-scuttle-0.1.0, and scuttle -help lists -drain-timeout.
• nix eval of worker-3: script passes -drain-timeout=600s, TimeoutStopSec=720.
• Not tested on a live drain. Suggested first test on a worker with an OSD: systemctl restart docker-kubelet (or reboot), then watch journalctl -u scuttle for eviction blocked, will retry → drained node, and check that Rook sets noout on that host (ceph osd dump | grep flags, ceph health detail).

Notes

• scuttle has restartIfChanged = false (fix(scuttle): don't restart scuttle on switch (it deletes the Node) #154), so deploying this takes effect at the next kubelet restart or reboot.
• Worst-case shutdown/kubelet stop is now ~12 min (drain timeout + delete) plus the pod clear from fix(scuttle): clear pods after kubelet stops, with a real RPC timeout #155.
• The HDD pools still stall while worker-4 or worker-5 is down; draining doesn't change that layout limitation.