[anthr76]

Why

During tonight's rolling reboots the clear-pods step (ExecStopPost of scuttle) logged DeadlineExceeded on most sandbox stop/remove calls and left 32–71 sandboxes per node, while still exiting 0. Reboots hid it; a kubelet restart without a reboot (the Oct 22 case) would leave stale-IP pods again.

Two causes:

1. crictl's per-RPC deadline defaults to 2s (--timeout). crictl stop --timeout 30 and rmp on a busy node exceed it and fail with DeadlineExceeded.
2. It ran while kubelet was still up. scuttle is After=kubelet, so on a kubelet stop scuttle (and its ExecStopPost) stops first. A live kubelet can recreate sandboxes underneath the clear, which is also why fix(scuttle): keep containerd up while clearing pods; fail if unreachable #153 could only warn about leftovers.

Changes

• crictl gets --timeout 60s.
• The clear moves out of scuttle into a new scuttle-clear-pods.service (oneshot, RemainAfterExit, ExecStop=clearPods), PartOf/WantedBy kubelet, Before=kubelet, After=criService. Stop order on any kubelet stop (restart, switch, shutdown) becomes: scuttle (drain, delete Node) → kubelet → scuttle-clear-pods → containerd.
• Sandboxes are stopped (stopp) and removed (rmp -f) individually in up to 3 passes; if any remain, the unit exits 1. With kubelet already stopped, leftovers are a real failure.
• restartIfChanged = false on the new unit too, so deploying/changing it never runs the clear mid-switch.
• scuttle drops the After=containerd / ExecStopPost from fix(scuttle): keep containerd up while clearing pods; fail if unreachable #153; the UID record (/run/scuttle/node-uid) is unchanged.

Verification

• nix eval of worker-3: scuttle-clear-pods.service has After=containerd.service, Before=docker-kubelet.service, PartOf=docker-kubelet.service, WantedBy=multi-user.target docker-kubelet.service, X-RestartIfChanged=false; scuttle keeps BindsTo/After kubelet and X-RestartIfChanged=false.
• Not tested on a live node. Suggested test on one worker after deploy: systemctl restart docker-kubelet, then journalctl -u scuttle -u scuttle-clear-pods should show the Node deleted → All pod sandboxes removed, and every pod IP should match the node's podCIDR.

Not addressed

A plain systemctl restart docker-kubelet stops scuttle via BindsTo but doesn't start it again (it is only WantedBy=multi-user.target). Pre-existing; worth a PartOf follow-up.