Repository navigation
fix(scuttle): keep containerd up while clearing pods; fail if unreachable - #153
Merged
Merged
Conversation
This was referenced Oct 24, 2026
anthr76
added a commit
that referenced
this pull request
Oct 25, 2026
…#155) ## Why During tonight's rolling reboots the clear-pods step (ExecStopPost of scuttle) logged `DeadlineExceeded` on most sandbox stop/remove calls and left 32–71 sandboxes per node, while still exiting 0. Reboots hid it; a kubelet restart without a reboot (the Oct 22 case) would leave stale-IP pods again. Two causes: 1. **crictl's per-RPC deadline defaults to 2s** (`--timeout`). `crictl stop --timeout 30` and `rmp` on a busy node exceed it and fail with `DeadlineExceeded`. 2. **It ran while kubelet was still up.** scuttle is `After=kubelet`, so on a kubelet stop scuttle (and its ExecStopPost) stops *first*. A live kubelet can recreate sandboxes underneath the clear, which is also why #153 could only warn about leftovers. ## Changes - crictl gets `--timeout 60s`. - The clear moves out of scuttle into a new `scuttle-clear-pods.service` (oneshot, `RemainAfterExit`, `ExecStop=clearPods`), `PartOf`/`WantedBy` kubelet, `Before=kubelet`, `After=criService`. Stop order on any kubelet stop (restart, switch, shutdown) becomes: scuttle (drain, delete Node) → kubelet → scuttle-clear-pods → containerd. - Sandboxes are stopped (`stopp`) and removed (`rmp -f`) individually in up to 3 passes; if any remain, the unit exits 1. With kubelet already stopped, leftovers are a real failure. - `restartIfChanged = false` on the new unit too, so deploying/changing it never runs the clear mid-switch. - scuttle drops the `After=containerd` / `ExecStopPost` from #153; the UID record (`/run/scuttle/node-uid`) is unchanged. ## Verification - `nix eval` of worker-3: `scuttle-clear-pods.service` has `After=containerd.service`, `Before=docker-kubelet.service`, `PartOf=docker-kubelet.service`, `WantedBy=multi-user.target docker-kubelet.service`, `X-RestartIfChanged=false`; scuttle keeps `BindsTo`/`After` kubelet and `X-RestartIfChanged=false`. - **Not tested on a live node.** Suggested test on one worker after deploy: `systemctl restart docker-kubelet`, then `journalctl -u scuttle -u scuttle-clear-pods` should show the Node deleted → `All pod sandboxes removed`, and every pod IP should match the node's podCIDR. ## Not addressed A plain `systemctl restart docker-kubelet` stops scuttle via `BindsTo` but doesn't start it again (it is only `WantedBy=multi-user.target`). Pre-existing; worth a `PartOf` follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
clearPodsOnDelete(#152) fired during the Oct 22 auto-upgrades but cleared nothing. From worker-4's journal at 04:49 UTC:switch-to-configurationstoppedcontainerd,docker-kubeletanddockerin one transaction.scuttle-clear-podscorrectly loggedNode worker-4 was deleted.crictlcalls then failed withvalidate CRI v1 runtime API ... Unavailablebecause containerd was already stopped.|| trueand the-prefix onExecStopPostmade the unit exit successfully.Changes
criServiceoption (defaultcontainerd.service). WithclearPodsOnDelete, scuttle is orderedAfter=it, so on a shared stop transaction systemd keeps containerd running untilExecStopPostfinishes.scuttle-clear-podswaits up to ~30s for the CRI endpoint and exits non-zero if it is unreachable. The-prefix is dropped fromExecStopPost, so this shows up as a failed unit (and a failednixos-upgrade) instead of a silent no-op.Verification
nix evalon worker-1 and worker-4:scuttle.after=["multi-user.target","docker-kubelet.service","network-online.target","containerd.service"];ExecStopPosthas no-prefix.journalctl -u scuttleforAll pod/Pod sandboxes remainingand that pod IPs match the node's podCIDR afterwards.Not addressed
scuttle still deletes the Node when the drain fails on a PDB. The pod clear covers the networking side of that.