Repository navigation
fix(scuttle): stop scuttle-clear-pods before the network on shutdown - #157
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
During today's switch+reboot rollout (new scuttle drain +
scuttle-clear-pods), worker-1's shutdown logged:The unit is only ordered against containerd and kubelet, so on shutdown it can stop after networking/DNS are already down. Its API check then fails, and it (correctly) refuses to clear anything. The reboot wiped the pods anyway, but on a kubelet stop where the network goes down in the same transaction the clear would be skipped. On worker-5 the same shutdown happened to run while the network was still up:
Node worker-5 was deleted→Removing 15 pod sandboxes→All pod sandboxes removed.Change
scuttle-clear-pods:After=network-online.target nss-lookup.targetplusWants=network-online.target, so it stops before the network and resolver on shutdown.Verification
nix evalworker-2:After=containerd.service network-online.target nss-lookup.target,Wants=network-online.target,Before=docker-kubelet.service. Not yet exercised on a live shutdown.restartIfChanged = falsestill applies, so deploying it doesn't run the clear.