Skip to content

fix(scuttle): don't restart scuttle on switch (it deletes the Node) - #154

Merged
anthr76 merged 1 commit into
mainfrom
fix/scuttle-no-restart-on-switch
Oct 24, 2026
Merged

anthr76 merged 1 commit into
mainfrom
fix/scuttle-no-restart-on-switch

Conversation

@anthr76

@anthr76 anthr76 commented Oct 24, 2026

Copy link
Copy Markdown
Owner

Why

Deploying #153 on worker-1 (manual nixos-upgrade, 2026-10-24 20:55 UTC) knocked it out of the cluster:

  • The switch changed only scuttle.service, so switch-to-configuration restarted scuttle and left kubelet alone.
  • Stopping scuttle drains and deletes the Node. kubelet only registers its Node at startup, so with kubelet still running the Node never came back (uncordon error: nodes "worker-1" not found, Node worker-1 not found after 2 minutes; not recording a uid).
  • Its OSD went down, its mon was replaced, and all its pods were garbage-collected, until kubelet was restarted (worker-1 was rebooted).

This hits every worker tonight: their scheduled auto-upgrades (0N:40 UTC) will pull #153, change only scuttle, and delete each Node in turn.

Change

systemd.services.scuttle.restartIfChanged = false. A changed scuttle unit is no longer restarted by a switch; the new definition takes effect the next time kubelet restarts (scuttle is BindsTo kubelet) or on reboot. switch-to-configuration reads X-RestartIfChanged from the new unit, so this also protects the switch that deploys it.

Verification

nix eval of worker-2's scuttle.service unit: X-RestartIfChanged=false, After=... containerd.service (from #153) still present. Not tested on a live node.

@anthr76
anthr76 merged commit 2df6a76 into main Oct 24, 2026
21 of 26 checks passed
@anthr76
anthr76 deleted the fix/scuttle-no-restart-on-switch branch October 24, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant