feat(doctor): surface the machine beneath the nodes (backend#2221)

[shujaatTracebloc]

What

Adds a Machine capacity check to tracebloc doctor: the four-level chain host → Docker VM → node capacity → unrequested, plus the one invariant that makes the double-count legible:

sum(node capacity) ≤ VM capacity

Uncapped k3d violates it by exactly the node count. This is the doctor half of backend#2221 (RFC-BACKEND-664 §P4) and touches no installer, so it cannot conflict with the client-side work on the same ticket.

Why

Every other check in this package reads the cluster and believes it. On a local k3d install that belief is misplaced — the node containers are created with NanoCpus=0 CpuQuota=0 Memory=0, so each one honestly reports the whole Docker VM. Measured on k3d v5.9.0 / k3s v1.35.5 / Docker 29.5.2, macOS aarch64:

Layer	CPU	Memory
Host	10	16.00 GiB
Docker Desktop VM	10	7.75 GiB
k3d node containers ×2	10 each	8126672Ki each — byte-identical to the VM's MemTotal
What Kubernetes believes	20	15.50 GiB (2.000×)

So checkNodeFit can truthfully report "a Ready node can schedule a training job" for two jobs that cannot both exist, and its drift nudge (this machine could give a run up to cpu=9,memory=4Gi) advertises half the VM twice. The gap between the levels is the customer's problem, and nothing surfaced it: doctor read level 3 only, the installer's preflight gates on level 1, and nothing looked at level 2 at all.

Verified on a live cluster, not just fakes

Run against a real 2-node edge:

· FAIL Node capacity — no Ready node can fit a training job (needs cpu=2, memory=8Gi)
· WARN Machine capacity — host 16.00 GiB → Docker VM 7.75 GiB (10 cpu)
       → 2 nodes claiming 15.50 GiB → 4.55 GiB unrequested — Kubernetes believes
       2.00× the memory this machine has, because the k3d node containers are
       uncapped and each reports the whole VM

The two lines together tell the true story that neither tells alone. As a cross-check, 7.75 − 4.55 = 3.20 GiB requested matches the ~3276 Mi the control plane is known to request.

The live run found a bug in this check that the fakes could not. "Unrequested" was computed from the inflated sum, so it reported 12.30 GiB free on a 7.75 GiB VM — the fourth level inheriting the third's error, this check repeating the very lie it exists to expose. Now measured against min(sum, VM), which is a no-op on an honest cluster. There is a named regression test for it.

Refuses to assert what it cannot know

StatusUnknown (no signal — never moves the verdict or exit code) in four cases:

• non-k3d or mixed cluster — on EKS the nodes are machines and docker info on this laptop describes something unrelated; asserting from it would be worse than silence
• unreadable VM — the VM is the level this check adds; guessing is what the bug already does
• no Ready node
• nodes reporting no capacity — "0 GiB claimed, all good" would be a green with nothing behind it

The host level is best-effort and omitted rather than zero-filled when unreadable: the VM is the constraint, the host is context.

The tolerance is measured, not picked

A node capped at 3 GiB (3221225472 B) reported capacity 3221225Ki — 2.4% above the limit it came from. A strict sum > vm would therefore report correct capping as over-commit. 1.05 sits far below the 2.00× the real bug produces and far above that rounding. There is a test for each side.

Uses capacity, not allocatable, deliberately: the invariant is about what the nodes claim the machine is. Making allocatable honest is P3's kubelet-reservation question, and k3s sets no reservations, so on the clusters this applies to the two are byte-identical anyway (confirmed while measuring).

Tests

internal/doctor/machine_test.go, 18 cases pinned to the measured numbers. Probes are injectable (Options.VMProbe / HostProbe) so the chain is fully exercisable without Docker.

Mutation-tested — each of these reddens its own tests and nothing else:

mutation	reddens
tolerance 1.05 → 100	the 2-node and 3-node over-count tests
allK3d always true	the non-k3d and mixed-cluster tests
count terminal pods as holding memory	the unrequested-level test

Full suite green. gofmt -s, goimports -local, staticcheck -checks all,-ST1005, ineffassign, misspell all clean locally.

TestRun_HealthyCluster's check count moves 9 → 10. Its fixture nodes are not k3d-named, so the new check reports StatusUnknown there — which the rollup ignores, so the healthy verdict is unaffected. That is the intended behaviour on a non-local cluster, and the comment says so rather than leaving it looking like a fixture accident.

20 new strings are catalogued in zz-all-strings.golden (regenerated, diff reviewed).

Scope

Deliberately only the surfacing half. Capping the node containers and collapsing the topology is the client-installer half — it changes the default topology on every install, so it is flagged for @LukasWodka rather than landing on review alone. The contract arithmetic it will read is tracebloc/client-runtime#363.

Audit of the ticket's claims, including two corrections that changed the fix, is on backend#2221.

🤖 Generated with Claude Code

---

Note

Medium Risk
Diagnostic-only doctor check that shells out to docker info/sysctl and lists cluster-wide pods/nodes. It can change doctor output and warn on default local installs, but does not alter scheduling, auth, or installer topology.

Overview
Adds a Machine capacity check to tracebloc doctor so local k3d installs show the real host → Docker VM → node capacity → unrequested chain, instead of trusting Kubernetes’ inflated node memory.

Uncapped multi-node k3d reports each node as the whole VM, so checkNodeFit can say a job fits when two such jobs would OOM the VM. The new check warns when sum(node capacity) exceeds VM size (with a 5% tolerance for k3d’s measured rounding) and remedies with single-node or --servers-memory/--agents-memory. “Unrequested” is bounded by min(sum, VM) so free memory is not computed from the lie.

It stays unknown (no verdict/exit-code impact) for non-k3d, mixed, remote k3d, missing VM/docker info, or no Ready capacity. Host RAM is best-effort and omitted if unreadable. Probes are injectable and time-bounded so a wedged Docker daemon cannot hang the sweep.

^{Reviewed by Cursor Bugbot for commit 6ebe9ae. Bugbot is set up for automated code reviews on this repo. Configure here.}

[shujaatTracebloc]

bugbot run

[shujaatTracebloc]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6ebe9ae. Configure here.}