Repository navigation
Give Dgraph's only pod a disruption budget, so node scale-down can't delete it - #4076
Conversation
Node consolidation deleted the pod 4 times in 30 days, and twice its disk stayed attached to the old node for about 40 minutes. A budget stops the autoscaler removing that node, and every other stateful service on the cluster already has one. A node upgrade still finishes, because GKE honours the budget for an hour. Refs #4066 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Lore is reviewing this PR — 7f45050d-605d-4387-82c2-39d61dcecdbf. Submit a review that requests changes to have Lore push a fix for it. Comment |
|
Nothing blocks this PR from merging now. Lore updates this comment when the reason changes. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🔍 Lore Spec Impact — advisoryThis PR touches 5 statement(s) across 2 spec(s), and changes the validating tests alongside every one of them. Weaker signals (5) — linked by a spec, not proven by a test runProject Constitution · 2 statement(s)
Principle 2: Zero Stored Credentials
validated by Principle 2: Zero Stored Credentials
validated by Feature Specification: Lore — Shared Context Infrastructure · 3 statement(s)
NFR-1: Security
validated by NFR-1: Security
validated by NFR-1: Security
validated by 1 new statement(s) have no test link yet. Deterministic · graph @ |
Why
Refs #4066. I'll close it once this is applied, because merging alone changes nothing on the cluster.
lore-dgraph-alphais the one stateful service on the cluster without a PodDisruptionBudget:lore-api,lore-db-primary,lore-stations,lore-uiandutopia-db-primaryall have one atminAvailable: 1. Node consolidation deleted its only pod 4 times in the 30 days to 2026-11-01. Twice its disk stayed attached to the old node, and the graph was down for 38m30s and 42m42s, failing ingests and impact reads across repos.What Changed
infra/terraform/lore-dgraph.tf: aPodDisruptionBudgetofminAvailable: 1selecting the Dgraph pod, as akubectl_manifestlike the Service and StatefulSet beside it.libs/shared/src/domain/infra-contract/security-posture.test.ts: a test that the budget exists, holdsminAvailable: 1and selects that pod, appended so no cited line moves.specs/1-lore-platform/spec.md: an NFR-2 statement citing that test.It needs a manual apply. CI never runs terraform (
docs/managing-secrets.md), so someone with access to Lore's terraform state has to run this after merging:It doesn't restart Dgraph, because it adds a new object and leaves the StatefulSet alone.
Alternatives Considered
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"annotation. It changes the pod template, so applying it restarts Dgraph and risks the same slow disk attach this is meant to prevent.headlamp.tfchooses for a dashboard. That file accepts an occasional restart because the dashboard is stateless. Here a restart can cost 40 minutes.lore-dgraph.tfalready defers it as a split zero/alpha HA setup, and it's a much bigger change.kubernetes_pod_disruption_budget_v1. It would work too, but this file declares its workloads askubectl_manifest.The cost: the node holding this pod won't be consolidated, and a node upgrade that reaches it waits up to an hour before GKE evicts the pod anyway, so an upgrade can still cause one slow restart. Autopilot bills pod requests rather than nodes, as
headlamp.tfnotes, so holding the node shouldn't add cost.ADRs & Architecture
None. This follows what
lore-api's values file already does for the same reason.Testing
Against the cluster, without persisting anything:
kubectl apply --dry-run=serveraccepts this exact manifest, its selector matches the one podlore-dgraph-alpha-0, and the namespace holds no budget today.Test:
LORE_SKIP_DGRAPH_TESTS=1 npx vitest run src/domain/infra-contract/inlibs/sharedpasses, and withminAvailableset to 0 the new test fails.Terraform:
terraform fmt -checkpasses.terraform validatewasn't run locally, because theterraform-validateCI job does it on this PR.GKE's documented behaviour, not tested here: scale-down is blocked when a node's deletion would violate a budget, and upgrades honour one for up to an hour, then evict.
After applying:
kubectl -n lore-dgraph get pdbshould showALLOWED DISRUPTIONS 0, andScaleDownevents on this pod should stop.Lint passes (
npx eslinton the changed test)Types check (
npx tsc --noEmit -p tsconfig.jsoninlibs/shared)Tests pass for changed packages (
src/domain/infra-contract/inlibs/shared)No secrets, tokens, API keys, or credentials in the diff
Migrations are idempotent (N/A, no
ui-helm/migrations/change)Commit trailers present if this is a Lore-managed branch (N/A, a human branch)
ADR created or updated if an architectural decision was made (N/A)
🤖 Generated with Claude Code