Skip to content

fix: prepare backup workers before quiesce and bound CSI captures - #772

Open
Baylem wants to merge 56 commits into
GameplanePanel:masterfrom
Baylem:fix/audit-backup-budgets
Open

Baylem wants to merge 56 commits into
GameplanePanel:masterfrom
Baylem:fix/audit-backup-budgets

Conversation

@Baylem

@Baylem Baylem commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Backup preparation could disable saving while a worker was still waiting for storage or repository access, and a stalled CSI capture could hold quiesce indefinitely. This prepares and verifies the real worker before save-off, then applies persisted deadlines to preparation, live copying and CSI capture.

  • Restic workers mount the source, initialize/check the repository and become Ready before quiesce. An owned projected ConfigMap authorizes only the exact Backup, worker Pod, source identity and expiry. Workers use resource defaults and prefer the source node, with an optional hard placement requirement. First binding waits for an initial source Pod to be created and scheduled within the preparation deadline; a replacement after binding still fails closed. They receive no Kubernetes API token.
  • Preparation, quiesce and CSI deadlines survive restarts and configuration changes. Copy failure/deletion revokes authorization and waits for actual worker drainage before save-on. Raw and already-stopped copies retain the full Job budget.
  • Aborted CSI snapshots remain invalid even if the driver later reports readiness. Cleanup verifies snapshot/content UIDs and respects Retain policy. Restore and new snapshot-backed PVCs reject incomplete, aborted or replaced captures.
  • Prepared workers record volume coverage before copy authorization and use the effective template under lifecycle ownership, including when a newer catalog revision is pending.

Includes controller, worker-script and snapshot regressions plus a dedicated two-node RWO preparation/restore CI drill. The Kind fixture does not claim qualification of attachable CSI drivers. Worker expiry assumes synchronized clocks; restoring saving after an operator outage still requires operator/agent recovery.

Dependencies included: #760 lifecycle exclusion, #766 stopped-server preparation, #767 capture metadata, #769 staged templates and its security/resource dependencies, #756 snapshot authorization, and #759 shared web build prerequisite. Review the commits for this issue group after those prerequisites. Shared restore cleanup retains ownership until failed startup drains, including force-deletion recovery. The preparation, placement and Restricted workload buckets remain registered in upstream change-based test selection.

Validation: production and regression-source compilation passed; required actionlint/offline zizmor and whitespace checks passed. Runtime unit, envtest and E2E execution remains in CI under the repository's compile-only local policy.

Closes #717.
Closes #719.

Baylem added 30 commits October 21, 2026 10:48
Signed-off-by: Baylem <zelathis@gmail.com>
…cement

Signed-off-by: Baylem <zelathis@gmail.com>
…opt-in

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@github-actions github-actions Bot added area: api REST/WebSocket gateway area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: fix Bug fix labels Oct 21, 2026
Baylem added 21 commits October 21, 2026 17:10
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: agent in-pod sidecar area: api REST/WebSocket gateway area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: fix Bug fix

Projects

None yet

1 participant