Skip to content

First-sync apply ordering starves the ArgoCD controller of cluster-admin, deadlocking gated installs #887

Closed
validatedpatterns/clustergroup-chart
#144

Description

@butler54

Summary

On a clean install, the root Application's first sync applies the imperative ServiceAccount objects before the ClusterRoleBinding that grants the ArgoCD controller cluster-admin, so those applies fail with forbidden: cannot create resource "serviceaccounts". Ungated installs self-heal on the next automated sync (~3 min). With gitops.applicationHealthCheckEnabled: "true" (app-of-apps sync-wave gating), the first sync operation parks waiting on wave-0 health and the retry never happens — a hard deadlock on every clean install.

Evidence (OCP 4.22.17, patterns-operator v0.0.82, singleArgoCD)

Time (UTC) Event
05:47:29–31 Operator creates the ArgoCD instance and its RBAC (argocd-operator roles, app.kubernetes.io/part-of: argocd)
05:51:51 Root Application first sync starts
05:51:51–58 Sync/0 resource /ServiceAccount:imperative/imperative-sa (SyncFailed, Failed, serviceaccounts is forbidden: User "system:serviceaccount:vp-gitops:vp-gitops-argocd-application-controller" cannot create resource "serviceaccounts" ...) — same for imperative-admin-sa
05:51:59 Same sync applies vp-gitops-cluster-admin-rolebinding (cluster-admin → the controller SA) — permitted, because the operator-provisioned role already has rbac.authorization.k8s.io: * / *
05:52 → stuck Operation: Running — waiting for healthy state of argoproj.io/Application/openshift-external-secrets; ESO is Degraded because the vault ClusterSecretStore cannot authenticate, because vault is unconfigured, because the imperative cronjobs cannot start (their SA never got created)

The operator-provisioned controller role (verbatim rules):

['']                          | ['serviceaccounts']  | ['impersonate']     ← no create
['rbac.authorization.k8s.io'] | ['*']                | ['*']

So the failure is purely apply ordering, not a missing permission in the operator's design: ArgoCD orders wave-0 resources by kind (Namespace → ServiceAccount → ConfigMap → ClusterRole → ClusterRoleBinding → …), and the cluster-admin grant ships in the same wave as the ServiceAccounts that need it.

This is not pattern-specific

The clustergroup chart's own argocd-super-role.yaml documents the same hazard:

# This CRB is also pre-created by the ACM policy (application-policies.yaml)
# on managed clusters to ensure the ArgoCD controller has cluster-admin
# before the first sync. It's kept here for non-ACM deployments.

i.e. managed/spoke clusters are protected by ACM pre-creation; standalone installs are not. Any pattern using singleArgoCD: true + imperative jobs (wave-0 ServiceAccounts) is affected the moment gating is enabled — which the framework needs for wave ordering to work at all, since ArgoCD 1.8 dropped the Application health check (argoproj/argo-cd#3781).

Option A (preferred): the operator creates the instance's cluster-admin binding

The operator already owns the ArgoCD instance identity: it creates the ArgoCD CR and injects global.vpArgoNamespace = getClusterWideArgoNamespace() (argo.go, ParamVpArgoNamespace) into the root Application's Helm parameters. It knows the name and namespace, so it can create — idempotently, at instance creation / reconcile, before the root Application exists — the equivalent of:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <vpArgoNamespace>-cluster-admin-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: <vpArgoNamespace>-argocd-application-controller
    namespace: <vpArgoNamespace>

Notes:

  • Mirrors exactly what the ACM policy already does on managed clusters.
  • Use activeArgoNamespace/activeArgoName (utils.go) so the legacy openshift-gitops path is handled identically.
  • Consider an opt-out CM key (e.g. gitops.createClusterAdminBinding, default true) so multi-tenant / least-privilege installs can keep the current behaviour — the chart warns "ONLY USE THIS FOR MANAGING CLUSTERS NOT FOR REGULAR USERS", and that concern applies to the operator-created binding equally.
  • Fixes the class for every pattern, permanently, and lets patterns delete their own out-of-band bridges.

Option B (alternative): move the chart's binding to an earlier sync wave

If the operator prefers not to create cluster-scoped RBAC, the equivalent fix is a one-annotation change in validatedpatterns/clustergroup-chart (templates/plumbing/argocd-super-role.yaml):

metadata:
  name: {{ $.Values.global.vpArgoNamespace }}-cluster-admin-rolebinding
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Why this is sufficient: the controller can already create ClusterRoleBindings before it has cluster-admin (rbac.* is fully granted by the operator-provisioned role — empirically it created the binding mid-sync), and ClusterRoleBindings have no health status, so wave −1 completes immediately and wave 0's ServiceAccount applies then run with cluster-admin.

Trade-offs vs Option A: smaller change and no operator release, but it only fixes patterns that consume a released clustergroup chart version — and for disconnected users it additionally requires re-mirroring the chart.

Interim mitigation (pattern-side, declarative)

Patterns can work around it today with a clusterGroup.extraObjects CRB under a distinct name at sync-wave: "-1"; that is what validatedpatterns/coco-pattern#165 is doing while this is decided. It is a bridge, not a fix: it hardcodes the operator's vp-gitops constant and leaves a redundant grant next to the chart's own binding.

Related

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions