A golden that never becomes READY retries without a visible cap, terminal state or template condition #2039

Summary

If a template's golden never passes its wakeupProbe, the golden is re-run every 30 seconds with no failed state and no visible retry cap (we observed retries for more than 10 minutes with no sign of a cap). The template shows no API-visible reason for the failure. The golden actor stays in RESUMING in the ate-golden atespace and holds a pool worker until an operator removes the worker it runs on.

Impact

• Who hits it: any integrator whose template's app never becomes healthy, for example because of a broken image, a wrong probe port or an entrypoint that exits.
• What breaks:
  • The retrying golden keeps taking a pool worker.
  • The template is unusable, but nothing in the API says so or why.
• Cost: the client cannot learn why the template is unusable without reading worker pod logs, and cleanup needs operator access to the worker pods.
• Severity: Med.
• Proposed priority: P2. A misconfigured template is operator error, but the failure mode is invisible through the API and did not end on its own while observed.

Environment

• Substrate d277088b (v1alpha).
• GKE 1.36.x.
• gVisor sandbox class, unprivileged WorkerPool workers.

Steps to reproduce / evidence

1. Create a template whose probe can never pass (verified live). For example, entrypoint sh -c "exit 1" with a wakeupProbe on / port 80.
2. Watch the golden (verified live). ateom logs Actor starting again every 30s. In one run there were 11 attempts over about 5 minutes and the template's golden status was still not READY (PENDING) after 5.5 minutes. In another run (an app that exited on startup) it retried for more than 10 minutes with no sign of a cap. The template shows no failure condition.
3. State of the golden actor (verified live). The retrying golden actors were in RESUMING in the ate-golden atespace. In our cleanup we removed the workers by deleting their WorkerPool; the goldens then moved to CRASHED, and DeleteActor on each golden succeeded. The templates were deleted afterwards.
4. Worker-pod deletion is enough (verified live, on actors stuck in RESUMING after a failed restore). Deleting only the worker pods hosting stuck RESUMING actors moved those actors to CRASHED, after which DeleteActor succeeded. The WorkerPool did not need to be deleted.
5. Delete precondition (code-read at d277088b). Without any_state, DeleteActor refuses an actor that is not SUSPENDED or CRASHED (cmd/ateapi/internal/controlapi/workflow_delete.go:309-320).

We did not create a regular actor against a never-READY template, so its behaviour is not covered here.

Expected vs actual

• Expected:
  • Golden retries are capped.
  • After the cap, the template is marked failed with the app's exit status and log tail, and the golden's worker is released.
• Actual: retries continued for as long as we observed (more than 10 minutes) with no sign of a cap and no API-visible reason, and the golden holds a worker until an operator intervenes.

Workaround

An integrator can:

• clean up as we did (verified live): delete the WorkerPool hosting the retrying golden, wait for the golden to reach CRASHED, call DeleteActor on the golden (in the ate-golden atespace), then delete the template. This removes every worker in that pool.
• alternatively, delete the template, then delete only the worker pod hosting the retrying golden. From step 4 we expect the golden to move to CRASHED so it can be deleted, but this narrower path was not tested on a golden. We also did not test whether the golden is rebuilt if the template is left in place.
• to avoid getting there, check that an image's app starts and passes its probe before creating a template for it, and apply a client-side timeout to first creates.

Additional detail

Proposed behaviour:

• Cap golden retries (a count or a deadline). Then set a failed condition on the template that carries the app container's exit status and log tail, and release the golden's worker.

Related issues:

• Golden is marked READY and checkpointed after its app container has exited; every restore then fails opaquely (golden-checkpointed-after-app-exit.md): the no-probe version of the same broken-template scenario.
• A failed actor delete leaves the actor in DELETING indefinitely, holding its worker (failed-delete-stuck-in-deleting-holds-worker.md): another failure with no terminal state.
• Each template needs its own golden per atespace: first creates are slow, and goldens compete with actors for pool workers (golden-per-template-per-atespace-slow-first-create.md): goldens compete with actors for workers.