Repository navigation
Define Golden build recovery after delete, revert, or template replacement #2552
Description
Design question
After Golden Actors move into their template's atespace (#2533), ordinary authorized lifecycle operations can interrupt construction. Define whether the reconciler should recover or fail when it observes DELETING or REVERTING, or when the temporary Actor has been repointed to another template.
The narrowed #2543 retains failure handling for these observed states. It handles ordinary pause/suspend/resume interruptions and preserves the existing creation path when the temporary Actor is already missing. Thus an in-progress deletion and a deletion completed between reconciles can have different outcomes.
Scope
Decide whether to re-drive delete/revert and rebuild the intended temporary Actor, and specify the cleanup and resource-cost limits. Any recovery must invalidate prior warmup and capture approval when the Actor identity, revision, generation, or template changes. It must not adopt a checkpoint from the wrong template.
Keep Actors subject to ordinary lifecycle and ingress policy. Do not restore Golden-specific rejection by atespace or name prefix. Public API preconditions are not a prerequisite; internal workflow parameters can preserve atomic checks.
Cover interrupted operations, controller restart, completed deletion between passes, and repeated external interference. Whole-build timeouts, terminal cleanup, and retry UX belong with #2039.
AI assistance: Codex helped draft this design question.
Activity
cloudonly commentedon Oct 14, 2026
Two non-blocking findings from the #2543 review to fold into this design:
Updated 2026-10-16: #2543 has since been narrowed, and #2554 is no longer stacked under it. The first finding is obsolete and has been replaced; the second has been corrected. The original text is in the edit history.
DeleteTag grabs the source template's lease.No longer applies:acquireTagTemplateLeasewas removed with the rest of theworkflow_tag.gochanges. The gap is now the opposite: a published golden tag has no deletion protection. It is an ordinary tag in the template's atespace, soDeleteTagsucceeds when no Actor borrows it. The reconciler never notices, because it returns as soon asGoldenTagis set, and every laterCreateActorwithout an explicitsource_tagfails withFailedPrecondition: Tag not found. The borrowed case would be covered by Reject deleting a tag if at least one actor references it #1718. For the unborrowed case, this design should choose between treating the template's reference as a borrow and rebuilding a missing golden tag.workerservice.RequestActorSuspendno longer rejects controller-created actors, by design (this issue's scope says not to restore name/atespace-specific rejection). Correction: with the narrowed Run golden snapshots in the template's atespace #2543, a worker-initiated suspend during warmup does not restart warmup. It runs the ordinary suspend workflow, and the reconciler then publishes the resulting snapshot as golden before warmup has finished. #2554 proposes publishing only snapshots the reconciler captured itself. With that change the original risk applies: an actor that keeps suspending restarts warmup indefinitely, so a restart counter or cap (A golden that never becomes READY retries without a visible cap, terminal state or template condition #2039) is still needed. Nothing in-tree callsinternal/ateworker/suspendtoday, so this remains a potential risk only.