You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Suspend and pause finalize don't check the actor is still SUSPENDING after an out-of-band crash #2676
Neither finalize step checks that the actor it re-reads is still SUSPENDING. An actor that was crashed out-of-band before finalize re-read it comes out wrong in both workflows:
ensureSuspendedFinalized (cmd/ateapi/internal/controlapi/workflow_suspend.go) commits SUSPENDED over the crash.
ensurePausedFinalized (cmd/ateapi/internal/controlapi/workflow_pause.go) keeps the crash, but PauseActor reports success.
Scenario
SuspendActor or PauseActor marks the actor SUSPENDING, and the atelet call succeeds.
Before finalize runs GetActor, the DeleteWorker workflow (which takes no actor lease) crashes the actor: state CRASHED, Crash set, WorkerAssignment cleared in the same update. crashActor clears the assignment the same way.
Finalize reads the CRASHED record.
Suspend: the latest snapshot still has its PENDING durable snapshot, so the DurableSnapshotMissing check passes. The release is skipped because there is no assignment. The UpdateActor precondition is pinned to the CRASHED version it just read, so the update succeeds. The actor ends up SUSPENDED, with the durable snapshot COMPLETED and AssignedNode cleared, but still carrying the stale Crash from step 2.
Pause: with no assignment, finalize returns the record unchanged. The actor stays CRASHED, which is right, but PauseActor returns OK, so the caller is told the pause succeeded.
Expected
Both finalize steps say they re-read the actor "so an out-of-band transition (e.g. the syncer crashing the actor after its worker died) is not overwritten". When the re-read actor is not SUSPENDING, finalize should leave the record alone and fail with FailedPrecondition, as the mark steps do for a CRASHED actor. Both workflows would then end CRASHED with FailedPrecondition.
The precondition only protects against a crash landing between finalize's read and its write: that case fails Aborted (covered by the KeepsCrashFromWorkerDelete tests). It does not protect against a crash landing before the read.
TestPauseActor_Finalized_AlreadyCrashed gets FailedPrecondition only because its crash keeps the worker assignment and has no AssignedNode, so pause finalize takes the node-unknown crash branch. A real crash clears the assignment, and then pause returns OK.
Notes
#2688 pins the current behavior of both workflows in TestSuspendAndPause_FinalizeWithoutAssignment, so that the upcoming refactor that merges the pause and suspend workflows stays behavior-neutral. The fix should update that pin: sync state SUSPENDED → CRASHED, and both codes OK → FailedPrecondition.
changed the title [-]SuspendActor finalize overwrites an out-of-band CRASHED with SUSPENDED[/-][+]Suspend and pause finalize don't check the actor is still SUSPENDING after an out-of-band crash[/+]on Oct 15, 2026
Activity
Drive suspend and pause workflow tests through SuspendActor and Pause…
Address review: pin worker release and fidelity label in suspend tests
Address review: pin worker release and fidelity label in suspend tests
[-]SuspendActor finalize overwrites an out-of-band CRASHED with SUSPENDED[/-][+]Suspend and pause finalize don't check the actor is still SUSPENDING after an out-of-band crash[/+]Drive suspend and pause workflow tests through SuspendActor and Pause…
Address review: pin worker release and fidelity label in suspend tests
Address review: key matrix outcomes and note both agent-substrate#2676 …
Drive suspend and pause workflow tests through SuspendActor and Pause…
Address review: pin worker release and fidelity label in suspend tests
Address review: key matrix outcomes and note both agent-substrate#2676 …
Suspend and pause preserve concurrent actor crashes and report failure (
Merge the pause workflow into the suspend workflow (agent-substrate#2694