ate-api-server applies atelet's ReportActorSnapshotState reports

[igooch]

Part of Actor Lifecycle v2 #2069. Second PR for #2570, after #2612 added the API.

This serves WorkerService.ReportActorSnapshotState. With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node, on its own timers. The report is how ate-api-server learns about both. Nothing sends reports yet; that comes with atelet's asynchronous upload work.

A report lists the snapshots atelet holds for an actor on its node. ate-api-server acts on two things in it:

• latest_durable means an upload finished. If it names a recorded snapshot with the same uuid and upload URI, that snapshot becomes COMPLETED, with its manifest's fidelity.
• A report with neither latest_local nor uploading means atelet deleted its copies (a detach). It covers this node's snapshots up to the reported latest_durable generation. Those localities are cleared, and a suspended actor with no snapshot left on the node is released from it.

ate-api-server ignores the rest of the report.

Flowchart: how a report is applied

flowchart TD
    A["WorkerService.ReportActorSnapshotState(ReportActorSnapshotStateRequest)<br/>TODO: atelet does not send it yet"] --> B{"Caller presents atelet's certificate?"}
    B -- no --> X1["Unauthenticated or PermissionDenied"]
    B -- yes --> C{"Request passes validation?"}
    C -- no --> X2["InvalidArgument, including a report with none of<br/>latest_local, uploading, and latest_durable"]
    C -- yes --> E{"Actor exists, actor_uid equals Actor.metadata.uid,<br/>and Actor is on the caller's node?"}
    E -- no --> X4["NotFound"]
    E -- yes --> D{"Actor's lease free?"}
    D -- no --> X3["Aborted, atelet retries"]
    D -- yes --> E2{"Same check again, under the lease"}
    E2 -- no --> X4
    E2 -- yes --> S{"status.state?"}
    S -- "CRASHED" --> X9["FailedPrecondition"]
    S -- "SUSPENDING, RESUMING,<br/>DELETING, REVERTING" --> X10["Aborted, atelet retries"]
    S -- "any other state" --> X12["Internal"]
    S -- "SUSPENDED or RUNNING" --> G{"latest_durable set?"}
    G -- no --> K{"latest_local and uploading both unset?"}
    G -- yes --> GF{"At or below the latest COMPLETED generation?"}
    GF -- yes --> K
    GF -- no --> GR{"Its generation is in status.snapshots?"}
    GR -- "no, and not newer than<br/>last_assigned_generation" --> K
    GR -- "no, and newer" --> X5["FailedPrecondition"]
    GR -- yes --> U{"Same uuid, and an upload URI is recorded?"}
    U -- no --> X6["FailedPrecondition"]
    U -- yes --> V{"Reported URI equals the recorded one?"}
    V -- no --> X7["InvalidArgument"]
    V -- yes --> H["Mark it COMPLETED"]
    H --> K
    K -- yes --> K2{"Latest snapshot on this node, up to<br/>latest_durable's generation, is durable?"}
    K2 -- no --> X8["FailedPrecondition, nothing recorded"]
    K2 -- yes --> J["Detach: clear this node's localities up to that generation.<br/>Clear assigned_node if status.state is SUSPENDED, it is this node,<br/>and no snapshot is left on this node"]
    K -- no --> P{"status changed?"}
    J --> P
    P -- no --> Q["Done, nothing written"]
    P -- yes --> R["Write the actor"]

Loading

Outcomes

Report	Result
Any report for a CRASHED actor	FailedPrecondition
Any report for a SUSPENDING, RESUMING, DELETING, or REVERTING actor	Aborted, retried
latest_durable for a recorded snapshot with matching uuid and upload URI	That snapshot becomes COMPLETED, with the fidelity its manifest records
latest_durable at or below the latest COMPLETED generation	No change
latest_durable for a generation status.snapshots no longer has, such as one a revert discarded	No change
latest_durable for an unrecorded generation newer than any assigned	FailedPrecondition
latest_durable with another uuid, or for a snapshot with no recorded upload URI	FailedPrecondition
latest_durable uploaded to a URI other than the recorded one	InvalidArgument
None of latest_local, uploading, and latest_durable set	InvalidArgument
Neither latest_local nor uploading set (a detach)	Clears this node's localities up to the latest_durable generation. Clears assigned_node if the actor is SUSPENDED there and no snapshot is left on this node. Refused with FailedPrecondition while the latest covered snapshot isn't durable
A late detach, after a newer snapshot on this node	Leaves the newer snapshot and assigned_node alone
A detach that's refused	Nothing is recorded, including an upload the same report carries
A report that changes nothing	Nothing is written

Notes

• Nothing is pruned or deleted. The existing workflows prune old snapshots and delete their objects at the next suspend or delete.

• A detach covers only generations up to its latest_durable, so a late or repeated report can't clear a newer snapshot. Repeating any report is safe.

• A detach waits for a durable snapshot. Clearing the locality of a snapshot that isn't durable would let the actor resume from an older one.

• A report applies as a whole or not at all. atelet's detach uploads the latest snapshot first, so a refused detach is rare.

• The actor is checked before the lease is taken, and again under it, so an atelet can't hold the lease of an actor on another node.

• Reports apply only to a SUSPENDED or RUNNING actor. A transitioning actor's snapshots belong to its workflow, and a CRASHED actor's wait for a revert.

• Every UID or node refusal is NotFound, so an atelet learns nothing about actors on other nodes.

• Tests pass

• Appropriate changes to documentation are included in the PR

[Michelle Au (msau42)]

🤖 question 🟢 – A detach carries no generation, so it clears every locality on the node, including snapshots newer than the ones atelet deleted. A delayed or retried report could arrive after a later pause or suspend has left a new durable snapshot here. It would then clear that snapshot's locality and release the actor, even though atelet still holds the files.

Per the atelet proto, a detach's latest_durable is the actor's latest snapshot. Could the detach clear only localities at or below that generation, and leave newer ones alone?

[igooch]

Updated detach to be bounded by generation.

[Zoe Zhao (zoez7)]

🤖 should-fix 🟡 – A generation that status no longer has is outdated, but this returns success silently. Could this log an error and return FailedPrecondition instead?

[igooch]

🤖 I'd keep this as a no-op. A report applies as a whole, so an error would also drop a detach in the same report. Then the record keeps pointing at a node whose files are gone. A revert discarding the snapshot is expected, and atelet can't fix it by retrying. Treating stale generations as no-ops also matches the generation floor you suggested, and keeps "repeating a report is safe" true.

[Zoe Zhao (zoez7)]

I see, this is because we have in applyActorSnapshotState

  if err := applyLatestDurable(status, state.GetLatestDurable()); err != nil {
      return err          // <- returns error
  }
  if state.GetLatestLocal() == nil && state.GetUploading() == nil {
      return applyDetach(...)   // <- doesn't happen
  }

So this means, an atelet that reports an older durable will get ignored. I think that should be ok, can we log a warning here for debugging? We can do it in the next PR.

[igooch]

Right, an older or discarded latest_durable is ignored, and a detach in the same report still applies. I'll add a log in the next PR. I'd log at Info rather than Warn, since a late report after a revert or a newer upload is expected. It'll name the reported generation and the latest recorded one.