diff --git a/CHANGELOG.md b/CHANGELOG.md index e08860249..690a629d7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -344,6 +344,9 @@ relevant information. * Rust SDK workers now warn when autoscaling task polling encounters errors continuously for one minute. Repeated warnings use exponential backoff up to 15-minute intervals and stop after polling recovers. +* Workflow workers now preserve the outstanding workflow-task token if an internal admission + invariant is violated, buffering the replacement task instead of overwriting the task in flight + in release builds. * Unhandled workflow payload conversion errors now fail the Workflow Task so it can retry instead of failing the Workflow Execution. Workflows may still explicitly handle these errors. * Workers no longer send worker heartbeats or appear in centralized heartbeat reports before diff --git a/arch_docs/streaming-poc-docs/README.md b/arch_docs/streaming-poc-docs/README.md new file mode 100644 index 000000000..b7973dc17 --- /dev/null +++ b/arch_docs/streaming-poc-docs/README.md @@ -0,0 +1,94 @@ +--- +doc_id: EWS-DOCS-INDEX +status: pre-production +audience: [all] +--- + +# External Workflow Streams — documentation map + +External Workflow Streams keep high-volume input and output payloads in a pluggable backend rather +than Temporal History while compact markers preserve deterministic replay. No Temporal Server +changes are required. + +The documentation is deliberately split by reading mode: + +| You need | Start here | Authority | +|---|---|---| +| A mental model, diagrams, guarantees, and operational tradeoffs | [`guide/`](guide/README.md) | Explanatory; intentionally simplified | +| Exact implementation behavior and every edge case | [`spec/`](spec/README.md) | Normative for accepted behavior | +| Why a non-obvious design was chosen | [`decisions/`](decisions/README.md) | Accepted rationale | +| Candidate or not-yet-accepted behavior | [`proposals/`](proposals/README.md) | Non-normative until promoted | +| Required observable coverage | [`required-tests/`](required-tests/) | Executable validation contract | + +The old [`overview.md`](overview.md) URL remains as a short compatibility landing page. + +## Human guide + +Read these in order when learning or reviewing the feature: + +| Guide | Answers | +|---|---| +| [`guide/architecture.md`](guide/architecture.md) | Which components own data, coordination, History, serialization, and provider state? | +| [`guide/input-lifecycle.md`](guide/input-lifecycle.md) | How does a Workflow Task move through active, quiescent, parking, parked, and no-open-task states? | +| [`guide/output-commit.md`](guide/output-commit.md) | How does output move from logical buffering through staging and History proof to visibility? | +| [`guide/replay-and-recovery.md`](guide/replay-and-recovery.md) | What is recorded, how exact-range replay works, and what survives retry, eviction, and Continue-As-New? | +| [`guide/operations.md`](guide/operations.md) | Which failures retry, which require intervention, and what must a production deployment operate? | + +Diagrams explain the shape of the system but never define an edge case by themselves. Each guide +links to the specification sections that own its exact behavior. + +## Normative implementation reference + +Use the structured [`spec/` index](spec/README.md) to route by subsystem or stable document ID. + +| If you need to… | Read | +|---|---| +| Change public names, handle roles, Worker configuration, or feature boundaries | [`spec/public-surface.md`](spec/public-surface.md) | +| Implement or evaluate a backend provider | [`spec/backend-contract.md`](spec/backend-contract.md) | +| Change Workflow Task retention, parking, rollover, shutdown, or output arbitration | [`spec/wft-lifecycle.md`](spec/wft-lifecycle.md) | +| Change replay markers, segmentation, budgets, or continuation state | [`spec/annotation-format.md`](spec/annotation-format.md) | +| Change proto messages, activation jobs, readiness calls, or Core state | [`spec/core-lang-protocol.md`](spec/core-lang-protocol.md) | +| Change Worker-side delivery, conversion, merge, close, or staging | [`spec/python-runtime.md`](spec/python-runtime.md) | +| Change wake encoding, deduplication, acknowledgement, or producer recovery | [`spec/wake-signal.md`](spec/wake-signal.md) | +| Classify errors, metrics, retries, or operator response | [`spec/failure-taxonomy.md`](spec/failure-taxonomy.md) | + +Every normative spec carries machine-readable front matter with a stable `doc_id`, status, audience, +canonical scope, and related ADRs. + +## Proposals and promotion records + +Each proposal has two entry points: + +- a short overview for deciding whether the direction is desirable; and +- a detailed design for API, protocol, failure, compatibility, and validation review. + +The implemented output direction has been promoted into `spec/`; its detailed proposal remains a +non-normative rationale record. Workflow-to-Workflow subscriptions remain a future enhancement and +must not be inferred as available. See [`proposals/README.md`](proposals/README.md) for current status. + +## Supporting sets + +- [`decisions/`](decisions/README.md) contains 47 current ADRs. A specification says what is true; an + ADR says why the rejected alternatives were not chosen. +- [`required-tests/`](required-tests/) contains the parsed Milestone 1 and Milestone 2 case lists. + Editing their counted headings or bullets changes what the validation gate demands. +- [`verification-hazards.md`](verification-hazards.md) records seven constraints on trustworthy test + evidence, including native-extension freshness, repository alignment, sandbox isolation, and + controlled fault injection. + +Status reports, review rounds, implementation plans, and investigation handoffs do not belong in the +design set. Their durable conclusions must be promoted into a specification, ADR, required test, or +proposal. + +## Documentation rules + +- **One normative fact, one home.** Other specifications link to the owning document. Human guides + may summarize it but are explicitly non-normative. +- **Exact details stay textual.** Every diagram has a nearby explanation and specification link, so + no transition exists only as an image. +- **No line-number references or restatement of local code.** Specifications name stable symbols and + files; repository search resolves their current locations. +- **Every rule explains its failure consequence.** That consequence is what distinguishes a + load-bearing invariant from an implementation preference. +- **Rejected alternatives live in ADRs.** Candidate behavior lives in proposals. Accepted current + truth lives in specifications. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-001-coexist-with-contrib-workflow-streams.md b/arch_docs/streaming-poc-docs/decisions/ADR-001-coexist-with-contrib-workflow-streams.md new file mode 100644 index 000000000..57edb16b7 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-001-coexist-with-contrib-workflow-streams.md @@ -0,0 +1,50 @@ +# ADR-001 — Coexist with `contrib.workflow_streams` rather than replacing it + +**Status:** Accepted · **Affects:** P9, P17 · **Spec:** `spec/public-surface.md` + +## Context + +The Python SDK already ships an experimental `temporalio.contrib.workflow_streams` package +(`WorkflowStream`, `WorkflowStreamClient`, `TopicHandle`, `WorkflowTopicHandle`), reserving the +Signal/Update/Query names `__temporal_workflow_stream_publish`, `__temporal_workflow_stream_poll`, +and `__temporal_workflow_stream_offset`. Shipping a second streaming feature alongside it risks +two overlapping APIs users must choose between. + +## Options + +**A. Replace it behind a compatible API.** One public surface; the external-backend +implementation slots underneath. + +**B. Coexist as a separate module.** Two features, two module names, users choose. + +**C. Add an external-backend storage mode to the existing package** without changing its public +surface. + +## Decision + +**B.** The two features are mirror images, not two implementations of one idea: + +| | `contrib.workflow_streams` | External Workflow Streams | +|---|---|---| +| Direction | Workflow **produces**; external clients consume | Both external-input and external-output directions | +| Workflow-side API | `topic().publish()`; deliberately no workflow-side `subscribe()` | Input topics `subscribe()`; output topics `publish()` and `finish()` on distinct handles | +| Storage | Append-only log in Workflow state, i.e. in History | Pluggable external backend; payloads never enter History | +| Offsets | Dense `int`, exposed in `subscribe(from_offset: int)` and the offset Query | Opaque, provider-defined, totally ordered tokens (Redis IDs are `-`) | +| Transport | Signals, long-poll Updates, SSE bridge | Direct backend reads; Signals carry no payload and supply only wakeups | + +A is not feasible: there is no consumption API in the contrib feature to be compatible *with*, and +its `int` offset surface is incompatible with opaque provider offsets even for the directions that +overlap. + +C stays open as future work but is explicitly not a dependency of this design — it requires solving +the dense-integer-offset mapping. External Workflow Streams instead include a distinct output +direction with opaque client cursors; its separation of handle types preserves this decision. + +## Consequences + +- Both features ship; neither is deprecated by this work. +- This feature takes distinct names: module `temporalio.contrib.external_workflow_streams`, entry + points `external_stream` and `external_output_stream`, reserved Signal + `__temporal_external_stream_wake`. +- **No name in this feature may begin with `__temporal_workflow_stream`.** P9's completion criteria + assert this. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-002-cursor-is-a-position-boundary.md b/arch_docs/streaming-poc-docs/decisions/ADR-002-cursor-is-a-position-boundary.md new file mode 100644 index 000000000..703af9727 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-002-cursor-is-a-position-boundary.md @@ -0,0 +1,47 @@ +# ADR-002 — A cursor is a position boundary, not a record identity + +**Status:** Accepted · **Affects:** P2, P3, P5 · **Spec:** `spec/backend-contract.md` + +## Context + +A subscription must persist "where I got to" so a later Run, or a replay, resumes correctly. The +representation has to work for a consumer sitting at the current tail, where the next record does +not exist yet. + +## Options + +**A. The cursor names the next record to deliver.** Intuitive; resume is "read from the cursor". + +**B. The cursor is a position boundary** — `BEGINNING | AFTER(last_consumed_offset)`. + +## Decision + +**B.** + +A is not implementable. After a consumer reads the current tail, the next record does not exist and +therefore has no offset. Redis offers no way to name the ID the next `XADD` will produce, so a +consumer parked at the tail has nothing to persist as a next-to-deliver cursor. + +```text +cursor := BEGINNING + | AFTER(last_consumed_offset) +``` + +`BEGINNING` is the provider's beginning-of-stream boundary and need not be the offset of any real +record. `AFTER(x)` names the boundary immediately following the record at offset `x`, whether or not +a record after `x` exists yet. + +A provider may represent the boundary as `(offset, inclusive | exclusive)` instead. What it may not +do is require the cursor token to be the offset of a record that does not exist yet. + +## Consequences + +- Two distinct provider primitives, not one: an **exclusive** watch strictly after a boundary + (Redis `XREAD BLOCK`, or from sentinel `0-0` for `BEGINNING`), and an **inclusive** range read over + an explicit `[first, last]` pair (Redis `XRANGE`). +- **Replay never reads "from the cursor".** It reads explicit recorded ranges; the answer is already + in the marker. +- Offsets are compared by the provider's ordering rule, not lexically. Redis IDs compare as numeric + `(ms, seq)` tuples — string comparison breaks as soon as the millisecond component changes width. +- The conformance suite must contain a case that parks a consumer at the tail, appends a record whose + ID could not have been predicted, and resumes; a backend requiring a nameable next ID fails it. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-003-structural-immutability-is-required.md b/arch_docs/streaming-poc-docs/decisions/ADR-003-structural-immutability-is-required.md new file mode 100644 index 000000000..63dcc9d40 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-003-structural-immutability-is-required.md @@ -0,0 +1,50 @@ +# ADR-003 — Structural immutability is required of every provider + +**Status:** Accepted · **Affects:** P2, P17, P5, P13 · **Spec:** `spec/backend-contract.md` + +## Context + +Replay re-reads records from the backend rather than from History. It must detect damage to those +records — otherwise a Workflow silently observes different data on replay than it did live. The +question is what class of damage the design must detect, and what that detection costs per record. + +## Options + +**A. Require every provider to guarantee stored bytes cannot change.** Replay then validates +presence, count, order, and control positions only. + +**B. Make it optional, with a per-record content-hash mode** for providers that cannot make the +guarantee. Replay validates hashes for those providers. + +**C. Detect nothing**; trust the backend entirely. + +## Decision + +**A.** Structural immutability is a **registration-time requirement**, checked when a backend is +registered on the Worker (P17), not a per-provider runtime mode. A backend that cannot make the +guarantee does not satisfy the contract and is rejected before any Workflow can name it. + +Redis Streams is the worked example and satisfies it: an entry can be deleted by `XDEL` or removed by +trimming, but its fields cannot be rewritten in place. + +B costs one hash per record in every marker — the single encoding in the whole design that would grow +with item count — to defend against a failure mode no supported backend exhibits. Requiring the +guarantee up front is both cheaper and stricter than making it optional and paying to detect its +absence. C gives up the feature's determinism claim. + +## Consequences + +- Given the guarantee, replay needs to detect exactly one class of damage — **a record that is no + longer there** — and offsets suffice. Deletion, trimming, and retention expiry are the realistic + operational failure modes, and all are caught by the four range checks in + `spec/annotation-format.md`. +- **Nothing in the annotation encoding is per-record.** A run costs two offsets, a count, and a sparse + control list whether it covers ten records or a hundred thousand. This is what makes the + single-stream marker-size claim unqualified. +- **Accepted cost, stated plainly:** if a provider silently violates immutability — a buggy custom + backend, or out-of-band surgery on the stream — replay delivers the altered bytes as though they + were original, and no error is raised. The risk is bounded by the registration requirement and the + conformance suite. +- `schema_version` leads the annotation encoding, so a content-hash mode can be introduced later + without a format break if a provider ever needs one. +- A decode failure is therefore never an integrity failure — see ADR-015. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-004-progress-and-quiescence-are-separate-commands.md b/arch_docs/streaming-poc-docs/decisions/ADR-004-progress-and-quiescence-are-separate-commands.md new file mode 100644 index 000000000..b821d5ae4 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-004-progress-and-quiescence-are-separate-commands.md @@ -0,0 +1,41 @@ +# ADR-004 — Progress and quiescence are separate commands + +**Status:** Accepted · **Affects:** C1, C6, C14a, P10a, P10b · **Spec:** `spec/wft-lifecycle.md` + +## Context + +At every activation return the runtime may need to do two things: commit what the Workflow observed, +and ask Core to hold the Workflow Task open. A single command carrying both is the obvious encoding. + +## Options + +**A. One command** that reports quiescence and carries the consumed records. + +**B. Two commands** — `WorkflowStreamProgress` (commits an observation delta) and +`WorkflowStreamQuiescent` (asks for retention). + +## Decision + +**B.** The two questions are independent, and conflating them loses progress on every path where the +answers differ: + +1. *Did replay-visible stream state change?* → `WorkflowStreamProgress`, on **every** completion + path. +2. *Should the Workflow Task be retained?* → `WorkflowStreamQuiescent`, which starts the idle timer. + +Under A, an activation that both consumed records **and** produced a server-bound or terminal command +is not quiescent, so it emits nothing — and the consumption is never committed. See ADR-005 for why +that is silently fatal. + +A completion may carry either command, both, or neither. `WorkflowStreamProgress` never implies +retention, and `WorkflowStreamQuiescent` carries no annotation data. + +## Consequences + +- Two command tags (23 and 24) rather than one. +- Retention applies only when external stream waits are the sole reason the activation cannot + progress. If the completion also carries server-bound commands, Python emits progress but not + quiescence, and the wakeup comes from a watcher instead — see `spec/wft-lifecycle.md`. +- **Command ordering is normative:** `WorkflowStreamProgress` precedes every command whose value could + depend on consumed data, so replay validates a record before matching the command derived from it. +- Core accumulates successive deltas for one Workflow Task and writes them as exactly one marker. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-005-progress-is-an-observation-delta.md b/arch_docs/streaming-poc-docs/decisions/ADR-005-progress-is-an-observation-delta.md new file mode 100644 index 000000000..0042aa3f8 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-005-progress-is-an-observation-delta.md @@ -0,0 +1,48 @@ +# ADR-005 — Progress is an observation delta, emitted on every completion path + +**Status:** Accepted · **Affects:** C14a, P5, P10b · **Spec:** `spec/annotation-format.md` + +## Context + +`WorkflowStreamProgress` commits what replay must reproduce. The emission rule decides which +activations produce one. + +## Options + +**A. Emit when records were consumed.** The natural reading of "progress". + +**B. Emit on quiescence only.** Fewer commands; one delta per park. + +**C. Emit whenever replay-visible state changed** — an observation delta. + +## Decision + +**C.** + +A loses the empty-stream case entirely. A Workflow that subscribes to an empty stream, becomes +quiescent, and parks has consumed nothing, so it emits no delta, so its marker carries no annotation +— and replay then has no provider identity, no stream key, no starting cursor boundary, and no +recorded no-data boundary, leaving it nothing to work from but live backend state, which this design +forbids. + +B loses progress on every non-quiescent completion path, which is the failure ADR-004 exists to +prevent. + +> Emit a `WorkflowStreamProgress` delta whenever the activation changed anything replay must +> reproduce. That includes deliveries, but also the first observation of a subscription, an +> activation that observed no records, and the boundary on which the activation returned. + +Progress must reach the marker on **every** Workflow Task completion path that consumed external +data: normal completion; a completion carrying server-bound commands; a terminal command; parking +after quiescence; and rollover. + +## Consequences + +- The **first** delta for any subscription always carries provider identity, provider format version, + stream key, and explicit starting cursor boundary, whether or not a record has ever been delivered. +- Every activation that ran a drain contributes a segment, **even an empty one**. The annotation + grammar is `segment*` and `run*`, not `+`, and both empty cases must round-trip through the codec. +- The failure this rules out is specific and silent: if a consumed record influences a command that + lands in History while the consumption is never marked, replay re-reads that record from the last + committed cursor and delivers it again, while the command it produced is already durable. The + divergence surfaces as an unrelated nondeterminism error much later. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-006-runs-record-both-endpoints.md b/arch_docs/streaming-poc-docs/decisions/ADR-006-runs-record-both-endpoints.md new file mode 100644 index 000000000..d4724acd9 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-006-runs-record-both-endpoints.md @@ -0,0 +1,46 @@ +# ADR-006 — A run records both endpoints, not a start plus a count + +**Status:** Accepted · **Affects:** P5, P13 · **Spec:** `spec/annotation-format.md` + +## Context + +The annotation encodes consecutive deliveries from one stream as a compact run. The encoding has to +let replay re-read exactly the records that were delivered, and detect it when one of them is gone. + +## Options + +**A. `(wait_id, first_offset, count, control_positions)`** — a start plus a length. + +**B. `(wait_id, first_offset, last_offset, count, control_positions)`** — both endpoints and the +count. + +## Decision + +**B.** + +A cannot detect a deletion inside a range. Backend offsets are ordered but **not dense**, so +`(first_offset, count)` does not determine where a run ends. A count-limited read from `first_offset` +over an open-ended range returns *some* `count` records — and if a record inside the range was +deleted, the last one it returns is a later record standing in for the missing one. That validates +cleanly and delivers the wrong data. + +Recording `last_offset` closes it: replay reads the inclusive range `[first_offset, last_offset]` and +verifies + +1. both endpoints are present, +2. the range contains exactly `count` records, +3. ordering is strictly increasing under the provider's comparator, +4. `control_positions` match. + +A first, middle, or last deletion each fails a **different** one of those checks, and all four are +cheap. + +## Consequences + +- One extra offset per run — not per record — so this costs nothing asymptotically. +- These four checks are the whole of replay validation, sufficient because ADR-003 guarantees the + bytes themselves cannot change. +- A provider that cannot support this verification over a compact range must encode the exact offset + sequence in place of a compact run. +- The test list requires deleting the first, the middle, and the last record of a recorded range as + three separate cases; the middle one is what a start-plus-count encoding would have passed. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-007-byte-budget-forces-rollover.md b/arch_docs/streaming-poc-docs/decisions/ADR-007-byte-budget-forces-rollover.md new file mode 100644 index 000000000..1941628d8 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-007-byte-budget-forces-rollover.md @@ -0,0 +1,89 @@ +# ADR-007 — A hard byte budget forces rollover rather than growing a marker + +**Status:** Accepted · **Affects:** P5, C12b, C14a · **Spec:** `spec/annotation-format.md` + +## Context + +Markers are emitted once per Workflow Task consumption batch, not once per item. But marker *count* +being bounded is not the same as marker *size* being bounded, and only the first follows from +batching. A single Workflow Task can consume unboundedly many records, and an alternating +multi-stream workload produces one run per delivery. + +## Options + +**A. Trust range encoding.** Runs compress well; assume the marker stays small. + +**B. Check the size after encoding** and fail the Workflow Task if it exceeds the event limit. + +**C. Enforce a hard budget at encode time** and roll the Workflow Task over before it is exceeded. + +## Decision + +**C.** `MAX_ANNOTATION_BYTES` is declared as a constant, chosen well below the server's event size +limit, and enforced at encode time: + +1. When the accumulated annotation passes a high-water fraction of the budget, Python sets + `request_rollover` on its next `WorkflowStreamProgress`. +2. Core rolls the task over as it does on deadline expiry, **minus the finalization round trip** — + the triggering progress command came from Python and already carries the terminal, so Core writes + the marker and completes with `force_new_wft = true` without issuing `FinalizeExternalStreams`. + This is the one Core-initiated completion that needs no finalization job, because Python decided + the boundary. +3. The next Workflow Task starts a fresh annotation whose header records the current cursors. +4. The triggering segment ends with `BUDGET_ROLLOVER`, which is how replay knows the batch continues + in the following marker. + +A is not a bound. B turns a capacity problem into a stuck Workflow: the encoding that overflowed will +overflow again on retry. + +**The high-water mark is not by itself the whole of C.** It is a *fraction* of the budget and it turns +true only once a frame that crossed it has been emitted, while the frames are indivisible and three of +the four carry strings whose length this side does not choose. A frame can therefore be larger than the +slack a fractional mark leaves, and the batch it records cannot be deferred to the next annotation -- +its deliveries happened in this Workflow Task and that task's marker is where replay must find them. +Enforcement at encode time is completed by three further rules, spelled out in +`spec/annotation-format.md`: + +- the terminal and any late bindings frame are **reserved** rather than checked, so closing an + annotation can never be the thing refused; +- **delivery stops** before a segment the annotation could not record, which is the last point at which + stopping is an option, and that activation ends with `BUDGET_ROLLOVER`. A record is priced by the + largest run *the annotation* has encoded -- not the largest in the open segment, which is emptied + every activation and so prices every activation's first record at the bare floor; +- a **spill margin** sits behind the closing reserve for a segment frame to overrun into, which turns a + misprice into a rollover instead of a refusal, and guarantees that at least one record is affordable + in every annotation -- without which a Workflow can deliver nothing, observe nothing, and therefore + never ask for the rollover that was supposed to save it; +- a run larger than that margin is a **capacity limit**, refused where the record has been drained but + not yet handed to Workflow code, as a non-retryable `ExternalStreamCapacityError` naming the + provider's offsets. No rollover helps there: the next annotation carries the same run; +- the rollover request is read **after** the activation's segment is closed, so the completion that + crossed the line is the one that carries it -- read before, it described the annotation as it stood + an activation earlier and gave the next activation a free frame; +- a subscription set is refused at `subscribe()` when an **empty annotation** could not carry it -- + header, terminal, one segment frame, and the margin -- because a rollover writes a fresh header of + the same size and so cannot help. Pricing only the header and terminal there accepts a set that + clears the check and then cannot encode its first completion. + +The encoder still refuses an oversized frame if all of that is circumvented, but as a **non-retryable +application failure** -- failing the Workflow, not the Workflow Task. The server retries Workflow Task +failures regardless of cause, so a plain exception there is option B by another route. + +## Consequences + +- **An annotation can never exceed the budget.** Approaching it is a runtime event, not an error. +- The annotation budget is a **second delivery budget**, sitting alongside the per-activation record + cap; the smaller of the two wins. Records it leaves buffered have their readiness re-reported exactly + as the record cap's do. +- `subscribe()` can fail on capacity grounds. It is deterministic -- the same subscriptions in the same + order give the same answer -- so replay reproduces the refusal rather than diverging on it. +- Replay reassembles multi-marker batches in Workflow Task order, matching each marker to its + Workflow Task rather than concatenating blindly. +- **Bounded marker size is bought with additional Workflow Tasks.** The alternating multi-stream + workload `A,B,A,B,…` has a schedule transition per record, cannot be compressed by range encoding, + hits the high-water mark sooner, and rolls over more often. That is the *only* driver of marker + growth once ADR-003 removes per-record fields. +- In the single-stream scope of Milestone 1 the budget is not expected to fire at all. It is still + enforced, because "not expected" is not a bound. +- Tests assert **encoded byte size**, not run count — a run-count assertion tells you nothing about + what a future per-record field would cost. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-008-no-marker-without-a-terminal.md b/arch_docs/streaming-poc-docs/decisions/ADR-008-no-marker-without-a-terminal.md new file mode 100644 index 000000000..c1ae771c3 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-008-no-marker-without-a-terminal.md @@ -0,0 +1,62 @@ +# ADR-008 — Core never writes a marker without a terminal from Python + +**Status:** Accepted · **Affects:** C14b, C15a, C15b, C12b, C8 · **Spec:** `spec/core-lang-protocol.md` + +## Context + +The annotation grammar is `annotation := header, segment*, terminal`, and the terminal is the blocked +cursor snapshot. **Only Python can encode it** — Core is annotation-blind by design. But several +completion paths are decided *inside Core*, with no Python activation outstanding at the moment of +decision: idle-timeout park, all-fenced park, rollover-deadline expiry, and shutdown or eviction with +a Workflow Task open. + +## Options + +**A. Core manufactures a terminal** from what it knows. + +**B. Core writes a best-effort marker** from the last accumulated delta when it cannot obtain a +terminal. + +**C. Core always obtains a terminal from Python first**, and writes nothing if it cannot. + +## Decision + +**C**, with no exceptions. Core obtains the terminal via +`ExternalStreamParkResult.final_observation_delta` for the park paths, and via +`FinalizeExternalStreams` → `ExternalStreamFinalized` for the rest. + +A is impossible: Core does not parse the annotation and does not know stream offsets. + +B produces a marker that violates its own grammar — a `header, segment*` with no `terminal` — and +replay would have to guess at the boundary it was supposed to record. **A truncated annotation is +durable and wrong.** An abandoned Workflow Task, by contrast, commits no cursor and loses no record; +the cost is one repeated Workflow Task. + +If a terminal cannot be obtained, Core writes nothing, the Workflow Task fails, and the server +retries it. The replacement attempt replays from the previous marker. + +## What makes C implementable + +> **An accumulated, unwritten annotation exists only while a Workflow Task is open.** + +Deltas arrive only on activation completions, activations exist only under a Workflow Task, and every +Workflow Task completion path writes the accumulated annotation as exactly one marker and clears it. +So a Core-decided boundary either has a Workflow Task to finalize against, or has nothing to write. +There is no third state in which Core holds a partial annotation with no way to complete it. + +Core asserts the invariant where it clears `ExternalWaitSet.replay_annotation` (C15a), and a test +drives eviction in both Run states to confirm the no-open-WFT case writes no marker at all. + +## Consequences + +- Refusing to emit a terminal-less annotation belongs to **emission** (C14b); obtaining a terminal for + a Core-decided boundary belongs to **finalization** (C15a). That is why the dependency runs + C15a ⇢ C14b and not the reverse. +- `FinalizeExternalStreams` is a runtime-only activation job in the same class as + `PrepareExternalStreamPark`: it runs no user Workflow code, cannot resolve futures, and its only + legal responses are `ExternalStreamFinalized` or an activation failure. +- **`ParkReason` lives in exactly one place** — the Core-readable + `ExternalStreamMarkerData.terminal_boundary`. Core knows the reason on every path, so duplicating it + inside the opaque terminal would add a second copy that could disagree with the first. +- The marker is written by the **finalization completion**, never by the eviction completion, which + reports nothing and may carry no commands. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-009-shutdown-is-two-transitions.md b/arch_docs/streaming-poc-docs/decisions/ADR-009-shutdown-is-two-transitions.md new file mode 100644 index 000000000..d77fc4842 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-009-shutdown-is-two-transitions.md @@ -0,0 +1,53 @@ +# ADR-009 — Shutdown and eviction are two transitions, split by whether a WFT is open + +**Status:** Accepted · **Affects:** C15b, P20 · **Spec:** `spec/wft-lifecycle.md` + +## Context + +When a Worker shuts down with subscriptions still active, something must create server-visible work +so another Worker can pick the Run up. `force_new_wft` is the obvious mechanism. + +## Options + +**A. Set `force_new_wft` on the way out**, uniformly, for every Run with active subscriptions. + +**B. Let the language SDK request the forced task** on its completion. + +**C. Two transitions**, chosen by whether the Run holds a Workflow Task. + +## Decision + +**C.** + +A is not implementable in the state that needs it most. `force_new_wft` rides on +`ActivationCompleteOutcome::ReportWFTSuccess`, which is built from `data.task_token` — the token of +the Workflow Task the Run currently holds. **With no open Workflow Task there is no task token, no +completion, and therefore nothing to set `force_new_wft` on.** The window after a command-producing +or rolled-over completion is exactly that state: subscriptions active, no park generation installed, +no Workflow Task. + +B is also unavailable: `force_new_wft` is a local variable computed inside +`ManagedRun::prepare_complete_resp`, not a field in the completion protocol, and Worker shutdown +routinely begins while no Python activation is outstanding, so there may be no completion to ask on. + +| Run state | Marker | Server-visible replacement | +|---|---|---| +| Workflow Task open | Core issues `FinalizeExternalStreams{SHUTDOWN}` and writes the marker from the finalized annotation | Same completion sets `force_new_wft = true` | +| No open Workflow Task | **None** — nothing is accumulated (ADR-008) | Python's manager sends an acknowledged unparked wake Signal per active subscription, then tears the watcher down | +| No open Workflow Task, wait set parked | None | None needed — a confirmed park generation exists, so the producer's wake path applies unchanged | + +## Consequences + +- Both transitions are **runtime-owned**, not something the consuming Workflow requests. +- Both are offers to the task queue, not promises by the shutting-down Worker. Any eligible Worker may + pick the task up and reconstruct the subscription from the marker; if none is available, the task + times out and the server retries it — ordinary Worker-shutdown behavior. +- The no-open-WFT case needs the read-only `external_stream_run_status` probe (C4) to tell the states + apart, and it must not use the readiness call, which would falsely claim a record is buffered. +- The sweep cannot ride on the eviction path: an idle cached Run with no Workflow Task has no pending + work, so `shutdown_done` drops it with no eviction activation at all. +- An unacknowledged shutdown wake is retried within the grace period under the same request ID, then + reported through `temporal_external_stream_shutdown_wake_failed`. **A shutting-down Worker can promise a + hand-off attempt; it cannot promise the hand-off.** +- C15b and P20 are declared as separate deliverables because they cover mutually exclusive Run states; + testing them together would hide the case where the wrong mechanism is applied to the wrong state. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-010-finalization-is-manager-state-only.md b/arch_docs/streaming-poc-docs/decisions/ADR-010-finalization-is-manager-state-only.md new file mode 100644 index 000000000..370d202bf --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-010-finalization-is-manager-state-only.md @@ -0,0 +1,52 @@ +# ADR-010 — The input terminal in `FinalizeExternalStreams` is manager-state-only + +**Status:** Accepted · **Affects:** C15a, P19 · **Spec:** `spec/python-runtime.md` + +## Context + +`FinalizeExternalStreams` asks Python for the annotation's terminal — the blocked cursor snapshot — +on paths where Core decides the boundary. Whether answering it requires touching the backend +determines its failure classes, its retry policy, and where it can be dispatched. + +## Options + +**A. Reconcile the terminal against the backend** before encoding it, so the recorded boundary +reflects current stream state. + +**B. Encode the terminal from the manager's in-memory state**, calling no provider method. + +## Decision + +**B.** + +The boundary is not "wherever the stream is now"; it is **where this Workflow Task's deliveries +stopped**, which is fixed the moment the last activation of that task returned. The manager already +holds it exactly. Refreshing it against the backend would be actively wrong — it could name a +position replay must not reproduce. + +A would additionally require specifying the transaction, its race against readiness, and a retry +policy for transient failure, all to reproduce a value already in hand. + +This decision is specifically about producing `final_observation_delta`. The later output direction +reuses the job with reason `OUTPUT_LATENCY`; in that case the Worker's outer async layer also stages +buffered output and sends `WorkflowOutputStreamCommit` on the same completion (ADR-045). That output +operation does not refresh or otherwise change the input terminal. + +## Consequences + +- **There is no input-provider transaction to race while encoding the terminal.** Output staging, + when required, has its own failure and retry contract. +- **Watchers keep running during finalization**, exactly as they do during park preparation. A record + arriving mid-finalization changes nothing about the terminal: it belongs to the next Workflow Task + and reaches Core through the normal readiness path or, if none is open by then, through the wake + Signal. +- **The terminal encoder's only failure mode is missing state** — the Run's manager entry is gone or + unreadable. There is no transient class in that encoder. A separate output stage may fail as + `StreamStorageError`; either failure means Core writes no marker and the Workflow Task is retried + (ADR-008). +- It is still dispatched outside the synchronous Workflow thread, but **for a different reason than + the other runtime-only jobs**: it must run no user code and must be answered from out-of-sandbox + state, not because it blocks. See ADR-011. +- A test asserts the choice rather than trusting it: for a finalization with no buffered output, a + provider that raises on every input method is registered, the finalization is driven, and the + terminal must still be produced — proving its encoding makes no provider call. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-011-runtime-only-jobs-run-outside-the-workflow-thread.md b/arch_docs/streaming-poc-docs/decisions/ADR-011-runtime-only-jobs-run-outside-the-workflow-thread.md new file mode 100644 index 000000000..f41e51ece --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-011-runtime-only-jobs-run-outside-the-workflow-thread.md @@ -0,0 +1,52 @@ +# ADR-011 — Runtime-only jobs are handled in `_handle_activation`, not `_apply` + +**Status:** Accepted · **Affects:** P19, P8, P11, P13 · **Spec:** `spec/python-runtime.md` + +## Context + +Four activation jobs are added. Two of them are *themselves* backend operations: +`ReplayExternalStreams` needs inclusive range reads plus integrity validation over the whole recorded +range, and `PrepareExternalStreamPark` needs a backend transaction. The natural place to handle an +activation job is the `job.HasField(...)` dispatch chain in `_apply`. + +## Options + +**A. Handle all four in the `_apply` dispatch chain**, like every other job. + +**B. Partition them in `_handle_activation`**, the async layer above `activate()`. + +## Decision + +**B.** + +A is wrong by construction. `_WorkflowInstanceImpl.activate()` is **synchronous**, runs on a +thread-pool executor under `asyncio.wait_for` with a **2-second** deadlock timeout, and the Workflow +event loop is a custom deterministic loop that cannot await real network I/O. Putting a multi-second +backend transaction there fails the Workflow Task for a *healthy* backend, and the failure mode gets +worse the more records replay must validate. + +`_handle_activation` is already `async` and already performs pre-activation await work — +`decode_activation` is awaited there before `workflow.activate` is handed to the executor. Stream jobs +are partitioned in the same place: + +| Job | Handling | +|---|---| +| `PrepareExternalStreamPark` | Handled entirely there; the worker synthesizes `ExternalStreamParkResult` without calling `activate()` at all | +| `FinalizeExternalStreams` | Same place; its input terminal is manager-only, while `OUTPUT_LATENCY` may additionally stage output in the outer async layer — see ADR-010 and ADR-045 | +| `ReplayExternalStreams` | *Prepared* there: buffers filled and validated, then the job passes through to `activate()` for in-memory delivery | +| `ResolveExternalStreamWaits` | Passes straight through — readiness already means "buffered" | + +## Consequences + +- **`_apply` performs no I/O, ever**, and completes in bounded time. It pops from the buffer and + converts what it popped, and conversion is the half of decoding that awaits nothing (ADR-028). +- Replay is indistinguishable from live delivery to the Workflow thread, because both drain the same + buffer — one filled by the replay reader, one by a live watcher. +- Failures propagate through the defined activation-failure path — transient backend error → + `WORKFLOW_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE`, integrity violation → `StreamIntegrityError` + — rather than surfacing as a deadlock timeout, which would misattribute a storage problem to the + Workflow's own code. +- This is a **named deliverable (P19)**, not an implementation detail of the manager: it is the only + thing standing between a slow backend and a spurious `_DeadlockError`. +- Tests drive a replay job and a park job that each take longer than the deadlock timeout and require + both to complete. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-012-park-intents-are-keyed-per-subscription.md b/arch_docs/streaming-poc-docs/decisions/ADR-012-park-intents-are-keyed-per-subscription.md new file mode 100644 index 000000000..fa594daa9 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-012-park-intents-are-keyed-per-subscription.md @@ -0,0 +1,47 @@ +# ADR-012 — Park intents are keyed per subscription, not per stream + +**Status:** Accepted · **Affects:** P2b, P3b, P21 · **Spec:** `spec/backend-contract.md` + +## Context + +Parking installs an intent in the backend so a producer can discover that a consumer is waiting and +send a wake Signal. The intent must carry a cursor boundary and a park generation. Two subscriptions +in one Workflow may target the same stream. + +## Options + +**A. Key by stream key.** One intent per stream; simplest. + +**B. Key by `(stream key, wait_id)`.** One intent per subscription. + +**C. Key by `(stream key, wait_id, run_id)`.** + +## Decision + +**B.** + +A breaks whenever one Workflow holds two subscriptions to one stream. They have independent cursors +and re-block independently, so a stream-keyed intent lets one overwrite the other's cursor, and a wake +for one silently consumes the other's generation. + +C is unnecessary and harmful. `wait_id` is stable across a Continue-As-New chain and the stream key +already contains the first execution Run ID, so `(stream key, wait_id)` is unique within a chain, and +only one Run of a chain is live at a time. Including the Run ID in the *key* would let a new Run's +intent accumulate alongside its predecessor's instead of replacing it. + +The current Run ID is carried as the intent's **value**, so a new Run's intent deterministically +replaces its predecessor's for the same key. + +## Consequences + +- Every coordination structure this design places in the backend or on the wire is keyed to include + `wait_id`. +- A wake Signal carries `(wait_id, park_generation)` — the only generation state that reaches the + backend or the wire. `wait_generation` never leaves the Core/lang boundary. +- The parking conformance suite must fail a stub that keys intents by stream alone, using the + two-subscription case. +- Claims must be **leased and renewable**, so a claim abandoned by a crashed producer becomes + takeable rather than standing for the life of the store. The lease is not what keeps such a crash + from stranding the generation — a producer that loses the claim signals anyway + (`spec/wake-signal.md`). A provider that cannot lease must expose observe-only semantics and let + every producer signal idempotently. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-013-readiness-result-distinguishes-five-states.md b/arch_docs/streaming-poc-docs/decisions/ADR-013-readiness-result-distinguishes-five-states.md new file mode 100644 index 000000000..8464332db --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-013-readiness-result-distinguishes-five-states.md @@ -0,0 +1,49 @@ +# ADR-013 — The readiness result distinguishes a cached Run from a missing one + +**Status:** Accepted · **Affects:** C4, P7, P8, P20 · **Spec:** `spec/core-lang-protocol.md` + +## Context + +A watcher that observes an append calls `notify_external_stream_ready()` and must decide what to do +next from the answer. The interesting case is a Run that is cached and has registered waits, but has +no open Workflow Task — the healthy window after a command-producing completion or a rollover. + +## Options + +**A. Three results** — `Accepted | Stale | Parked` — with everything else an error. + +**B. Four results**, folding "cached but no open WFT" into `RunNotFound`. + +**C. Five results**, with `NoOpenWorkflowTask` distinct from `RunNotFound`. + +## Decision + +**C.** `Accepted | Stale | Parked | NoOpenWorkflowTask | RunNotFound`. + +B collapses the normal, healthy path into the cache-eviction path. Two things go wrong: the metric +cannot distinguish routine operation from eviction pressure, and the watcher tears itself down at +exactly the moment it is still needed — the Run is still cached, its Python watchers still exist, and +Core simply has no open Workflow Task. + +| Result | Meaning | Watcher action | +|---|---|---| +| `Accepted` | Readiness serialized into an open WFT | Nothing further; Core will activate | +| `Stale` | The wait exists but its `wait_generation` moved on | Re-probe; do **not** signal | +| `Parked` | A confirmed `park_generation` exists | Send the wake Signal | +| `NoOpenWorkflowTask` | Run cached, waits registered, no WFT open | Send the wake Signal; **keep** the watcher | +| `RunNotFound` | Run absent from this Core worker's cache | Send the wake Signal, then tear the watcher down | + +The three signal-sending results are distinguished by what the watcher does *afterwards* and by what +an operator should conclude, not by whether a Signal is sent. + +## Consequences + +- A separate **read-only** companion call, `external_stream_run_status(run_id) -> WftOpen | Parked | + NoOpenWorkflowTask | RunNotFound`, exists for the shutdown sweep. It is deliberately not the + readiness call: readiness means "a record is buffered", so probing with it would assert something + false and manufacture a spurious activation on the way out. +- Both are answered on the same serialized local-input lane, so a status answer is as authoritative as + a readiness acknowledgement. Core already answers a read-only question on that lane through + `GetStateInfoMsg`, so this is not a new kind of mechanism. +- Each result gets its own metric — see `spec/failure-taxonomy.md`. +- The bridge surfaces both enums to Python (P7). diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-014-integrity-loss-blocks-rather-than-fails.md b/arch_docs/streaming-poc-docs/decisions/ADR-014-integrity-loss-blocks-rather-than-fails.md new file mode 100644 index 000000000..24ab6b251 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-014-integrity-loss-blocks-rather-than-fails.md @@ -0,0 +1,52 @@ +# ADR-014 — Integrity loss blocks the Workflow; there is no terminal-failure opt-in + +**Status:** Accepted · **Affects:** P18, P13 · **Spec:** `spec/failure-taxonomy.md` + +## Context + +When replay finds a recorded record missing — deleted, trimmed, or expired — the Workflow cannot make +correct progress. What should the externally visible outcome be? + +## Options + +**A. Fail the Workflow** with a terminal `FailWorkflowExecution`. + +**B. Block the Workflow**: fail the Workflow Task repeatedly with a distinct error type and metric, +making no further progress until an operator intervenes. + +**C. Block by default, with an opt-in to fail terminally** for deployments that prefer it. + +## Decision + +**B**, with no opt-in. + +A destroys recoverable state. Retention loss is usually an operational error and usually repairable; +**a blocked Workflow can be resumed after repair, a failed one cannot.** A Workflow Task completion is +also a success/failure `oneof`, so failing the task and emitting a terminal command are mutually +exclusive. + +C is not implementable. Integrity loss is usually discovered *while replaying a historical Workflow +Task*, and that task's commands are already durable in History. Emitting `FailWorkflowExecution` at +that point contradicts the recorded commands and produces a nondeterminism error instead of the +intended clean failure — and replay cannot reach the current Workflow Task without first reading the +records it cannot read. A deployment that wants the Run gone terminates it, which is already the +documented operator action. + +> Integrity loss **blocks the Workflow**: the Workflow Task fails repeatedly with a distinct failure +> cause and a non-retryable error message, and the Workflow makes no further progress until an +> operator repairs the backend or terminates the Run. + +## Consequences + +- **"Non-retryable" describes the error surfaced to the user and to metrics, not the Workflow Task**, + which the server retries regardless. There is no protocol-level non-retryable Workflow Task failure. +- The distinction from a transient backend outage is in the error type and the metric, not in whether + the task is retried. An outage clears on its own; integrity loss does not. +- **Operators must alert on the `StreamIntegrityError` metric specifically**, not on Workflow Task + failure counts, which transient outages also increment. +- P18 registers **no** `workflow_failure_exception_types` entry. (The Python options are + `workflow_failure_exception_types` on the Worker and `failure_exception_types` on `@workflow.defn`; + there is no `WorkflowFailureErrorTypes`.) +- Should a Core mechanism to defer the failure until replay reaches a writable current Workflow Task + ever be designed, the opt-in can be reconsidered. +- **Integrity loss must never resolve to an alternate stream result.** diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-015-decode-failure-is-not-integrity-failure.md b/arch_docs/streaming-poc-docs/decisions/ADR-015-decode-failure-is-not-integrity-failure.md new file mode 100644 index 000000000..291905c3d --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-015-decode-failure-is-not-integrity-failure.md @@ -0,0 +1,41 @@ +# ADR-015 — A decode failure is a separate class from an integrity failure + +**Status:** Accepted · **Affects:** P18, P13 · **Spec:** `spec/failure-taxonomy.md` + +## Context + +Replay reads a recorded record and hands it to the `DataConverter`. The conversion can fail. So can +the read. Both surface at the same point in the code. + +## Options + +**A. One error class** for "replay could not produce the recorded record". + +**B. Two classes** — `StreamIntegrityError` and `StreamDecodeError` — with separate metrics. + +## Decision + +**B.** + +The operator actions are different, and A sends operators to the wrong one. A converter mismatch is a +configuration error on the **consumer**: the stream is fine, and reporting it as stream integrity loss +sends someone to restore a backend that was never damaged. + +Because ADR-003 requires structural immutability of every provider, the classification rule is +mechanical and unconditional: + +> **If the range validated, the bytes are the bytes that were written**, so any subsequent decode +> failure is a decode error. Only a missing offset or a range that fails validation is integrity loss. + +## Consequences + +- Two error types, two metrics, two documented operator responses: + - `StreamIntegrityError` → repair or restore the backend, or terminate the Run. + - `StreamDecodeError` → align the consumer's converter/codec with the producer's. +- Producer binding requires the producer to use **the same `DataConverter`** the consuming Workflow + uses, including any codec. A mismatch is detected at decode time on the consumer and surfaces as + `StreamDecodeError` — not as arbitrary user data, and not as a stream-integrity failure. +- P18 declares both types *before* the replay read path that raises them, so the taxonomy is not + invented incidentally by the first caller. +- A unit test asserts the classification rule in both directions, and the Milestone 1 list includes a + case where a record's bytes are intact but the converter configuration does not match. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-016-idle-timeout-reduces-by-min.md b/arch_docs/streaming-poc-docs/decisions/ADR-016-idle-timeout-reduces-by-min.md new file mode 100644 index 000000000..538619fe8 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-016-idle-timeout-reduces-by-min.md @@ -0,0 +1,42 @@ +# ADR-016 — The idle timeout is a Workflow-Task policy reduced by `min` + +**Status:** Accepted · **Affects:** C6, P10a, P21 · **Spec:** `spec/wft-lifecycle.md` + +## Context + +`idle_timeout` is configured per subscription through `with_options`. Core runs **one** timer for the +whole quiescent set, so several configured values must reduce to one before the completion is sent. + +## Options + +**A. Per-subscription timers.** Each subscription parks on its own schedule. + +**B. One timer, reduced by `max`.** The most patient subscription wins. + +**C. One timer, reduced by `min`.** The shortest configured timeout wins. + +## Decision + +**C**, applied over the quiescent set in `wait_id` order, then clamped below the rollover deadline. + +A defeats the point: the idle timeout exists to decide whether the *Workflow Task* can be released, +and one idle stream must not park the task while another is still delivering records. It is a property +of the set, not of a member. + +B breaks the option's meaning for whoever set the shortest one — a subscription configured to stop +waiting after 200 ms must not be held for 5 s because another member of the set is more patient. + +`min` also degrades safely: a shorter idle timeout costs an extra Workflow Task, never a lost record. + +## Consequences + +- The inputs are the configured values of the quiescent set and nothing else — no wall-clock input — + so the result is **deterministic and reproduces on replay**. +- Default is one second. +- **Validation rejects a non-positive or missing value at `with_options` time rather than coercing + it**, and Core independently rejects a `WorkflowStreamQuiescent` whose `idle_timeout` is + non-positive as a malformed completion. +- Core clamps the effective value below the Workflow Task rollover deadline so rollover stays + authoritative (ADR-017). +- Core never runs per-stream timers. +- Milestone 2 tests the reduction directly, including that the same reduction is observed on replay. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-017-rollover-is-mandatory-and-needs-its-own-timer.md b/arch_docs/streaming-poc-docs/decisions/ADR-017-rollover-is-mandatory-and-needs-its-own-timer.md new file mode 100644 index 000000000..f5eaee264 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-017-rollover-is-mandatory-and-needs-its-own-timer.md @@ -0,0 +1,58 @@ +# ADR-017 — Workflow Task rollover is mandatory and needs a sink-independent timer + +**Status:** Accepted · **Affects:** C13, C12a, C12b · **Spec:** `spec/wft-lifecycle.md` + +## Context + +Retaining an open Workflow Task is how the runtime keeps consuming without writing an event per +record. But a retained task is bounded by the server's Workflow Task timeout, and Core already has a +mechanism that forces a task to roll over: the local-activity heartbeat deadline. + +## Options + +**A. Defer rollover** to a later milestone; ship retention first. + +**B. Reuse `sink_heartbeat_timeout_start`**, the existing local-activity heartbeat timer. + +**C. Add a per-Run timer facility on `ManagedRun` independent of the local-activity request sink**, +and make the local-activity heartbeat one caller of it. + +## Decision + +**C**, from the first Workflow Task retained for external-stream delivery. + +**Why A is not available.** A stream whose inter-record gaps stay under the idle timeout never reaches +the idle parking path, so the retained Workflow Task runs until the server's Workflow Task timeout +expires and the task *fails*. With a one-second idle timeout, a producer emitting 100 records at +sub-second intervals holds a task open far past a default ten-second Workflow Task timeout. Any slice +that retains Workflow Tasks needs rollover in the same slice. + +**Why B is insufficient.** `sink_heartbeat_timeout_start` schedules the deadline by pushing a +`LocalActRequest::StartHeartbeatTimeout` into the local-activity request sink, inside +`if let Some(la_sink) = sink`. With no sink it **silently** returns an `AbortHandle` for a timer that +was never started. The sink is `Option`al on `ManagedRun` and is constructed only when +`config.task_types.enable_local_activities`, and Python sets +`enable_local_activities = self._activity_worker is not None`. + +So **a Python Worker that registers Workflows but no Activities has no rollover timer at all** — and +external streams must work on exactly that kind of Worker. + +The `force_new_wft` plumbing itself is fine and already exists; it is the *timer* that must be lifted +out. + +## Consequences + +- One code path for "deadline that forces a Workflow Task to roll over", avoiding two timers racing on + the same Run. +- The rollover deadline derives from the same `wft_timeout` that drives the local-activity heartbeat, + but is scheduled independently of the sink. +- The idle timeout is clamped below the rollover deadline so rollover stays authoritative. +- Every active subscription, cursor, annotation delta, and readiness generation survives rollover. +- Rollover also bounds a second effect: while a task is retained the server cannot start another one, + so Signals, Updates, and non-legacy Queries queue until it completes. Retention latency for those + inputs is bounded by the rollover deadline and nothing else; callers needing lower latency must + lower the Workflow Task timeout, not the idle timeout. +- The sink-independent timer is required for anything that retains a Workflow Task, and a test must + exercise a Worker with `enable_local_activities = false`. +- Rollover transport and marker finalization remain separate responsibilities so Core can request a + replacement task without writing an annotation until Python supplies its terminal boundary. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-018-replay-segmentation-is-reproduced.md b/arch_docs/streaming-poc-docs/decisions/ADR-018-replay-segmentation-is-reproduced.md new file mode 100644 index 000000000..38200971f --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-018-replay-segmentation-is-reproduced.md @@ -0,0 +1,62 @@ +# ADR-018 — Replay reproduces activation segmentation rather than collapsing it + +**Status:** Accepted · **Affects:** P5, P13, C10 · **Spec:** `spec/annotation-format.md` + +## Context + +A retained Workflow Task may span many `ResolveExternalStreamWaits` activations but produces exactly +one marker. On replay, Core issues one `ReplayExternalStreams` job for that marker. The question is +whether the *k* original activations must be reproduced as *k* drains. + +## Options + +**A. Collapse.** Deliver all recorded records in one drain; record order is what matters. + +**B. Reproduce, by issuing *k* activations from Core.** Requires Core to understand the annotation. + +**C. Reproduce inside Python.** The annotation carries one segment per original activation; the replay +driver performs one event-loop drain per segment. + +## Decision + +**C.** + +A changes behavior. Under `_single_batch_activation`, each stream activation drives exactly one +`_run_once(check_conditions=True)` drain, and stream jobs land in the non-query, non-signal job set, +so `workflow.wait_condition` predicates are evaluated **once per activation**. Reproducing only the +record order while changing how many drains occurred changes when conditions fire. + +B would require Core to parse the annotation, which it is designed not to do. + +Under C the live run's *k* activations become *k* drains inside one replay activation, so coroutine +scheduling and condition evaluation match. + +*k*, and not *k + 1*. The replay driver is not the only thing that drains: the job lands in the +non-query job set, so the activation runs one `_run_once` of its own for that set whatever the driver +did. The driver therefore drains the first *k - 1* segments and arms the last, leaving that one drain +to the activation -- which is also what the live run did, where each activation's single trailing drain +served the records it had just been handed. Closing the replay moves to after that drain, in the +activation's own `finally`. + +## Consequences + +- The annotation grammar carries `segment*`, one per original activation, each with its own + `segment_end_reason`. +- **An empty segment is meaningful** and must round-trip: an activation that drained and found nothing + still ran one `_run_once`. +- Segmentation reproduces **how many** drains happened; the global order of runs within a segment + reproduces **which wait** each drain served. A replay that kept the segment boundaries but let a + drain search the segment for its own wait would reproduce the count and reorder the deliveries, + which is the half of the live schedule concurrent coroutines can observe. +- This is safe with respect to Workflow time: all segments of a marker belong to one Workflow Task, so + `workflow.now()` is constant across them in both the live run and replay. Commands produced during + a retained Workflow Task are buffered until the task completes, in both directions. +- Core stays annotation-blind. +- The test list requires a `wait_condition` registered mid-stream to fire on the same delivery under + replay as it did live, for a marker spanning several activations. +- A test that drives only the replay driver cannot see the count: the drain it misses is the + activation's. The drain count is asserted across the whole activation, driver plus trailing + `_run_once`, or it is not asserted at all. +- **A zero-segment marker is the exception and closes before that drain**, because with no segment to + serve the drain is a live one and a reposition after it would retract records it had just delivered. + See `spec/annotation-format.md`. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-019-producer-binding-is-fully-explicit.md b/arch_docs/streaming-poc-docs/decisions/ADR-019-producer-binding-is-fully-explicit.md new file mode 100644 index 000000000..4c0d6a038 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-019-producer-binding-is-fully-explicit.md @@ -0,0 +1,55 @@ +# ADR-019 — Producer binding is fully explicit, and the stream name appears once + +**Status:** Accepted · **Affects:** P6a, P6 · **Spec:** `spec/backend-contract.md` + +## Context + +A producer — an Activity, or a plain external process — must address the right stream on the right +Workflow chain, encode payloads the consumer can decode, and append idempotently under retry. Some of +that could plausibly be inferred from ambient context. + +## Options + +**A. Infer what we can** from `temporalio.activity.Info` and Worker configuration; require the rest. + +**B. Require every binding input explicitly**, and have the producer verify the key before its first +append. + +For the stream name specifically: + +**B1.** Name inside the key passed to `connect()` *and* in `topic()`, validated for equality. +**B2.** Name only in `topic(name)`; `connect()` takes the Workflow *chain* key. + +## Decision + +**B** with **B2**. + +A cannot work for the most important field: `temporalio.activity.Info` exposes `workflow_run_id` but +**not** the first execution Run ID, so an Activity cannot derive the chain key. The Workflow passes it +to the producer explicitly — as an Activity argument, or through whatever channel a non-Temporal +producer already uses. + +B1 creates two sources of truth for one value. B2 removes the possibility of disagreement instead of +validating against it, and lets one connection serve several topics. + +A producer needs five things, none of which it can infer: + +- **The Workflow chain key**, including `first_execution_run_id`, verified by describing the Workflow + before the first append. Publishing under an unverified key is a configuration error, not a silent + no-op. +- **A backend connection.** A Worker has one configured backend; a plain process constructs a + provider directly. +- **A Temporal client**, for the wake Signal. +- **The same `DataConverter`** the consuming Workflow uses, including any codec (ADR-015). +- **A stable producer session ID and sequence**, which is what makes append idempotent under Activity + retry. + +## Consequences + +- Consumer-side and producer-side handles are **distinct types**: `ExternalStreamTopic` has + `subscribe()` and no `publish()`; `ExternalStreamProducerTopic` has `publish()` / `finish_writing()` + and no `subscribe()`. They are not the same object passed across a process boundary. +- Activities derive a default session ID from their own identity so a retried attempt reuses it; plain + processes **must** supply one — the API requires it rather than defaulting to a fresh random value. +- P6a's criteria include a wrong first-execution Run ID failing loudly. +- P6 stops at the append; the acknowledged-wake contract is P6b. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-020-append-idempotency-is-on-identity.md b/arch_docs/streaming-poc-docs/decisions/ADR-020-append-idempotency-is-on-identity.md new file mode 100644 index 000000000..60a162636 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-020-append-idempotency-is-on-identity.md @@ -0,0 +1,45 @@ +# ADR-020 — Append is idempotent on identity, not on key alone + +**Status:** Accepted · **Affects:** P2, P3, P6 · **Spec:** `spec/backend-contract.md` + +## Context + +Producer append must be safe under Activity retry, so it is keyed by `(session_id, sequence)`. A +retried attempt reusing the key must not create a second record. But a nondeterministic Activity may +retry with *different* bytes under the same key. + +## Options + +**A. Idempotent on key.** Any reuse of `(session_id, sequence)` is a no-op returning the original +offset. + +**B. Idempotent on key and identity.** Reuse with byte-identical content is a no-op returning the +original offset; reuse with different bytes is an **error**. + +## Decision + +**B.** + +Under A, a nondeterministically retried Activity silently aliases two different writes to one offset. +Replay cannot detect that — the offset is present, the range validates, the count matches — so the +Workflow observes bytes that no longer correspond to what the producer last intended. If it is ever +noticed, it is misdiagnosed as backend corruption. + +Making the mismatch an error surfaces the real problem, which is a producer that is not deterministic +across retries. + +## Consequences + +- Providers must store enough to compare content, not just the key. +- **The producer side owes the key its stability**, and that is a constraint on when the sequence is + drawn, not only on where the session ID comes from: at the call, before the payload's encode is + awaited. Drawn afterwards, the number follows the codec's completion order, which is external I/O + and not reproducible across attempts — so under B two concurrent publishes that complete the other + way round on a retry hit exactly the "same key, different bytes" error this record makes an error, + and the producer that earns it was deterministic. `spec/backend-contract.md` carries the rule. +- The conformance suite must include both directions: reusing a `(session_id, sequence)` pair with + byte-identical content is a no-op returning the original offset; reusing it with different bytes is + rejected as an error. +- Only successfully appended records may trigger wakeup. +- Exactly-once producer execution remains a **non-goal**; the backend adapter provides idempotent + append semantics instead. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-021-delivery-is-broadcast.md b/arch_docs/streaming-poc-docs/decisions/ADR-021-delivery-is-broadcast.md new file mode 100644 index 000000000..a4f3fc873 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-021-delivery-is-broadcast.md @@ -0,0 +1,39 @@ +# ADR-021 — Delivery to multiple subscriptions is broadcast, not work-sharing + +**Status:** Accepted · **Affects:** P9, P21 · **Spec:** `spec/wft-lifecycle.md` + +## Context + +One Workflow may hold two subscriptions to the same stream name. Two readings are possible: each sees +every record, or the two split the records between them. + +## Options + +**A. Work-sharing.** The two subscriptions compete for records; each record goes to one of them. + +**B. Broadcast.** Each subscription sees every record from its own cursor. + +## Decision + +**B.** + +Work-sharing requires a shared cursor that both subscriptions advance, which means a single mutable +position raced between two coroutines inside one Run. That is not reproducible on replay without +recording the arbitration of every record, which defeats the run encoding (ADR-006) — and the +arbitration itself has no deterministic rule to reproduce. + +Competing consumers means separate Workflows or separate Runs, each with its own subscription and +cursor, not two subscriptions racing for one cursor inside a single Run. + +## Consequences + +- Cursors are per **subscription**, not per stream. Two subscriptions to one stream name have + independent wait IDs, independent cursors, independent park intents keyed by + `(stream key, wait_id)` (ADR-012), independent `wait_generation`s, and independent entries in the + annotation header and the Continue-As-New continuation state. +- Cancelling a subscription commits its cursor at the next marker and removes it from the wait set. +- Re-subscribing to the same stream name creates a **new** subscription with a new wait ID; it does + not resume the cancelled one. +- Resumption across Continue-As-New is by wait ID, which is why re-ordering `subscribe()` calls is a + nondeterministic change. +- Work-sharing between subscriptions inside one Workflow is an explicit **non-goal**. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-022-continue-as-new-cursor-in-a-reserved-header.md b/arch_docs/streaming-poc-docs/decisions/ADR-022-continue-as-new-cursor-in-a-reserved-header.md new file mode 100644 index 000000000..a992b5160 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-022-continue-as-new-cursor-in-a-reserved-header.md @@ -0,0 +1,48 @@ +# ADR-022 — The Continue-As-New cursor travels in a reserved internal header + +**Status:** Accepted · **Affects:** P15 · **Spec:** `spec/annotation-format.md` + +## Context + +A stream spans a full Continue-As-New chain. A new Run must resume from the position the previous Run +committed, and must do so on replay as well as live. + +## Options + +**A. Read the current position from the backend** when the new Run establishes its subscription. + +**B. Store it in backend coordination state** keyed by chain, written at Continue-As-New. + +**C. Attach it to a reserved internal header on the Continue-As-New command**, persisted in the new +Run's `WorkflowExecutionStarted`. + +## Decision + +**C.** + +A and B both derive a cursor from **mutable backend state**, which replay must never do: the value +read on replay is whatever the backend holds now, not what the Run originally started from, so two +replays of the same history can diverge. + +C makes the starting position durable in History itself. It is restored before any subscription is +established, and populates the same annotation header `start_cursor` field that a first-execution Run +fills with `BEGINNING`. + +> A cursor is never derived from mutable backend state, on any Run. + +- **First execution of a chain:** `BEGINNING`, recorded in the header's `start_cursor` in the + subscription's **first** observation delta — emitted whether or not a record was ever delivered. +- **Subsequent Runs:** the committed continuation state arrives in the reserved header as an + `AFTER(offset)` boundary. + +## Consequences + +- Replay reads an explicit starting boundary in **every** case, including the case where the stream + was empty for the subscription's entire life. +- The stream key `(namespace, workflow ID, first execution Run ID, stream name)` uses the *first + execution* Run ID, which prevents collisions after Workflow ID reuse while remaining stable across + the chain. +- P15 depends on C14b, not only on header propagation: a Continue-As-New that drops its final segment + restarts the new Run at a stale cursor. The observation delta must be committed on the + terminal-command path. +- Two same-stream subscriptions restore independently, each keyed by its own `wait_id`. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-023-park-generation-zero-is-the-unparked-wake.md b/arch_docs/streaming-poc-docs/decisions/ADR-023-park-generation-zero-is-the-unparked-wake.md new file mode 100644 index 000000000..0678294b3 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-023-park-generation-zero-is-the-unparked-wake.md @@ -0,0 +1,48 @@ +# ADR-023 — `park_generation = 0` is the unparked wake + +**Status:** Accepted · **Affects:** C1, C11, P14, P20 · **Spec:** `spec/wake-signal.md` + +## Context + +The wake Signal carries `(wait_id, park_generation)`, and Core ignores a Signal naming a generation it +does not recognize as stale. But three senders need to wake a Run when **no park generation exists at +all**: a watcher seeing `NoOpenWorkflowTask`, a watcher seeing `RunNotFound`, and the Worker's +shutdown sweep. + +## Options + +**A. A separate Signal or command** for the unparked wake. + +**B. Send the parked envelope with an arbitrary generation** and let Core treat it as stale-but-wake. + +**C. Reserve `park_generation = 0`** to mean "no park generation — this is a recheck request". + +## Decision + +**C.** Park generations are quiescence generations, which start at 1, so 0 is free. + +A adds a wire format, a Core handler, and a Python send path to express something the existing +envelope can carry. + +B is self-defeating: an unrecognized non-zero generation is *supposed* to be ignored as stale, because +there the sender is making a claim that turned out to be wrong. Overloading it removes the design's +only defense against genuinely stale Signals. + +Core validates chain identity for an unparked wake and otherwise accepts it as a recheck request. The +runtime rechecks every active subscription on wakeup regardless, so an unnecessary unparked wake costs +at most one empty Workflow Task — which this design already permits. + +## Consequences + +- **Parked and unparked wakes use the same envelope.** One Signal name, one message, one interception + path. +- A **non-zero** generation the current Run does not recognize is still ignored as stale. +- The derived request ID needs care for unparked wakes. It is normally derived from + `(namespace, workflow_id, first_execution_run_id, stream_name, wait_id, park_generation)` so retries + deduplicate server-side. With `park_generation = 0` there is no attempt identity, so the derivation + additionally includes **the sender's identity and a per-sender monotonic wake counter**, held fixed + across retries of that one attempt. Without that, two Workers shutting down at different times would + derive the same request ID and the server would deduplicate the second wake away — turning a correct + retry mechanism into silent loss. +- Tests must assert both directions: an unparked wake is accepted as a recheck, and two Workers + shutting down at different times derive **different** request IDs with both wakes delivered. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-025-wake-signal-bypasses-the-data-converter.md b/arch_docs/streaming-poc-docs/decisions/ADR-025-wake-signal-bypasses-the-data-converter.md new file mode 100644 index 000000000..7ef5cd434 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-025-wake-signal-bypasses-the-data-converter.md @@ -0,0 +1,40 @@ +# ADR-025 — The wake Signal bypasses the user's `DataConverter` + +**Status:** Accepted · **Affects:** C1, C11, P14 · **Spec:** `spec/wake-signal.md` + +## Context + +Core must read the wake Signal's fields — wait ID, park generation, chain identity — to decide whether +to resume the Run. Core has no access to the user's `DataConverter` or codec. + +## Options + +**A. Send it as a normal user Signal** through the configured `DataConverter`, and have Core ask the +language SDK to decode it. + +**B. Define the envelope at the protocol level** and send it through a raw +`SignalWorkflowExecution` request built with the protocol's own serialization. + +## Decision + +**B.** The envelope is a single argument whose `Payload` uses metadata `encoding = "binary/protobuf"` +and `messageType = "coresdk.external_stream.WakeSignal"`. + +Under A, a user codec that encrypts payloads makes the envelope unreadable to Core — the one component +that must read it. Round-tripping through the language SDK to decode a Signal that exists to wake the +language SDK inverts the dependency and cannot work when the Run is not cached. + +**The Signal carries no user data, so bypassing the codec leaks nothing.** + +## Consequences + +- `WakeSignal` lives in its own proto file, `external_stream/external_stream.proto`. It is not a + command or activation variant; it is the wire format of the reserved Signal's payload. +- The producer does **not** reuse the public Python Signal path. That path also generates a fresh UUID + request ID per attempt, which would defeat the stable-request-ID requirement that lets the server + deduplicate retries — a second reason the wake path is separate. +- `envelope_version` starts at 1; Core rejects unknown versions harmlessly. +- **Core suppresses the Signal from user handlers whether or not it validates**, so an unknown envelope + version or a stale generation can never reach Workflow code as an unhandled Signal. +- The Signal name `__temporal_external_stream_wake` is fixed and versioned by the envelope rather than + by the name. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-026-delivery-is-bounded-by-a-record-count.md b/arch_docs/streaming-poc-docs/decisions/ADR-026-delivery-is-bounded-by-a-record-count.md new file mode 100644 index 000000000..be9f95cd1 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-026-delivery-is-bounded-by-a-record-count.md @@ -0,0 +1,103 @@ +# ADR-026 — Delivery within one activation is bounded by a record count + +**Status:** Accepted · **Affects:** P11, P8, P10b, P13 · **Spec:** `spec/python-runtime.md` + +## Context + +An activation runs on a thread-pool executor under a **2-second** deadlock timeout. ADR-011 keeps +I/O off that thread, which bounds how long any *single* step of a drain takes — and says nothing +about how many steps there are. + +Delivery volume is the unbounded direction. The Workflow thread drains the subscription's buffer +while the watcher refills it concurrently from the Worker's own asyncio loop, so a producer that +never lets the buffer run dry produces an iterator that never blocks and an activation that never +ends. Measured against the real iterator: 316,086 records consumed in 2 seconds, blocking zero +times. The Workflow Task then fails on the deadlock timeout, and every retry does the same, so the +Workflow is stuck permanently — a healthy producer and a healthy backend, and a Workflow that never +progresses. + +Workflow Task rollover does not cover this. Rollover bounds the Workflow *Task*, is decided by Core +between activations, and cannot interrupt an activation in progress. + +## Options + +**A. Bound by elapsed time.** Stop delivering when the activation has spent some fraction of the +deadlock budget. + +**B. Bound by the buffer size alone.** Deliver until the buffer is empty; rely on the buffer bound +to make that finite. + +**C. Bound by a fixed record count per activation**, and re-arm readiness for whatever is still +buffered when the budget is exhausted. + +## Decision + +**C.** `MAX_RECORDS_PER_ACTIVATION` is 256 records handed to Workflow code per activation. When the +budget is exhausted the subscription blocks **even though records are still buffered**, which is +what ends the activation, and readiness is re-armed for those buffered records so the next +activation continues from them. + +A is nondeterministic. How many records fit in a time slice depends on machine speed, record size, +converter cost, and load, so two runs of the same input segment differently. Segment boundaries are +recorded in the annotation (ADR-018), so replay would drain a different number of times than the +live run did, and `wait_condition` predicates would fire a different number of times — a +nondeterminism failure attributed to Workflow code that did nothing wrong. + +B does not bound an activation at all. The buffer bounds memory held ahead of delivery, not the +number of records that pass through it: the watcher refills on the Worker loop while the Workflow +thread drains, and a producer that keeps pace makes "drain until empty" a condition that is never +reached. + +Re-arming readiness is not an optimization but part of the decision. The watcher signals readiness +on arrival and has already advanced `prefetch_cursor` past the buffered records, so no further +notification is coming for them. A budget that blocked without re-arming would leave the Workflow +waiting forever on data already in front of it — trading a deadlock timeout for a silent stall, +which is worse, because the stall reports nothing. + +**The count is charged at delivery, and an activation begins already charged for what Workflow code +is holding.** Both follow from the budget being a *reservation* rather than a check, and each closes a +schedule that a plainer reading of "256 records per activation" let through. + +Charging at consumption — one record at a time, as each is handed over — reads the budget at the drain +and takes nothing, so the next subscription's drain sees the same room and takes it as well. Two +subscriptions consumed by two independent coroutines then take a whole budget each, and `n` of them +take `n`; `merge()` was immune only because it fills one record at a time, which charges and checks in +the same place. Charging at delivery also puts the count on the same quantity the annotation records, +so the cap bounds the segment replay will divide by rather than a different number. + +Resetting the count to zero per activation leaves the other half open. A batch is delivered whole and +consumed one record at a time, so a Workflow that stops iterating part-way through carries the +remainder in its ready list — where the next activation takes it with no drain and no check. Free, and +it accumulates: one subscription per activation can leave a nearly full list behind, so `n` of them +arrive holding `n` budgets between them. Starting the count at what the ready lists already hold makes +carried-over-plus-newly-delivered exactly one budget. + +## Consequences + +- **One activation is bounded by construction**, independently of producer rate, backend speed, and + buffer size. No workload can make an activation run to the deadlock timeout by volume. +- **"The budget stopped delivery" is now a conservative signal**, and the two things that read it — + re-arming readiness, and withholding immediate parkability — are safe in that direction. An + activation pre-charged for a full ready list reports the budget spent without having tried to + deliver; the re-arm is then skipped by the manager for an empty buffer, and a wait withheld from + parking is retained for its idle timeout rather than parked. Under-reporting is what loses records, + since nothing else announces what a budget left behind. +- **A record drained but never consumed still costs its slot.** It has been recorded in the + annotation, so it is delivery whatever the Workflow does with it, and the Run keeps it in hand. + Closing a subscription drops its ready list and gives the slots back, since those records can no + longer be consumed by anyone. +- The segment that exhausts the budget ends with `BATCH_LIMIT`, which is what that + `segment_end_reason` means and how replay reproduces the same split. +- **The budget does not apply during replay.** Delivery comes from the recorded segments, which + already fix how many records each activation received; re-imposing a live budget on top could only + disagree with the recording. +- A fast producer costs more activations rather than one long one. Activations within a retained + Workflow Task are cheap — they produce no History events and accumulate into one marker — so this + is paid in segments, not in Workflow Tasks. +- Changing the constant changes recorded segment boundaries for runs written after the change. + Existing markers replay from their own recorded segments, so in-flight Runs are unaffected, and + the value is a constant rather than a Worker option so that one deployment cannot split segments + two ways. +- A producer that never lets the buffer run dry must still let every activation complete, the + Workflow Task park or roll over, and the marker be written. That is the behavior to hold the + implementation to. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-027-a-late-binding-rides-its-own-frame.md b/arch_docs/streaming-poc-docs/decisions/ADR-027-a-late-binding-rides-its-own-frame.md new file mode 100644 index 000000000..06ea15a30 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-027-a-late-binding-rides-its-own-frame.md @@ -0,0 +1,55 @@ +# ADR-027 — A subscription created after the header is bound by its own frame + +**Status:** Accepted · **Affects:** P5, P10b, P13 · **Spec:** `spec/annotation-format.md` + +## Context + +The annotation opens with a header frame carrying one binding per subscription — stream key, start +cursor, and the configured provider identity. The header goes to Core in the +first observation delta of a Workflow Task. + +`subscribe()` is ordinary Workflow code, and a retained Workflow Task spans many activations, so a +subscription can be created after that delta has already been sent. Core appends observation deltas +to a byte buffer and never rewrites what it holds, so the header it already has cannot be amended. + +A wait with no binding still produces runs and a terminal entry, and replay then reads a `wait_id` +with no stream key and no start cursor — reported as a wait "the Workflow did not +create", against code that never changed. + +## Options + +**A. Refuse a subscription after the header.** Keep one binding table, written once. + +**B. Re-emit a header frame carrying the enlarged table.** No new frame kind. + +**C. A bindings frame: the header's body under its own tag, carrying only the new waits.** + +## Decision + +**C.** + +A makes an ordinary Workflow illegal. A subscription created inside a loop, after an activity, or on +a branch reached only once records arrive is normal code, and the annotation is an encoding detail +that must not be visible in what Workflow code is allowed to do. + +B is ambiguous where C is not. A header frame also carries the schema version, so a decoder meeting +two of them has to decide whether the later replaces the earlier — for the version, and for every +binding that appears in both. Both readings are defensible, which is the problem: the frame would +mean one thing to the encoder and possibly another to a decoder written against the same grammar. A +bindings frame says exactly one thing, *these waits exist too*, and needs no such rule. + +The frame is emitted with the delta of the activation that registered the wait, which puts it ahead +of both the segment that first records a run for the wait and the terminal — the two frames that can +name a `wait_id`. Decoding merges every bindings frame into the header's table, so replay reads one +flat table and never asks when a wait joined. + +## Consequences + +- `SCHEMA_VERSION` is **1**, the only format for this unreleased feature. There is no legacy decoder. +- A wait is bound **exactly once** per annotation. A second binding for one `wait_id`, or any + binding after the terminal, is a decode failure — replay must not choose between two stream keys, + since the one it chose could be the one the records were not written to. +- A late wait carries **its own** start cursor rather than the position the waits already bound have + reached, so replay begins it where the subscription began. +- The grammar is `header, (bindings | segment)*, terminal`: bindings and segments interleave, and + neither ordering between them is fixed beyond a wait being bound before it is named. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-028-decoding-is-split-across-the-boundary.md b/arch_docs/streaming-poc-docs/decisions/ADR-028-decoding-is-split-across-the-boundary.md new file mode 100644 index 000000000..bf63537b2 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-028-decoding-is-split-across-the-boundary.md @@ -0,0 +1,89 @@ +# ADR-028 — Decoding a record is split across the Worker/Workflow boundary + +**Status:** Accepted · **Affects:** P4, P19, P13, P18 · **Spec:** `spec/python-runtime.md` + +## Context + +Turning a record's payload bytes into a value means running the Workflow's own `DataConverter`, +codec included, because producer and consumer must agree on one (ADR-015). The value's type comes +from the topic, and a topic is Workflow code — so the obvious place to decode is where the record is +handed over: the subscription iterator, on the Workflow thread. + +That place cannot run a `DataConverter`. `DataConverter.decode` is three steps — external-payload +retrieval, the user's `PayloadCodec`, then `from_payloads` — and the first two are arbitrary +asynchronous work. On the Workflow thread they run inside `activate()`, under the deterministic +event loop: an `await` inside a codec becomes a **real Temporal timer command in Workflow +History**, so History depends on the codec's internals and replays only while that codec behaves +identically; a codec that blocks rather than awaits fails against the Workflow sandbox's +restrictions instead; and either way a multi-second round trip sits under the 2-second deadlock +timeout. A codec doing real I/O hits one of the three. + +The observation that resolves it: **only the third step needs the type, and that step is +synchronous.** + +## Options + +**A. Split the decode at that seam.** Retrieval and the codec run on the Worker's loop before the +record reaches the Workflow thread; `from_payloads` runs there, with the topic's type. + +**B. Decode on the Workflow thread, and send the asynchronous half back.** A delivery that finds an +unprepared record ends the activation, the Worker's loop prepares it, and readiness is re-notified. + +**C. Decode on the Workflow thread, but hop.** Submit the asynchronous half to the Worker's loop +with `run_coroutine_threadsafe` and block the Workflow thread on the result. + +**D. Keep the whole decode in the Workflow-facing API**, reaching through the runtime handle into +the manager to borrow the Worker's converter. + +## Decision + +**A.** + +B costs **an extra Workflow Task per record** — the one cost this design spends everything else to +avoid — and it cannot serve replay at all. Replay's records are delivered from a marker's recorded +segments inside a single activation, so there is no readiness round trip to re-run between them, and +they must all be prepared before the first one is handed over. + +C converts an await into a **guaranteed** deadlock-timeout failure rather than an occasional one. +The Workflow thread is the thread `activate()` runs on and the thread the 2-second timeout watches; +blocking it for a codec's round trip is precisely what that timeout exists to catch. Hopping changes +which thread waits, not whether the Workflow Task waits. + +D leaves the asynchronous work on the Workflow thread — it changes where the converter comes from, +not where it runs — and it makes the Workflow-facing module depend on the manager's internals. That +module knows its runtime only as an opaque handle exposing named methods, which is what keeps every +Worker-side object out of Workflow code's reach; reaching past it for a field is the first crack in +that. Under A the handle grows one method instead, and what it hands back is a codec already bound +to the topic's type and capable only of the synchronous half. + +## Consequences + +- **Preparation has exactly two sites**, because a record reaches Workflow code exactly two ways: + the watcher, before a record enters a subscription's buffer, and the replay plan's segments, when + the replay job is prepared. A third delivery path would need a third preparation site, and one + that forgot would be caught rather than tolerated — see the refusal below. +- **A preparation failure is data, not an exception.** It is carried on the record and raised by the + delivery that would have yielded its value. Raised where it happened it would end the watcher for + the whole Run, taking every later record with it, and would fail a Workflow Task over a record + Workflow code might never have asked for. +- **An unprepared record on the Workflow thread is refused, not converted**, whenever the + converter has a payload codec or external storage. A codec's output is just another payload, so + the difference is invisible in the bytes and converting anyway yields a *plausible wrong value* — + worse than a loud failure, because nothing reports it. A converter with neither has an empty + asynchronous half and still converts correctly, so the refusal is conditional on the converter + rather than on the path. +- **Preparation reaches two private `DataConverter` helpers** — + `_external_retrieve_payload_sequence` and `_decode_payload_sequence`. `decode()` is those two + calls followed by `from_payloads`, so the split does byte-for-byte the same work in the same + order; but it is coupled to internals. A public seam exposing everything except the payload + converter is the cleaner long-term home, and would make this a supported split rather than a + tolerated one. The condition deciding whether an unprepared record may be converted is asked of + the converter's two public members instead, because the equivalent private predicate is marked in + its own source as temporary: a safety property must not stop holding the day an unrelated cleanup + lands upstream. +- **The type never leaves Workflow code and the converter never enters it.** The manager prepares + with no type hint at all — sound precisely because the payload converter is the only step that + takes one. +- Every record read is prepared, including ones Workflow code never takes. That is the same bargain + `decode_activation` already makes for an activation's payloads, and it costs one codec call per + record later discarded. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-029-a-closed-subscription-keeps-its-state.md b/arch_docs/streaming-poc-docs/decisions/ADR-029-a-closed-subscription-keeps-its-state.md new file mode 100644 index 000000000..ebd2b171b --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-029-a-closed-subscription-keeps-its-state.md @@ -0,0 +1,70 @@ +# ADR-029 — A closed subscription keeps its recorded state + +**Status:** Accepted · **Affects:** P9, P8, P10b, P15 · **Spec:** `spec/python-runtime.md` + +## Context + +`close()` ends a wait: iteration stops, the wait leaves the quiescent snapshot, and the Worker stops +serving it. The symmetric teardown is to undo what `subscribe()` did — drop the runtime's record of +the subscription along with the manager's — and that is what "the subscription is over" appears to +mean. + +The runtime's record is not only a live wait, though. Two things are built from the set of registered +subscriptions at moments that have nothing to do with any individual wait still being open: the +annotation binding the runtime emits for each wait, which replay reads, and the per-wait cursor in +the Continue-As-New continuation, which a successor Run resumes from. Both are consulted at the +*Run's* boundaries. A Workflow that reads a stream to its fence and closes it reaches both of them +afterwards. + +## Options + +**A. Delete the state**, in the runtime and in the manager alike. + +**B. Keep the runtime's state and mark it closed**; drop only the manager's. + +**C. Delete the state and keep a separate record** of the binding and cursor a closed wait still +owes. + +## Decision + +**B.** + +A is durably wrong in two directions at once, and in the ordinary case rather than an exotic one. A +wait with no binding appears in the annotation as a `wait_id` with no stream key, no backend, and no +start cursor, which replay reports as a wait the Workflow did not create (ADR-027) — against code +that never changed, for a Workflow whose only unusual act was to stop reading a stream it had +finished with. And a successor Run reconstructs that same wait from the same deterministic +`subscribe()` call, so a continuation missing its cursor restarts the stream at `BEGINNING` and +re-delivers every record the closed subscription consumed. Closing is the point at which a cursor is +most worth keeping, not least: nothing later will move it, so it is final rather than stale. + +C keeps the same two fields under a second name. Every reader of the registered set becomes a reader +of two sets that must be merged, and a reader that consults only the live one is wrong only after a +Workflow has closed a subscription — rare, silent, and durable in a marker or in a successor's +starting position. One set with one flag makes the correct reading the default: a reader that never +looks at `closed` still reads the right binding and the right cursor. + +## Consequences + +- **The two halves of teardown are deliberately asymmetric.** The manager's entry holds resources — a + watcher, a buffer, a backend connection, and possibly a park intent (ADR-030) — and is dropped in + full. The runtime's entry holds the record of what the wait *was*, and is kept. They are not two + copies of one thing, and only one of them costs anything to hold. +- **Closing changes exactly one property of the state**: the wait can never be blocked again. That is + what keeps a closed wait out of Core's hands. A wait that re-entered the blocked set would be + registered, retained for, and eventually parked on behalf of a coroutine that no longer exists, and + only a wake would end the Workflow Task it held open. +- **The teardown is requested through the runtime handle, not started from the subscription.** + `close()` runs on the Workflow thread, and a task created there is not scheduled at all, so the + request would be lost without a trace and the watcher would keep running. This is ADR-011's rule + reached from the other side: Worker-side work is not merely misplaced on the Workflow thread, it + does not happen. +- `wait_id` is allocated from a per-Run counter and never reused, so a closed entry can never be + confused with a live one and no reader has to tell them apart by identity. +- A Run that subscribes and closes repeatedly accumulates one kept entry per iteration. That is the + growth its annotation already has — each new wait must be bound in it regardless — so retention + introduces no new bound, and a Workflow whose subscription count grows without limit is already an + annotation-size problem (ADR-007). +- The continuation and multi-stream tests must close a subscription and still find its binding in the + annotation and its cursor in the continuation, because the failure of A is invisible until a + Continue-As-New or a replay happens. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-030-cancellation-removes-its-own-park-intent.md b/arch_docs/streaming-poc-docs/decisions/ADR-030-cancellation-removes-its-own-park-intent.md new file mode 100644 index 000000000..ef3774b75 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-030-cancellation-removes-its-own-park-intent.md @@ -0,0 +1,69 @@ +# ADR-030 — Cancelling a subscription removes its park intent itself + +**Status:** Accepted · **Affects:** P8, P2b, P9 · **Spec:** `spec/wft-lifecycle.md` + +## Context + +A park intent is durable backend state; the record of which Worker installed it is not. Two +enforcement points already remove one (`spec/wft-lifecycle.md`): the resolve that ends a park this +Run is sitting in, and the reconciliation a Worker performs when it registers a wait it finds an +intent for. Both are driven by a registered subscription — the resolve iterates the ones the manager +holds, the reconciliation fires as one is registered. + +Cancelling a subscription is the one event that removes what both of them read. Whatever it does not +do about the intent, nothing else is going to do later. + +## Options + +**A. Leave it to the two existing points.** Cancellation stops the watcher and drops the buffer only. + +**B. Remove the intent in the cancel path**, before the watcher is stopped. + +**C. Remove it at Run teardown**, along with the buffers and connections. + +## Decision + +**B.** + +A has nothing left to work from. A resolve removes the intents recorded for subscriptions the manager +still holds, and the cancelled one is no longer among them; the reconciliation fires when a wait is +registered, and a closed wait is not registered again for the life of the Run. Reconciliation does +reach the intent eventually *if* the Run is evicted and rebuilt by a replay that recreates and +re-registers the subscription — but that is a coincidence of eviction, not a mechanism, and a Run +that stays cached to completion never provides one. + +C narrows the window to the length of the Run instead of closing it, which is the same defect +measured differently: a Workflow that closes a subscription in its first activation and then runs for +a week advertises a park for a wait that ended a week earlier. + +What the window costs is not a leak but a lost wake, and it is exactly the failure the intent +invariant exists to prevent. `current_park_generation` keeps answering a generation Core has +discarded; a producer that reads it names that generation in its wake Signal, and Core drops the +Signal as stale. The record is appended, the Signal is sent, the producer's `publish()` returns +successfully, and the Workflow is never woken — with the retry behaviour and the request-ID +deduplication that make this permanent described under the same invariant in `spec/wft-lifecycle.md`. + +The asymmetry with the other two points is the whole argument. They can afford to log a failed +removal and be reached again later, because the subscription they work from is still registered and +both will run again against it. Cancellation is the removal of that registration, so a failure there +has no successor. + +## Consequences + +- **The intent goes before the watcher.** A cancellation interrupted between its two steps — by a + crash, an eviction, a Worker exiting — leaves either a stopped watcher and a permanent intent, or a + removed intent and a watcher that the Run's own teardown will stop regardless. Only one of those + orders has a recoverable failure in the middle of it. +- **Removal is serialized against this Run's parking and resolving**, on the same terms as + reconciliation, so a cancellation racing a park in flight is ordered before or after it and cannot + remove an intent that park is in the middle of installing. +- **A removal that fails is retried, and then owed; the cancellation proceeds regardless.** Holding + the cancellation open on the backend would keep a watcher, a buffer, and a connection alive for a + wait the Workflow has already ended, and would not make the intent go away. But dropping the + subscription must not drop the obligation with it, which is what a removal recorded on the + `Subscription` would do: the removal is bounded-retried here and whatever is left is carried by + the Run's owed-removal ledger, which outlives both the subscription and this cancellation + (ADR-031). +- A test that closes a subscription with an intent installed must assert the intent is **gone from + the backend**, not merely that the watcher stopped; the two are separable and only one of them is + durable. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-031-an-owed-removal-belongs-to-the-run.md b/arch_docs/streaming-poc-docs/decisions/ADR-031-an-owed-removal-belongs-to-the-run.md new file mode 100644 index 000000000..8201dc4a2 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-031-an-owed-removal-belongs-to-the-run.md @@ -0,0 +1,137 @@ +# ADR-031 — An owed park-intent removal belongs to the Run, not to the subscription + +**Status:** Accepted · **Affects:** P8, P2b, P9 · **Spec:** `spec/wft-lifecycle.md` + +## Context + +A park intent is durable backend state. Everything that knows one exists is not: the +`installed_park_generation` a Worker keeps is a mirror of a park *that Worker* confirmed, and an +intent inherited from an evicted Run or a departed Worker is mirrored nowhere at all. A removal +keyed on the mirror alone therefore does nothing — silently, and by returning early — for exactly +the class of intent that no other mechanism can reach. + +Every path that removes an intent reaches it through a `Subscription`: the resolve iterates the ones +the manager still holds, the rollback walks the ones the attempt just installed, the cancellation +works on the one it is dropping. So a removal that fails has nowhere to be recorded that survives +the next step. The close drops the subscription, the eviction drops the Run's map, and a failure +written onto either is a failure written onto the object about to be discarded. + +The two of those are one problem: cleanup responsibility is being stored on the thing whose +disappearance is what made cleanup necessary. + +## Options + +**A. Record the failure on the `Subscription`** and retry the next time that wait is registered. + +**B. A per-Run ledger of owed removals**, keyed `(stream key, wait_id)`, with lifecycle-triggered +drains. + +**C. A background retry task** owned by the manager, retrying on its own schedule. + +**D. B, but drain unconditionally** — delete the key rather than re-reading it first. + +## Decision + +**B and C, using provider-atomic compare-and-delete.** + +A is the shape that produced the defect. "The next registration" is a coincidence of eviction rather +than a mechanism: registration happens once for the life of a cached Run, so a Run that stays cached +never provides another one, and the Run that most needs the removal — cached and blocked on +something other than this stream — is precisely the one that never registers the wait again. For a +cancelled wait it is not even a coincidence; a closed wait is never registered again by definition. + +D is unsafe in a way that is worse than the leak it would be closing. `wait_id` restarts at 1 in a +Continue-As-New successor while the stream key does not change, so an entry a predecessor Run left +behind can name the key of an intent a **successor** installed and is currently parked on. Deleting +that unparks a Run whose park is real, whose producers therefore send nothing, and whose Workflow +Task no wake will ever create. A leaked intent costs wakeups for one stream; this costs the Run. So +the provider atomically removes an intent only when the `park_generation` and Run ID both still +match what was recorded, and the manager forgets the entry on either a removal or a mismatch. + +B alone was the original decision, with C deferred until cleanup had an owner that teardown awaits +and the provider contract could close the read/delete race. Both prerequisites now exist. The +manager strongly holds one retry task per Run, retries with bounded exponential backoff, and cancels +and awaits those tasks during Worker shutdown. The conditional provider operation makes the +autonomous retry safe across a Continue-As-New successor replacing the same key. Lifecycle drains +remain eager fast paths, but another Workflow or Core event is no longer required: a backend that +recovers while the Run stays cached and idle is enough to retire the stale intent. + +## Consequences + +- **A ledger entry means "a removal was decided on and has not been confirmed", not "an intent + exists".** That is what makes draining safe from any holder of the Run's park lock rather than + only from the path that recorded it. The entry also retains nothing a teardown could invalidate — + no watcher, no buffer, no connection beyond the backend the removal must go through. +- **The entry is made before the backend call**, not when one fails. A call that never returns — + the backend raised, the task was cancelled mid-await — owes the removal exactly as an error does, + and the subscription it was reached through is frequently dropped in the same breath. +- **Reading an inherited intent is what records it.** It is the only mirror such an intent ever + gets; without it, the resolve, the rollback and the cancellation all go on short-circuiting on an + empty `installed_park_generation` for it. +- **A fresh install at the same key supersedes what is owed there**, or a removal recorded against a + generation that has since been overwritten would be retried against the park now sitting behind + it. +- **Eviction ends cache ownership, not cleanup ownership.** The manager retains the ledger, its + park lock and its retry task until the debt is retired. Shutdown cancels and awaits retry tasks + within its grace period, and then makes one last bounded pass at what is still owed — a loop + cancelled during its backoff has an attempt left in it, and eviction stands aside while shutting + down rather than waiting on a lock a stuck loop may hold. +- **An autonomous retry has to be bounded per call, not per attempt.** It is the only thing left + holding a removal once the inline attempts are spent, so a backend call that hangs rather than + raises does not cost one attempt — it ends the mechanism, silently, and holds the Run's park lock + while it does. Every park-lock hold outside an activation is therefore time-bounded, and a hold + that runs out of time is a failed attempt. Holds an activation takes are not, because that wait is + the backend exposure the park handshake already owns. +- **Cleanup announces what the stale intent silenced, and that obligation is not the + subscription's either.** Wakes sent while the intent was installed named a generation Core + discards, so removing it stops the suppression without delivering the record it suppressed. With + the wait still registered, local readiness announces it — the reconciliation's first-time success + included, which is the *ordinary* path for an inherited intent and the one that never touches the + ledger. With the wait gone, nothing about the record is knowable from here: the buffer went with + the subscription that the close or the eviction dropped, and a remote producer's append into the + suppression window was never visible from this Worker at all. So a confirmed removal with no + subscription behind it sends one *unconditional* unparked wake. Conditioning it on a registered + subscription is the same mistake this ADR is about, one level up: it keeps the leak fixed and puts + the record loss straight back, for exactly the intents whose owner is gone. The cost is an empty + Workflow Task when nothing had arrived; the alternative is silence when something had. +- **No wake this Worker sends names an intent it has already decided to remove.** A ledger entry + says that generation is one Core has discarded, so a wake composed from it is a Signal the service + accepts and Core ignores — a success the sender reports and the Workflow never sees. The manager + therefore decides what a wake names, rather than the sender reading `current_park_generation` + itself, and answers "unparked" both for an entry in the ledger and for a handoff whose intent was + just retired. That is what makes the shutdown sweep's accounting true: the sweep wakes the + subscriptions of a Run whose stale intents the final removal pass has not reached, and a counter + that reported a clean shutdown on the strength of an obsolete-generation Signal was reporting a + handoff nobody received. +- **The sweep's wake and the removal's handoff are one obligation.** Both are the unparked wake for + the same wait, and the sweep runs first, so a successful sweep wake is recorded on the ledger entry + and the final pass does not send it again. Recorded only on success — a wake the sweep could not + get acknowledged leaves the handoff still owed. +- **A cleanup handoff is work of its own, and shutdown waits for it.** Every drain reaches it holding + the Run's park lock, under a budget that bounds one single-key backend call rather than a Signal + with retries behind it. The last of these is created by the final owed-removal pass, which nothing + runs after, so shutdown awaits them within what is left of its grace period and counts what the + bound cuts off. A wake merely scheduled as the process exits is a wake never sent, and worse than + one reported lost. +- **The provider reports three outcomes, and two of them mean "gone".** A delete that commits and + loses its reply leaves a retry meeting a key it cleared itself, so *absent* and *mismatch* cannot + be one answer: the first ends the suppression and may owe an announcement, the second is a park to + leave alone. Collapsing them costs a record, not an intent. +- **Discovery is retried like the removal, and bounded by the subscription.** The ledger holds an + identity, so a read that never succeeds records nothing at all — no entry, no task, no owner. The + reconciliation therefore keeps retrying its read with backoff while this Worker holds the + subscription, and hands over to the ledger as soon as one succeeds. It stops at cancellation + because the justification for removing what is at that key is holding the Run; without it, the + intent found there could be a park another Worker is in. +- **The provider closes the cross-Run race.** Two Runs of a chain hold different park locks and may + be held by different Workers, so nothing process-local orders a predecessor's drain against a + successor's install. `remove_park_intent_if_matches` compares and deletes atomically. +- Tests assert autonomous recovery after both resolve-time and inherited-intent failures, cleanup + continuing after eviction, shutdown owning an in-flight retry, and a conditional removal leaving + a replacement intent intact. Three more cover the mechanism itself: a removal call that hangs + rather than raises does not end the retry loop or keep the park lock, shutdown makes an attempt a + cancelled backoff would have skipped, and a confirmed removal re-announces the record its intent + had silenced. Four cover the handoff once the subscription is gone: a removal confirmed after + eviction still sends the unparked wake, the sweep does not compose one from an intent it has + decided to remove, a removal the final pass confirms carries its own handoff and shutdown does not + exit before the Signal lands, and the sweep's wake is not sent a second time by that pass. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-032-park-rollback-catches-cancellation-and-is-not-shielded.md b/arch_docs/streaming-poc-docs/decisions/ADR-032-park-rollback-catches-cancellation-and-is-not-shielded.md new file mode 100644 index 000000000..28e5a533c --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-032-park-rollback-catches-cancellation-and-is-not-shielded.md @@ -0,0 +1,65 @@ +# ADR-032 — Park rollback catches cancellation, and is not shielded + +**Status:** Accepted · **Affects:** P8, P2b · **Spec:** `spec/wft-lifecycle.md` + +## Context + +The park handshake installs an intent per subscription and then rechecks every stream in the set. An +attempt that does not reach a confirmed result must take every intent it installed back out: +producers read those intents, so a park visible for a generation Core never confirmed is one they +send wakes against and Core discards as stale, and the eviction that follows takes with it the only +record that the intents exist. + +Failure is not the only way an attempt ends early. Cancellation is a normal termination path here — +Core withdrawing the activation, the Worker shutting down, the task being torn down — and in Python +`CancelledError` derives from `BaseException`, so it does not pass through a handler written for +`Exception`. Neither do `KeyboardInterrupt` and `SystemExit`. + +## Options + +**A. Catch `Exception`,** re-raising cancellation ahead of the handler so it is visibly not rolled +back. + +**B. Catch `BaseException`** and roll back inline, unshielded. + +**C. Catch `BaseException` and shield the rollback**, so it completes even if cancellation is +delivered again. + +**D. Leave a cancelled park to the resolve, the reconciliation, and the owed-removal ledger.** + +## Decision + +**B.** + +A abandons exactly the case worth catching. The likeliest way a half-installed park is abandoned is +a Worker on its way down, which is also the case with the least recoverable aftermath: the mirror of +what was installed leaves with the process, so the next Worker can only reach those intents through +registration-time reconciliation of a wait that may never be registered again. A handler that +re-raises cancellation ahead of the rollback reads as deliberate care and does nothing whatsoever, +which is worse than no handler, because it answers the question for the next reader. + +C is the tempting refinement and is wrong in a specific way. A shielded rollback detaches from the +cancelled task and can complete *after* `prepare_park` has released the Run's park lock. That lock +is what orders a rollback against the next park, so a rollback outliving it can remove an intent a +newer and entirely legitimate park has just installed — an unparked Run produced by the cleanup +path, which is the same hazard the owed-removal ledger's generation check exists for (ADR-031), +arrived at from the other side. Against that, the shield buys little: a single `Task.cancel()` still +lets an already-running handler's `await` complete, so it only protects against repeated +cancellation, and what repeated cancellation leaves behind is owed rather than lost. + +D has nothing to work from. An intent installed by an attempt nobody rolled back is recorded only on +the subscription that attempt was about to abandon; the ledger holds what a removal *decided on* +leaves behind, not what was never attempted. + +## Consequences + +- **The rollback is total and its failures are swallowed.** Every subscription the attempt installed + is attempted regardless of what the others do, and the error that reaches the caller is the one + that ended the attempt — a removal that fails there stays owed for the Run's next drain. +- `KeyboardInterrupt` and `SystemExit` take the same path, which is wanted: they are the interpreter + going away, which is when a leaked intent is least recoverable. +- **Nothing in the handler may block for long.** It runs on a cancelled task, so a second delivery + cuts it short, and whatever is left over is owed rather than retried in place. +- A test must cancel `prepare_park` between two installs, await the `CancelledError`, and assert + that no intent for the attempted generation survives — asserting on the exception alone passes + against the shape this decision rejects. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-033-replay-checks-bound-waits-one-directionally.md b/arch_docs/streaming-poc-docs/decisions/ADR-033-replay-checks-bound-waits-one-directionally.md new file mode 100644 index 000000000..01e868a5e --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-033-replay-checks-bound-waits-one-directionally.md @@ -0,0 +1,69 @@ +# ADR-033 — Replay checks that every bound wait was recreated, in one direction only + +**Status:** Accepted · **Affects:** P5, P10b, P13 · **Spec:** `spec/annotation-format.md` + +## Context + +Replay joins a recorded delivery to a subscription by `wait_id` and nothing else, and `wait_id` is a +counter over `subscribe()` call order. Two checks reason from what the code did: each registered +wait is compared against its recorded binding, and a replay that ends holding a recorded delivery +nobody took fails. Neither says anything about a recorded wait that the code no longer creates, +because both start from the subscriptions that exist. + +A binding is written for a subscription whether or not anything was delivered through it: the first +observation must carry provider identity, stream key, and start cursor even for a stream that stayed +quiet for the whole Workflow Task. A binding with no runs behind it is therefore the ordinary shape +of a quiet `subscribe()`, not an edge case — and it is invisible to any check that reasons from +deliveries. + +## Options + +**A. Nothing further.** The binding comparison and the unconsumed-delivery check are the whole of +it. + +**B. `bound ⊆ registered`,** checked once after the last segment of the marker. + +**C. Set equality** — every bound wait registered *and* every registered wait bound. + +**D. B, but checked before each segment**, alongside the unconsumed-delivery check. + +## Decision + +**B.** + +A leaves two removals accepted, and the second is the dangerous one. Removing the **last** +`subscribe()` on a quiet stream renumbers nothing and leaves nothing undelivered, so replay +completes against a Workflow that now holds one subscription fewer than History says it did, with +that stream's live reads never made. Removing a **middle** `subscribe()` where the later waits name +the same stream and backend renumbers every survivor down by one, so the binding comparison compares +equal for all of them and the records recorded for wait *k* are consumed by what was subscription +*k+1* — a different cursor, a different consumer, and nothing left over for the delivery check to +notice. That is the silently different stream result the failure taxonomy says must never happen. + +C forbids a supported change. Replay runs the Workflow **forward past** the Workflow Task the marker +covers, and the subscriptions it makes there belong to the next marker's header, so a registered +wait with no binding is the ordinary case rather than a disagreement. Under C, adding a `subscribe()` +at the end — the one subscription change the determinism rule explicitly permits — would fail +against every marker written before it. + +D is C's timing mistake without C's semantics. A wait bound by a later frame legitimately does not +exist while an earlier segment is being delivered, so the same check run per segment reports +nondeterminism against unchanged code whenever a `subscribe()` ran after the first activation of a +retained Workflow Task (ADR-027). + +## Consequences + +- The failure is row four of `spec/failure-taxonomy.md` — nondeterminism, naming the missing waits — + and not integrity loss. The recorded ranges are exactly where they were written; it is the + Workflow code that moved, and sending an operator to repair the backend would be the wrong + instruction. +- **A closed subscription must stay in the registered set** (ADR-029). It was created, so the marker + binds it; a close that forgot it would fail this check against a Workflow whose only unusual act + was to finish reading a stream early. +- **One residue stays undetected**, and it is the price of permitting an addition at the end: + inserting a `subscribe()` in the *middle* of several subscriptions to the same stream on the same + backend renumbers them without changing any binding comparison and without changing the bound set's + membership in the registered one. The determinism rule covers it; no check can. +- A test must replay a marker that binds a wait with **no runs at all** against Workflow code that + creates no subscription, and require the error. A test built on a wait with recorded deliveries + passes against option A. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-034-merge-rotates-its-start-position.md b/arch_docs/streaming-poc-docs/decisions/ADR-034-merge-rotates-its-start-position.md new file mode 100644 index 000000000..62cd6fbe5 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-034-merge-rotates-its-start-position.md @@ -0,0 +1,74 @@ +# ADR-034 — `merge()` rotates its start position across passes + +**Status:** Accepted · **Affects:** P11, P13 · **Spec:** `spec/python-runtime.md` + +## Context + +`merge()` iterates several subscriptions as one wait, taking at most one record from each per pass, +in `wait_id` order. The `wait_id` order is what makes the interleaving reproduce on replay; the +one-record limit is what makes it a merge rather than a priority order. + +The delivery budget, however, is charged per record handed to Workflow code and covers the merged +set as a whole (ADR-026). A pass can therefore be cut anywhere inside it, and where it is cut is +decided by where it began. + +## Options + +**A. Begin every pass at the lowest `wait_id`.** One record each, no cursor. + +**B. A rotation cursor local to the generator**, resuming after the subscription that last took a +record. + +**C. Cap the number of subscriptions a merge accepts** below the delivery budget, making A's claim +true. + +**D. Record the rotation in the annotation** so replay reproduces it, or give each subscription its +own budget. + +## Decision + +**B.** + +A fails twice, and the second failure is the general one. With more continuously ready subscriptions +than the budget has records — 257 against 256 — the pass is cut in the same place on every +activation and the last subscription is never reached at all. It is not merely served nothing: +filling returns on the spent budget before it consults the manager, so the Worker is never asked +whether that stream holds anything, for the life of the Run. And starvation is only the extreme of a +defect that needs no cap to be exceeded: any count that does not divide the budget leaves the pass +cut mid-way, so 100 always-ready waits against 256 records give the first 56 one record per +activation more than the other 44, forever, with the gap growing by one per activation and nothing +bounding it. The single-record skew A is credited with is a property *within* a pass; only resuming +where the last pass stopped makes it a property across them. + +C is a limit on user code imposed to make an implementation's fairness claim true, and it does not +even make it true: a count comfortably under the budget still fails to divide it, which is the +unbounded case above. It would also have to be enforced somewhere, turning a merge over many streams +into an error rather than a slower merge. + +D pays for something already free. A per-subscription budget multiplies an activation's length by +the number of streams, which is ADR-026's deadlock with a larger constant in front of it. Recording +the cursor promotes a fairness heuristic into replay state that every later change to the heuristic +would have to version — and it is unnecessary, because ask order provably cannot change what a +replayed segment yields. + +## Consequences + +- **The cursor is not replay state and is not reconstructed.** Under replay the budget does not + apply, so passes are not cut where they were cut live and the rotation reaches positions the live + run never started from. That is safe because a replay drain serves from the front of the recorded + segment and only while the front belongs to the asking wait: a wait asked out of turn is told + nothing and the record stays for whoever asks next. Every active wait is still asked exactly once + per pass, so the front's owner is reached in every pass and the recorded global order comes out + whole whatever position the pass started at. +- **"Asked exactly once per pass" is load-bearing** and is pinned by a test that replays a segment + interleaving three waits and repeating one. Any later variant that lets a pass skip a wait — an + early return on a spent budget, a per-subscription budget — would let a replayed segment be + yielded in an order the live run never delivered. +- Post-replay the cursor generally sits somewhere other than where the live run left it. It steers + only future **live** fairness, and no live schedule was ever recorded for History to contradict. +- **A control record advances the rotation**; a subscription that had nothing does not. The former + spends its turn and the budget, because it occupies an offset inside a run. Advancing past the + latter would cost a wait the turn it never got, which is the starvation this decision ends, + reintroduced from the other side. +- A closed subscription leaves the set, and a cursor naming it wraps to the front rather than + pinning the pass. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-035-both-sides-bind-the-workflows-serialization-context.md b/arch_docs/streaming-poc-docs/decisions/ADR-035-both-sides-bind-the-workflows-serialization-context.md new file mode 100644 index 000000000..1bb1c3dcf --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-035-both-sides-bind-the-workflows-serialization-context.md @@ -0,0 +1,76 @@ +# ADR-035 — Both sides of a stream bind the Workflow's serialization context + +**Status:** Accepted · **Affects:** P4, P6, P18 · **Spec:** `spec/python-runtime.md` + +## Context + +A `DataConverter` component may implement `WithSerializationContext`, which exists so that a codec or +payload converter can behave differently per Workflow — an encryption key derived from the Workflow +ID is the reason the interface is there. Everywhere else in the SDK the Worker binds a +`WorkflowSerializationContext(namespace, workflow_id)` before converting a payload for a Workflow, +on both the asynchronous and the synchronous side. + +An external stream record is a payload delivered to a Workflow, converted by the same +`DataConverter`, in the same activation as the payloads that do get the binding. Producer and +consumer are required to share that converter (ADR-019), and a stream is written by a producer and +read by a consumer that may be a different process, a different Worker, and a later Run of the chain. + +Both sides were context-free. That is the fact this decision turns on: they were **internally +consistent**, so every deployment with a context-aware converter was working. + +## Options + +**A. Leave both sides context-free.** Consistent, and no deployment breaks. + +**B. Bind the consumer only.** The defect is on the consumer's decode path, so fix it there. + +**C. Bind both sides**, from the same Workflow identity. + +**D. Bind with `_with_contexts`**, adding the storage driver's store context alongside. + +## Decision + +**C**, and the symmetry is the whole of the argument. + +A is not tenable once a Workflow reads a stream record and anything else in the same activation. The +Workflow's own argument arrives bound and the record does not, so a converter that derives anything +from the Workflow sees two different worlds one line apart. The consequence is not a clean error: a +context-aware codec that refuses without a context fails during preparation, the failure is carried +on the record and raised at delivery, and the Workflow Task retries forever against a record that +will never decode — reported as row three of `spec/failure-taxonomy.md`, whose operator instruction +is to align the consumer's converter with the producer's, when they already are aligned. + +**B is the trap.** It reads as the minimal fix and it breaks every currently working deployment +whose codec keys on the Workflow: the producer would encrypt with no context while the consumer +decrypts with a Workflow-derived key, so records that appended successfully become undecodable on +the far side. Two context-free sides agree; two bound sides agree; one of each is the only +combination that does not. A defect that is symmetric has to be fixed symmetrically or not at all. + +D binds something untrue. The store context names the Workflow as the payload's *storer*, and a +stream record is stored by its **producer** — an Activity or a plain process — not by the consuming +Run. Nothing on the consumer's decode path reads it in any case: retrieval resolves the driver from +the claim embedded in the payload rather than from a store context. + +## Consequences + +- **The identity is `workflow_id`, never a Run ID**, on all three holders. A stream spans a + Continue-As-New chain, so a successor Run decodes records its predecessor wrote; anything + Run-scoped makes a chain's own records unreadable from its first continuation onward. +- **Who holds a bound converter follows from what that holder is**, and getting it wrong is not + cosmetic. The runtime is per Run and is built bound — on the Worker's side, because it crosses + into the sandbox and `with_context` runs user code. The manager is per Worker and must **not** + hold one: it prepares records for every Run at once, so it derives the context per record from the + record's own stream key and memoizes the clones for that one call rather than on itself, where + they would accumulate an entry per Workflow ID the Worker ever served. The producer serves one + chain and binds once. +- **The ordinary deployment sees no change.** `with_context` returns the converter unchanged unless + a component implements `WithSerializationContext`, so a default converter is not cloned and not + altered. +- **The producer's store context stays empty**, naming no target, so a driver selector that routes + by target has nothing to route on. That is a store-context gap rather than a serialization-context + one, it is not needed for the symmetry above, and closing it requires deciding what target an + Activity-hosted producer should name — which would change where blobs land. +- A test that proves this must use a converter whose behaviour actually **differs** by context. One + that merely records what it was handed passes against option A on both sides, and a round trip + through two context-free sides passes as well; only a per-Workflow transformation separates + "both bound to the same context" from "both bound to nothing". diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-036-cancellation-after-a-durable-append-is-an-unacknowledged-wake.md b/arch_docs/streaming-poc-docs/decisions/ADR-036-cancellation-after-a-durable-append-is-an-unacknowledged-wake.md new file mode 100644 index 000000000..629d4a816 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-036-cancellation-after-a-durable-append-is-an-unacknowledged-wake.md @@ -0,0 +1,87 @@ +# ADR-036 — Cancellation after a durable append is reported as an unacknowledged wake + +**Status:** Accepted · **Affects:** P6b, P14 · **Spec:** `spec/wake-signal.md` + +## Context + +This record is about what happens *after* an acknowledged append; the append's own acknowledgement +window is ADR-038. + +Once the append is acknowledged, `publish()` distinguishes two outcomes: the append succeeded and the +wake did not, or both did. Every failure after the append carries the durable offset and names one +recovery, which +is what stops a caller re-publishing a record that already landed — a second `publish()` draws a new +sequence number, and therefore a new idempotency key, so it appends the value twice. + +Cancellation was outside all of that. `CancelledError` derives from `BaseException`, so it passed +through the coordination handler (which re-raised it ahead of the conversion, deliberately), through +the Signal loop (which caught `Exception`), and through `publish()` itself (which catches only the +durable-but-unacknowledged error). What reached the caller was a bare `CancelledError` with no +offset, no `pending` and no `restart` — indistinguishable from cancellation *before* the append, and +so unrecoverable in both directions at once: the caller could not wake a record it had not been told +about, and could not safely re-publish the value either. + +Cancellation here is ordinary, not exotic. An Activity being cancelled, a Worker shutting down, and a +`asyncio.timeout` around a publish all produce it, and all of them can land after `append()` has +returned. + +## Options + +**A. Re-raise the bare `CancelledError`.** The status quo. + +**B. Defer cancellation through the post-append section**, completing the wake and then re-raising, +so the state is clean and the cancellation is honoured. + +**C. Shield the wake**, letting it finish detached from the cancelled task. + +**D. Raise a cancellation-specific exception** carrying the offset and the recovery — a new type the +caller has to learn. + +**E. Report it as the existing durable-but-unacknowledged outcome**, with a flag saying cancellation +is what ended the attempt. + +## Decision + +**E.** Any `Exception` *or* `asyncio.CancelledError` raised after a durable append leaves +`publish()`, `finish_writing()` and `wake()` as `WakeNotAcknowledgedError` carrying `.offset`, one of +`pending`/`restart`, and `cancelled=True` when cancellation was the cause. `KeyboardInterrupt` and +`SystemExit` are not converted: they are the interpreter going away, and there is no caller left to +recover. + +A is what the review found. The two states it fuses — nothing appended, and appended but +unannounced — have opposite recoveries, and the caller cannot tell them apart. + +B and C both try to make the problem disappear by finishing the wake, and neither can promise it. The +handler would run on a cancelled task, so a second delivery cuts it short and the ambiguous state is +back; ADR-032 already establishes that nothing on a cancelled path may block for long, and the wake's +third step is a Signal RPC. C additionally detaches: a shielded wake outlives the call that owns it, +so `publish()` returns or raises while a Signal it can no longer report on is still in flight — the +same "someone else may be sending" that the acknowledged-wake contract exists to refuse. + +D is E with a second name for one state. The state *is* the durable-but-unacknowledged one — durable +record, unsent wake, one recovery — and the caller's existing handler already knows what to do with +it. A separate type would mean every caller writing the same recovery twice, and a caller that +forgot the second one would be back to the ambiguity this record is about. + +The cost of E is real and is accepted: cancellation is refused rather than propagated, so a +`Task.cancel()` ends the task with this error instead of as cancelled, and an `asyncio.timeout` +wrapped around a publish will not convert to `TimeoutError`. `cancelled` is what makes that +recoverable — a caller that wants to honour the cancellation re-raises after recovering the wake, +which is the only order that leaves nothing owed. + +## Consequences + +- **The post-append section has no exit that says nothing.** Every way out of it names the offset and + one recovery, cancellation included. +- Cancellation delivered *before the backend is called* still raises `CancelledError`. Nothing was + sent, so there is no offset to carry and nothing to recover; reporting an unacknowledged wake there + would send the caller to wake a record that does not exist. This record originally said "before the + append", which assumed an append that did not return had not happened — true of a local backend and + false of a remote one. **ADR-038 corrects that half**: cancellation delivered while `append()` is in + flight is its own outcome, `AppendNotAcknowledgedError`, because the record may already be durable. +- `wake()` converts cancellation even though it can be called without an append of its own, because + it is only ever reached after one — `wake=False` followed by one `wake()` is a batch whose records + are already durable. +- A test must cancel a publish *after* the append and assert the error carries the offset and exactly + one recovery, perform that recovery, and find one record in the stream. Asserting on the exception + type alone passes against a conversion that dropped the offset. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-037-a-subscription-has-one-consumer.md b/arch_docs/streaming-poc-docs/decisions/ADR-037-a-subscription-has-one-consumer.md new file mode 100644 index 000000000..31cde5f9c --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-037-a-subscription-has-one-consumer.md @@ -0,0 +1,72 @@ +# ADR-037 — A subscription has one consumer, and the refusal is on the waiter + +**Status:** Accepted · **Affects:** P9, P21 · **Spec:** `spec/python-runtime.md` + +## Context + +`__aiter__` returns a new async generator on every call, but the generators are not independent views: +the cursor, the readiness future, and the blocked flag all belong to the subscription. In particular +the readiness future is a single slot in two places at once — on the subscription, where `close()` +finds it, and in the runtime's pending map keyed by wait id, where the readiness activation finds it. + +A second coroutine blocking on the same subscription therefore did not queue behind the first, it +replaced it in both places. Readiness resolved only the newer future; its `finally` removed the map +entry and cleared the field; and the older future became unreachable by the readiness activation and +by `close()` alike. That coroutine is stuck for the life of the Run — and because the blocked flag is +shared too, the runtime can report the wait as no longer blocked while a coroutine is still sitting on +it, so Core may not even be retaining a Workflow Task on its behalf. + +Two consumers of the same *stream* is a supported shape and already has an answer: subscribe twice. +Delivery is a broadcast (ADR-021), so each wait gets every record and keeps its own cursor. + +## Options + +**A. Support several waiters per subscription**, with a list of futures resolved together. + +**B. Refuse a second *iterator*** — claim the subscription when `_iterate` starts, release it when the +generator finishes. + +**C. Refuse a second *waiter*** — refuse at the point a coroutine tries to block on a wait that +already has one blocked on it. + +**D. Leave it.** Document single-consumer and rely on callers. + +## Decision + +**C.** `ConcurrentStreamConsumerError` — non-retryable, for the reason +`ExternalStreamCapacityError` is — is raised when a coroutine tries to block on a subscription whose +readiness future is already held, on both waiting paths: the single-subscription one and `merge()`. +`merge()` checks every member before registering any, so a refused merge leaves no wait registered and +blocked with no coroutine behind it. + +A makes the *stranding* go away and leaves the rest. Two coroutines would still share one cursor and +one blocked flag, so records would interleave between them in an order nothing records, and the +`wait_id`-ordered interleaving that makes `merge()` replay would have no equivalent here. It is a +feature nobody asked for, standing in for one that already exists. + +B is the tempting shape and refuses correct code. Breaking out of an `async for` leaves the generator +*suspended at its yield*, not closed — CPython finalizes it later, at collection — so an +iterator-level claim cannot tell "took a few records and came back" from "two live consumers", and the +first is ordinary Workflow code that works today. + +C catches every case B would, because two coroutines can only be inside `_iterate` at once while one +of them is suspended, and the only suspension point in there is the wait. A generator suspended at its +`yield` is not inside `_iterate` at all — it is between `__anext__` calls, with the caller's own code +running. Where records are already buffered, two `__anext__` calls run to completion one after the +other with no interleaving and no shared future, which is harmless and is left alone. + +D was the state the review found, and the failure it produces is the worst shape available: a Workflow +permanently stuck on a future nothing can resolve, with no error, no metric, and a blocked-set snapshot +that says the wait is not blocked. + +## Consequences + +- **Iterating the same subscription again, after the previous consumer has stopped, is allowed** and + resumes where it left off. That is the whole reason the refusal is not on the iterator. +- The error is deterministic and raised on the Workflow thread, so it reproduces under replay and + fails the Workflow rather than looping a Workflow Task retry that meets the same code every time. +- The refusal is *not* a general concurrency check: two `__anext__` calls that both find records ready + are served. What is refused is a second *blocked* waiter, which is the only state that loses one. +- A test must block one consumer, start a second, and assert the second is refused while the first + still receives the next record and is resumed by `close()`. It must also assert that + break-then-iterate-again on one subscription still works, or the guard passes by refusing everything. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-038-an-append-with-no-answer-is-an-unknown-outcome.md b/arch_docs/streaming-poc-docs/decisions/ADR-038-an-append-with-no-answer-is-an-unknown-outcome.md new file mode 100644 index 000000000..fa7f2cce7 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-038-an-append-with-no-answer-is-an-unknown-outcome.md @@ -0,0 +1,146 @@ +# ADR-038 — An append that reports no outcome is unknown, not failed + +**Status:** Accepted · **Affects:** P6, P6b · **Spec:** `spec/wake-signal.md` + +## Context + +ADR-036 gave the post-append window an outcome that names the offset and one recovery, cancellation +included. It left one assumption in place: that cancellation or a failure *before* `append()` returns +means nothing landed. For a local backend that is true. For a remote one it is not. + +A backend commits on its own side and only then answers. The Redis provider runs an atomic append +script server-side and receives its result in a separate client-side step, so a cancellation or a +dropped connection between those two steps leaves a durable record whose offset nobody holds. What +reached the caller there was a bare `CancelledError` or a bare `ConnectionError`: no offset, no +record, nothing that identifies the write whose fate is in question. + +That is unrecoverable in both directions, and each direction breaks a different half of the producer's +contract: + +- **Retrying `publish()`** draws a new sequence number, and therefore a new idempotency key, so a + record that did land is appended a second time under a key the backend has no way to relate to the + first. For `finish_writing()` the duplicate is a second write fence, which reads back as a producer + session that ended twice. +- **Not retrying** can leave a durable record that no wake was ever sent for, stranding a parked + Workflow for its whole idle timeout — the exact state the acknowledged-wake contract exists to + refuse. + +The caller cannot choose between them, because nothing it was handed says which history happened. + +## Options + +**A. Keep reading "did not return" as "did not happen".** The status quo after ADR-036. + +**B. Probe on the interrupted path** — read the stream back, shielded, and decide whether the record +is there before reporting anything. + +**C. Auto-resolve** — re-append the exact record from inside the interrupted call, so the caller only +ever sees a settled outcome. + +**D. Fold it into `WakeNotAcknowledgedError`**, the way ADR-036 folded cancellation, with the offset +left unset. + +**E. Give the window its own outcome**, carrying the exact record, with a recovery operation that +re-appends *that* record — and refuse further appends on the stream until it is settled. + +## Decision + +**E.** Everything out of `backend.append()` that is neither a return nor a contractual refusal raises +`AppendNotAcknowledgedError`, carrying `.record` — byte-identical, still holding its +`(session_id, sequence)`, offset unset — plus `.wake`, `.lease` and `cancelled=True` when cancellation +ended the attempt. `ExternalStreamProducerTopic.resolve_append(record)` re-appends that record and +then runs the wake, so a failure in the wake stage raises `WakeNotAcknowledgedError` with the offset +exactly as `publish()` does. + +**The recovery is correct for both histories, which is what makes an unknown outcome actionable +rather than merely honest.** ADR-020 already requires that a repeat append of byte-identical content +under a used key write nothing and return the original offset, and a key the backend never saw is +simply appended now. One call, one record, one offset, either way. + +**Until it is settled, the stream refuses further appends from that producer with the same error.** +This is the substantive half. Reporting the state and leaving the caller free to `publish()` again +still permits the duplicate — the failure is not that the caller lacked information, it is that the +obvious next call is the wrong one. The refusal is checked at entry, so a concurrent publish already +past that point completes; concurrent publishes have no defined order between them to preserve. + +**The recovery is bound to the operation, the stream, and the producer instance**, and each binding +answers a distinct way the record could be duplicated or misrouted: + +- **The operation, not just the record.** What the interrupted call owed includes whether a wake was + due, under which lease, and whether cancellation is still to be honoured. The producer holds all of + it, and every raise about that append reports the producer's current canonical state for the + operation. A refusal that described the *refused* call instead would make its own instructions + wrong: a caller following them settles with `wake=False` a record that owed a Signal, and the parked + Workflow stays parked on a durable record nobody announced. If `resolve_append()` itself is + interrupted, its effective wake and lease replace the earlier attempt's: it is now the call whose + outcome is unknown. Cancellation is cumulative rather than replaceable, because once delivered it + remains for the caller to honour after settlement. The retained state and the newly raised error + are therefore produced from that same updated operation. +- **The stream.** A `StreamRecord` names its producer session and sequence but not its stream, and + idempotency is scoped per stream. On any other topic that key has never been used, so settling there + appends a second copy of the value onto a stream no consumer of it is watching, and leaves the real + stream blocked. `resolve_append` refuses a record that is not *this* topic's outstanding append, and + refuses one whose bytes differ from it, before touching the backend. +- **The producer instance.** A replacement producer built with the same session id has its sequence + and wake counters back at zero, so settling there leaves both invalid: its next publish reuses a + sequence number the recovered record already holds, and its next unparked wake re-derives a request + ID an earlier and *different* wake already used — which the server deduplicates away, leaving a + durable record unannounced. The recovery for a producer that is gone is the one the Activity retry + already performs: rebuild with the same session id and re-run the same calls in the same order. That + re-derives the same sequences, so the appends deduplicate, and the counters advance correctly + because the calls actually ran. `resolve_append` is the in-process recovery for the instance that + still owes the append, and says so when refusing. + +**`AppendConflictError` is exempt**, and is the only exemption the contract can support: it says the +key was used with *different* bytes, so this record did not land and re-appending it would raise the +identical error. Treating it as unknown would send the caller to settle an append that has no +settlement and would wedge the stream behind it. `KeyboardInterrupt` and `SystemExit` are not +converted either, for ADR-036's reason — the interpreter is going away and there is no caller left to +recover. + +A is what the review found, and its cost is a silently duplicated record. + +B cannot be relied on, for the reason ADR-036 rejects deferring or shielding the wake: the probe runs +on a cancelled task, a second delivery cuts it short, and the ambiguity is back. It is also not free — +without an index from idempotency key to offset, the probe is a scan — and it does not remove the need +for E, because the failure that loses the append's answer is usually the one that loses the probe's +too. Two mechanisms where one is sufficient. + +C hides the choice the caller has to make. Cancellation is refused rather than propagated already +(ADR-036), but auto-appending goes further: it publishes a value on a task that was asked to stop, in +the history where the first attempt never landed. Whether to complete the write or honour the +cancellation belongs to the caller; what the SDK owes is that both remain reachable. + +D repeats ADR-036's argument in a case where it does not hold. Folding cancellation into the wake +error was right because the *state* was identical — durable record, unsent wake, one recovery. This +state is not that one: the offset is unknown, the recovery is an append rather than a Signal, and +`retry_wake` on it would announce a record that may not exist. A caller's existing +`WakeNotAcknowledgedError` handler is wrong here rather than merely incomplete, which is precisely the +case a second type is for. + +## Consequences + +- **`publish()` has three outcomes, not two**: appended and announced, appended but unannounced, and + outcome unknown. The third is the only one whose recovery touches the backend's append path. +- **A cancellation is "before the append" only if the backend was never called.** The producer draws + its sequence and encodes the payload first, and an interruption there still raises `CancelledError` + with nothing appended and nothing owed. Once the call is in flight, the window is ambiguous by + construction and is reported as such. +- **An unsettled append blocks its stream.** That is deliberate: the alternative is a duplicate. A + caller that cannot settle it — a dying Activity — drops the producer, and the retried attempt reuses + the session id and re-derives the same key, so its own re-append is the same no-op. +- **Recovery is in-process only.** `resolve_append` names the producer, the topic and the exact + record, and refuses anything else with a `ValueError` that says which of the three did not match. + Cross-process and cross-attempt recovery is the Activity retry, which needs no new mechanism. +- **Repeated recovery updates one operation rather than creating a parallel version of it.** A + re-interrupted `resolve_append()` makes its effective wake and lease the defaults for the next + attempt, and cancellation stays set if any attempt received it. Later refusals and recovery errors + report exactly that retained state. +- The `.wake` and `.lease` of a recovery default to what the interrupted call was doing, rather than + to the method's own defaults. A recovery that chose its own policy would give a `wake=False` fence a + Signal and take one away from a `wake=True` publish. +- Nothing changes for a backend that never loses an answer; the new outcome is unreachable when + `append()` always returns or raises `AppendConflictError`. +- A test must commit inside `append()` and *then* lose the answer, settle it, and find exactly one + record — one fence, for `finish_writing()`. A test that blocks before the commit exercises the same + code path and proves nothing about the duplicate, because the producer cannot tell the two apart. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-039-the-continuation-carries-the-binding-and-it-is-must-understand.md b/arch_docs/streaming-poc-docs/decisions/ADR-039-the-continuation-carries-the-binding-and-it-is-must-understand.md new file mode 100644 index 000000000..6f756a630 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-039-the-continuation-carries-the-binding-and-it-is-must-understand.md @@ -0,0 +1,29 @@ +# ADR-039 — The continuation carries the provider binding, and it is must-understand + +**Status:** Accepted · **Affects:** P15 · **Spec:** `spec/annotation-format.md` + +## Context + +A Continue-As-New cursor is meaningful only in the store and provider format that produced it. The +successor Run can perform its first live read before it has written a marker, so marker replay cannot +protect that first restoration. + +The feature is private and unreleased. There is no deployed continuation format to preserve. + +## Decision + +Each continuation entry carries its `wait_id`, cursor, stream name, `provider_id`, and +`provider_format_version`. The successor checks the stream name for Workflow nondeterminism and the +provider identity/version for unsafe Worker reconfiguration before handing the cursor to the +backend. + +The binding is part of the only current schema. The encoder always writes that schema and the +decoder accepts exactly that schema. There is no legacy decoder, configurable writer version, +reader stage, writer stage, or downgrade path. + +## Consequences + +- A Worker configured with a different provider or provider format fails before backend I/O. +- An incompatible schema fails while the successor runtime is built, before Workflow code runs. +- Format changes before release replace the current schema and its fixtures directly; they do not + add compatibility code for prototype formats. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-040-a-fence-is-ordered-behind-data-writes-only.md b/arch_docs/streaming-poc-docs/decisions/ADR-040-a-fence-is-ordered-behind-data-writes-only.md new file mode 100644 index 000000000..b94d263f0 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-040-a-fence-is-ordered-behind-data-writes-only.md @@ -0,0 +1,77 @@ +# ADR-040 — A fence is ordered behind data writes, and behind nothing else + +**Status:** Accepted · **Affects:** P6 · **Spec:** `spec/backend-contract.md` + +## Context + +`finish_writing()` claims that every write in this producer session preceding the fence has been +appended, and a consumer that drains through a fence may park on it. The claim is about *invocation* +order, because that is the order a caller wrote its calls in — and a `publish()` draws its sequence +number before awaiting the payload codec, deliberately, so that an idempotency key belongs to the +call rather than to whatever order an external payload store answers in (ADR-020, and "Producer +binding" in the spec). + +So a publish invoked first can still be encoding when the fence is called, invisible to both the +backend (it has appended nothing) and the unresolved-append check (it has no unresolved append). A +fence appended there parks a consumer in front of data that was never written, and in a `wake=False` +batch it also spends the batch's only wake before the records exist. + +Enforcing the claim needs a per-stream record of which calls are in flight, shared by every handle +`topic()` returned for the name, since the stream is one thing and the handle is not. The question +this record answers is *what belongs in that order*. + +## Options + +**A. Nothing.** Document that the fence is ordered by the sequence number it drew and leave the +append unordered. + +**B. One order holding publishes and fences alike**, each call waiting on everything unsettled when +it began. + +**C. An order of publishes only.** A fence reads it and waits behind it; a fence does not join it. + +**D. One order, with a kind on each entry**, and a fence propagating failures only from the entries +marked as data. + +## Decision + +**C.** + +A was the state a review found. The sequence number fixes the record's *place* in the stream and says +nothing about when it is sent, so the fence's own claim was the one thing the fence did not check. + +B is the obvious enforcement and puts a second defect where the first was. A fence is not a data +write, and it has no record whose absence a later fence needs to know about — so an earlier fence +that never reached the backend, cancelled while it waited or refused, presents to a later fence +exactly as a publish whose record went missing. The later fence is then refused with +`PrecedingWriteFailedError`, an error documented as reporting a failed `publish()`, at a moment when +every write it claims is durable. Two fences also have no ordering to preserve between them: each +asserts something about the publishes it was invoked after, and neither assertion is inside the +other. + +D is B with the symptom patched. It keeps fences in a structure whose only purpose is to be waited +on by fences, then adds a field so that they are skipped when they are. C deletes the entry instead +of learning to ignore it. + +## Consequences + +- Concurrent fences are unordered with respect to each other, and both claims are true whichever + order the backend puts them in. +- A fence has no settled outcome for anything to read, so it neither registers nor settles. What + remains of the fence path is: read the order, wait, append. +- **The fence reads every earlier outcome after its last wait, not as each one settles.** An + append whose answer was lost is *unknown* rather than failed, and `resolve_append()` can turn that + into a durable record or an `AppendConflictError` while the fence is still waiting on a later + publish. An outcome read in the loop is a stale one, and the stale value it reads is the permissive + one. +- For the same reason, **a resolution is reported to the operation a fence captured**, rather than + left for the fence to infer from the record no longer being outstanding: resolving it durably and + refusing it with a conflict both stop it being outstanding, so that inference reads a proven-absent + write as a durable one and releases a fence over the hole. +- Where a fence has both an unresolved and a definitely-failed write ahead of it, the unresolved one + is reported, because it blocks the recovery the other error prescribes — republishing is itself + refused while an append on that stream is unsettled (ADR-038). +- A test for this needs a publish held inside its codec and *two* fences, with the first ending + without appending: a single-fence test passes under B. A test for the resolution rule needs a + backend that loses one answer and then refuses the recovery, which is contract behaviour on both + halves rather than a broken store. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-041-a-reported-task-keeps-the-readiness-promise.md b/arch_docs/streaming-poc-docs/decisions/ADR-041-a-reported-task-keeps-the-readiness-promise.md new file mode 100644 index 000000000..0dc91e82c --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-041-a-reported-task-keeps-the-readiness-promise.md @@ -0,0 +1,83 @@ +# ADR-041 — A task on its way to the server keeps the readiness promise it made + +**Status:** Accepted · **Affects:** C7, C15b · **Spec:** `spec/core-lang-protocol.md` + +## Context + +`notify_external_stream_ready` answers one of five results, and `Accepted` is the only one that +tells the watcher to do **nothing**: Core has taken the readiness and will produce the activation +that delivers it. Every other result puts the obligation back on the watcher, which sends the wake +Signal itself. + +`Accepted` is returned whenever a Workflow Task is open, and a Workflow Task is open from the moment +it is applied until `mark_wft_complete` runs — which is *after* the server has answered the +completion. Between those two points sits every completion that reports rather than retains: a +replaying one, one carrying a server-bound command, one answering a query. Each registers the wait +set lang described and then reports the task, and neither the registration path nor the report path +turns pending readiness into a `ResolveExternalStreamWaits` job — `_check_more_activations`, which +is where readiness accumulated during an outstanding activation is normally picked up, is not on +either of them. + +So a readiness accepted anywhere in that window is stranded. Core holds it in `ready`, no activation +is ever issued for it, the task closes, and the watcher — told to do nothing — owes no Signal. The +record stays buffered behind a Run that Core believes it has already told, and nothing in the system +is waiting for anything: the Workflow is blocked, the Worker is idle, the server has no task to +schedule. + +A Run resumed by a wake Signal walks straight into this. Its replayed `subscribe()` starts a watcher +that finds records already in the stream, so readiness arrives during the very activation whose +completion registers the wait set — and that completion refuses retention because the Run was still +replaying when it began. + +## Options + +**A. Retain the task instead.** Decide retention from the machines' state *after* the completion is +applied, so a completion that catches the Run up holds the task open the way a live one does. + +**B. Revoke the accepted readiness** when the task is reported: clear it, put the waits back to +blocked, and let the next report of the same record cover it. + +**C. Let the wake Signal cover it** — treat the window as the watcher's problem. + +**D. Keep the promise on the next task.** End local delivery when Core commits to reporting, so +readiness arriving *after* that point is answered `NoOpenWorkflowTask` and the watcher signals for +itself; and for readiness already accepted, ask the server for the replacement task the resolve job +will be issued on. + +## Decision + +**D.** + +A is the better shape and is not this change. `replaying` is sampled before lang's commands reach +the machines, so it describes the activation rather than its outcome, and a completion that catches +a Run up is a live completion by the time retention is decided — deciding it there would also make +this window narrower. But retention changes what is written to History: the marker for a boundary +that no longer closes is not emitted, so a Run of this shape records different history than it does +today, and existing histories would replay against a Worker that writes a different command +sequence. That is a versioned change, and it does not remove the need for D — a completion carrying +a server-bound command must be reported however retention is decided. + +B is silent loss dressed as bookkeeping. The watcher was told `Accepted`; it has already advanced +past the buffered record and will not report it again. Revoking leaves nobody holding the +obligation, which is the defect, not the fix. + +C is what the code did. The watcher owes a Signal only for the results that say so, and `Accepted` +is the one that says the opposite — so nothing sends one. + +## Consequences + +- Local delivery ends when the completion is prepared for the server, not when the server answers. + The window in which `Accepted` can be returned is exactly the window in which Core can still act + on it. +- A completion reported with readiness pending sets `force_create_new_workflow_task`. The + replacement task re-opens the wait set in `apply_new_wft`, and `_check_more_activations` issues + the resolve job the pending readiness was promised — no Signal round trip, and no dependence on + the watcher having kept anything. +- Readiness landing after that point takes the `NoOpenWorkflowTask` path, which is the documented + healthy state between Workflow Tasks and already sends the wake Signal. +- The extra task is bounded: it is requested only when readiness is actually pending, and the + activation it carries drains that readiness, so the next completion has nothing to force. +- A test for this cannot be written from a quiescent command alone, because the wait set does not + exist until the completion that registers it. It needs the set seeded with a task open, readiness + accepted against it, and then a completion carrying a server-bound command — which is what + `readiness_accepted_against_a_reported_task_is_not_stranded` does. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-042-one-run-has-one-outstanding-wake-cycle.md b/arch_docs/streaming-poc-docs/decisions/ADR-042-one-run-has-one-outstanding-wake-cycle.md new file mode 100644 index 000000000..5d2da2ca0 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-042-one-run-has-one-outstanding-wake-cycle.md @@ -0,0 +1,57 @@ +# ADR-042 — One Run has one outstanding wake cycle + +**Status:** Accepted · **Spec:** `spec/wake-signal.md` + +## Context + +When Core cannot accept buffered readiness locally, the Worker sends a Signal that asks the server +for a Workflow Task. Signal acknowledgement proves that request reached History; it does not prove +that the task it created completed successfully. The task may fail, its report may be rejected, or +the Run may be evicted and replayed before the same buffered range is reconstructed. + +If every reconstructed readiness report sends a new wake, each failed task can recursively create +another task. Concurrent subscriptions make the same mistake independently even though Workflow +Tasks are scoped to the Run, not to a subscription. Distinct wakes entering the execution while a +terminal command is closing it can sustain a `BUSY_WORKFLOW` / `UnhandledCommand` report-and-replay +cycle. + +## Options + +**A. Signal every readiness report.** Treat replay and every wait independently. + +**B. Deduplicate by buffered range or wait generation.** Reuse durable-looking readiness facts as +the wake identity. + +**C. Keep one Run-wide wake cycle until the task it caused completes.** Correlate that completion to +the particular successful Signal attempt, coalescing replay and concurrent waits behind it. + +**D. Cap retries or treat `BUSY_WORKFLOW` as terminal.** Bound excess work by abandoning the wake. + +## Decision + +**C.** A Workflow Task services the Run's complete wait set, so the outstanding request is Run-wide. +The first readiness result that cannot be delivered locally opens a cycle and draws one wake counter. +All retries keep that counter and therefore the same request ID. Replay, eviction, and other waits +may reconstruct buffered readiness, but they do not open another cycle. + +Each Signal attempt records the activation sequence at which it began. Only a successful completion +of a later activation can close that attempt's cycle, and it counts only after the Signal attempt is +acknowledged. A completion racing an in-flight attempt is retained until its result is known; a +failed attempt discards that correlation before retrying. This prevents a task that was already +running when the send began from being mistaken for the task the Signal caused. + +D is not available. `BUSY_WORKFLOW` is transient, and abandoning the wake can strand a durable +record whose watcher has already advanced past it. A retry cap converts excess work into silent data +loss rather than restoring a bound. + +## Consequences + +- A nonterminal completion closes the cycle and re-reports any buffered readiness, which may open a + genuinely later cycle with a new counter. +- A terminal completion closes the cycle without rearming; there is no future activation that could + consume the buffer. +- Retiring a stale park intent invalidates the cycle that intent silenced before issuing the required + unparked reannouncement. +- Failed Signal attempts remain retries of one wake, not new asks, and keep one request ID. +- Tests must cover replay reconstruction, concurrent waits, a completion racing a failed attempt, + a genuinely later range, terminal completion, and stale-intent cleanup. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-043-a-second-workflow-task-is-buffered.md b/arch_docs/streaming-poc-docs/decisions/ADR-043-a-second-workflow-task-is-buffered.md new file mode 100644 index 000000000..0f45c3d4a --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-043-a-second-workflow-task-is-buffered.md @@ -0,0 +1,44 @@ +# ADR-043 — A second Workflow Task is buffered, never substituted + +**Status:** Accepted · **Spec:** `spec/wft-lifecycle.md` + +## Context + +`ManagedRun` owns the task token for one outstanding Workflow Task until that task is reported. A +new polled task can arrive on a different input lane before the local completion message clears the +old one. The Run may have no pending activation jobs at that instant, but it still owns the original +task and its token. + +Overwriting the slot loses the original token. A debug-only assertion detects the invariant +violation during development, but in a release build the assertion logs and continues; continuing +with substitution would silently corrupt task ownership. + +## Options + +**A. Substitute the newer task.** Let the latest poll result win. + +**B. Make the invariant an unconditional process panic.** Stop the Worker rather than continue. + +**C. Guard admission and defensively buffer.** Normal admission buffers whenever a task is already +owned; the callee retains its debug panic but release builds preserve the old task and queue the new +one. + +## Decision + +**C.** `must_buffer_wft` includes ownership of an outstanding task even when no activation job is +pending. `_incoming_wft` keeps the debug panic so the impossible call path remains loud in debug +builds. After that diagnostic, the release path buffers the replacement and returns without changing +the original slot. + +The original task is reported with its original token. Only then may the buffered replacement drain +into the Run. + +## Consequences + +- No scheduling race can transfer or discard ownership of a task token. +- Release recovery is local to this admission invariant; no global `dbg_panic!` policy changes. +- The regression must use a real `ManagedRun` and two permitted tasks, clear the first activation + while leaving its task outstanding, prove the second buffers, report the first, and prove the + second drains. +- Fault injection covers both profiles: debug panics at the invariant, while release preserves the + original token and later admits the replacement. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-044-output-stage-tokens-and-exact-history-floors-prove-commit.md b/arch_docs/streaming-poc-docs/decisions/ADR-044-output-stage-tokens-and-exact-history-floors-prove-commit.md new file mode 100644 index 000000000..4c9c29ec2 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-044-output-stage-tokens-and-exact-history-floors-prove-commit.md @@ -0,0 +1,43 @@ +# ADR-044 — Output stage tokens and exact History floors prove commit + +**Status:** Accepted · **Affects:** output staging, reconciliation · **Spec:** +`spec/backend-contract.md` + +## Context + +Workflow output must be staged before its Workflow Task is reported, but it may become visible only +after that task commits. A Worker can die on either side of the server acknowledgement. Speculative +Workflow Tasks make Scheduled and Started event IDs unsuitable identities: discarded attempts may +leave neither event durable, and later attempts may reuse the IDs. + +## Options + +**A. Use Workflow Task event IDs as the stage identity.** Reuse collides across speculative +attempts. + +**B. Resolve an old pending stage after a timeout.** Absence of a marker is not proof while the +task outcome is still undecidable. + +**C. Mint a unique token per attempt and reconcile from the exact durable History predecessor.** + +## Decision + +**C.** The Worker mints one opaque `stage_token` per Workflow Task staging attempt and reuses it only +for retries of that exact stage. Core supplies `history_floor_event_id`, the event immediately before +the producing task's Scheduled event in the ordered History view; staging is refused if that exact +floor is unavailable. + +A resolver scans strictly above the floor. The exact token in an external-stream marker commits the +stage. Otherwise the first durable Workflow Task completed, failed, or timed-out boundary, or +Workflow closure, aborts it. With neither fact the stage remains pending. History loss is an +integrity failure, never an inferred abort. + +## Consequences + +- Byte-identical retries and reused speculative event IDs cannot alias another attempt. +- Workers reconcile opportunistically; `ExternalOutputStreamClient` performs the same History + predicate lazily at a pending head. +- Repeated stage, commit, abort, and reconciliation operations are idempotent. Reusing a token with + another immutable manifest is a conflict. +- There is no timeout-based escape from an undecidable stage. Operators restore the needed History + or provider metadata, or terminate the affected execution. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-045-output-latency-and-input-parking-share-one-terminal-race.md b/arch_docs/streaming-poc-docs/decisions/ADR-045-output-latency-and-input-parking-share-one-terminal-race.md new file mode 100644 index 000000000..c5dad7514 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-045-output-latency-and-input-parking-share-one-terminal-race.md @@ -0,0 +1,43 @@ +# ADR-045 — Output latency and input parking share one terminal race + +**Status:** Accepted · **Affects:** retained Workflow Tasks, output visibility · **Spec:** +`spec/wft-lifecycle.md` + +## Context + +A Workflow Task retained on external input can also hold Workflow-originated output. Letting input +parking and output visibility use independent timers and handshakes can write two terminals, confirm +a park generation after output already forced rollover, or leave installed intents for a park Core +discarded. + +## Decision + +Core serializes readiness, idle/all-fenced parking, the output deadline, ordinary rollover, and +server-bound completion as one terminal race. The first accepted output in a batch arms a distinct +deadline; the default and first-release quantum is 100 milliseconds, reduced by `min` as shorter +topic policies join the batch and clamped below ordinary rollover. + +- A confirmed park stages the output and reports `WorkflowOutputStreamCommit` before + `ExternalStreamParkResult(confirmed)`. One shared marker completes the task without forcing a + replacement. +- A recheck that became ready reports `WorkflowOutputStreamBuffered` before + `ExternalStreamParkResult(became_ready)`. The task remains open and the earliest output deadline + remains armed. +- If readiness wins after Python already staged for a confirmation, Core discards the stale park + terminal, resolves the abandoned park, then finalizes with `TASK_COMPLETED`, records the carried + output commit once, and forces a replacement. +- If the output deadline wins during park preparation, Core invalidates that quiescence generation, + resolves the abandoned park so Python removes or records every owed intent removal, then issues + `FinalizeExternalStreams(OUTPUT_LATENCY)`. A commit already returned by the losing prepare is + carried forward; otherwise Python stages during finalization. The output marker completes with + `force_new_wft = true` so input consumption continues on a replacement task. + +Late results are stale and cannot produce another terminal or marker. + +## Consequences + +- Healthy visibility is bounded by the configured quantum plus staging and Workflow Task completion + latency; it is not an availability SLA during backend or server outage. +- Parking may flush earlier than the output deadline and avoids an extra Workflow Task lifecycle. +- Replay follows the recorded terminal and never arms the wall-clock output timer. +- The output winner never confirms the losing park generation. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-046-output-capacity-is-logical-and-segmentation-is-shared.md b/arch_docs/streaming-poc-docs/decisions/ADR-046-output-capacity-is-logical-and-segmentation-is-shared.md new file mode 100644 index 000000000..1a6a3dbdc --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-046-output-capacity-is-logical-and-segmentation-is-shared.md @@ -0,0 +1,35 @@ +# ADR-046 — Output capacity is logical and segmentation is shared + +**Status:** Accepted · **Affects:** output batching, replay · **Spec:** +`spec/annotation-format.md` + +## Context + +Encoded bytes can change when a payload codec is randomized, upgraded, or uses external storage. +A retained Workflow Task can span several activations, and one marker may contain both consumed +input and produced output. Letting each direction reproduce its own activation drains would double +the live schedule. + +## Decision + +Workflow output capacity is per Workflow Task and is computed from deterministic, pre-codec logical +frames. Fingerprint version 1 hashes ordered, length-prefixed frames whose payload metadata keys are +sorted by UTF-8 bytes. The first successfully staged encoded bytes remain authoritative; logical +counts and SHA-256 fingerprints define retry and replay identity. + +The marker records one activation-segment schedule shared by input and output. Output contributes a +per-topic record-count vector to each segment, including empty segments. Replay disables live +capacity and latency policy, validates the recorded logical manifests, and performs the live number +of event-loop drains once across both directions. + +When another publish would exceed the record or logical-byte limit, `publish()` waits, the current +batch is staged with an output-capacity terminal, and Core forces a replacement Workflow Task. A +single oversized record or manifest is rejected before unsafe external I/O or marker growth. + +## Consequences + +- Codec size and randomness cannot move a deterministic batch boundary. +- Capacity is not reset by each activation of a retained task. +- Input and output replay drivers attach to one schedule; they never sum their drain counts. +- Marker bytes scale with topic and segment metadata, not payload size, and hard marker bounds still + apply. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-047-pending-output-is-a-topic-ordering-barrier.md b/arch_docs/streaming-poc-docs/decisions/ADR-047-pending-output-is-a-topic-ordering-barrier.md new file mode 100644 index 000000000..f28d9e8c4 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-047-pending-output-is-a-topic-ordering-barrier.md @@ -0,0 +1,30 @@ +# ADR-047 — Pending output is a topic ordering barrier + +**Status:** Accepted · **Affects:** output providers and clients · **Spec:** +`spec/backend-contract.md` + +## Context + +A Workflow batch is durable in the provider before Temporal decides whether its producing task +commits. Direct Activity or external-process output can append while that decision is pending. +Skipping unresolved positions would expose later data ahead of output that may still commit. + +## Decision + +A pending Workflow stage occupies provider order but is not readable. `read_output_after()` returns +only the committed prefix before the first unresolved stage and identifies that pending barrier. +Readers never pass it. `ExternalOutputStreamClient` resolves the barrier with ADR-044's built-in +History predicate, then retries the provider read. + +Direct `ExternalOutputStreamProducer` records are immediately committed singleton appends and may +share a topic with Workflow batches. Their retry identity remains `(session_id, sequence)`. + +## Consequences + +- If the pending stage commits, its records precede direct records already placed behind it. If it + aborts, it yields nothing and a retried Workflow Task may append after those direct records. +- The guarantee is per-topic provider order, not a deterministic total order between independently + running Workflow and Activity producers. +- There is no atomic snapshot or order across topics. Applications needing one order publish + envelopes to one topic. +- `tail()` returns the readable committed boundary and cannot cross a pending head. diff --git a/arch_docs/streaming-poc-docs/decisions/ADR-048-finished-output-topics-use-a-must-understand-header.md b/arch_docs/streaming-poc-docs/decisions/ADR-048-finished-output-topics-use-a-must-understand-header.md new file mode 100644 index 000000000..535f944e8 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/ADR-048-finished-output-topics-use-a-must-understand-header.md @@ -0,0 +1,35 @@ +# ADR-048 — Finished output topics use a must-understand header + +**Status:** Accepted · **Affects:** Continue-As-New, topic termination · **Spec:** +`spec/annotation-format.md` + +## Context + +Workflow output topics span a Continue-As-New chain. A successor must reject a publish after the +Workflow explicitly finished that topic without consulting mutable provider state, and a provider +offset or marker from another implementation cannot safely answer that question. + +## Decision + +`finish()` in Workflow code and `finish_writing()` in a direct producer are the only operations that +append a `FINISH` record. Workflow completion, failure, cancellation, and termination synthesize no +topic terminal. + +Before Continue-As-New, the Workflow runtime writes the topics it finished plus provider identity and +format version into a reserved output continuation header. The format is must-understand: the +successor decodes and validates it before Workflow code or backend I/O, restores the finished set, +and deterministically rejects another Workflow publish to a finished topic. + +A direct producer's `finish_writing()` closes that producer's topic handle; it cannot mutate a +Workflow's continuation header. Applications mixing direct and Workflow publishers on one topic +must therefore choose one terminal owner. A client stops at the first `FINISH` it reads. + +## Consequences + +- Finished state survives across Runs without using mutable backend state as a replay fact. +- Direct producers do not silently impose Workflow state through the backend. +- An unknown schema or mismatched provider binding fails before user code runs. +- Clients that want to stop on Workflow closure also observe Temporal execution status; backend + idleness is never an implicit finish. +- The input cursor continuation and output finished-state continuation remain separate reserved + headers because they carry different must-understand schemas. diff --git a/arch_docs/streaming-poc-docs/decisions/README.md b/arch_docs/streaming-poc-docs/decisions/README.md new file mode 100644 index 000000000..7575e1546 --- /dev/null +++ b/arch_docs/streaming-poc-docs/decisions/README.md @@ -0,0 +1,75 @@ +# Decision records + +One file per decision. Each states the context, the options that were on the table, the choice, and +what follows from it. **The specs state current truth only; the reasoning for why an alternative was +not taken lives here.** + +Read one of these when you are about to change a spec and want to know whether the current shape is +load-bearing. Adding a decision means adding a file and a row below. + +| ADR | Decision | Primary spec | +|---|---|---| +| [001](ADR-001-coexist-with-contrib-workflow-streams.md) | Coexist with `contrib.workflow_streams` rather than replacing it | `spec/public-surface.md` | +| [002](ADR-002-cursor-is-a-position-boundary.md) | A cursor is a position boundary, not a record identity | `spec/backend-contract.md` | +| [003](ADR-003-structural-immutability-is-required.md) | Structural immutability is required of every provider | `spec/backend-contract.md` | +| [004](ADR-004-progress-and-quiescence-are-separate-commands.md) | Progress and quiescence are separate commands | `spec/wft-lifecycle.md` | +| [005](ADR-005-progress-is-an-observation-delta.md) | Progress is an observation delta, emitted on every completion path | `spec/annotation-format.md` | +| [006](ADR-006-runs-record-both-endpoints.md) | A run records both endpoints, not a start plus a count | `spec/annotation-format.md` | +| [007](ADR-007-byte-budget-forces-rollover.md) | A hard byte budget forces rollover rather than growing a marker | `spec/annotation-format.md` | +| [008](ADR-008-no-marker-without-a-terminal.md) | Core never writes a marker without a terminal from Python | `spec/core-lang-protocol.md` | +| [009](ADR-009-shutdown-is-two-transitions.md) | Shutdown and eviction are two transitions | `spec/wft-lifecycle.md` | +| [010](ADR-010-finalization-is-manager-state-only.md) | The input terminal in `FinalizeExternalStreams` is manager-state-only | `spec/python-runtime.md` | +| [011](ADR-011-runtime-only-jobs-run-outside-the-workflow-thread.md) | Runtime-only jobs are handled in `_handle_activation`, not `_apply` | `spec/python-runtime.md` | +| [012](ADR-012-park-intents-are-keyed-per-subscription.md) | Park intents are keyed per subscription, not per stream | `spec/backend-contract.md` | +| [013](ADR-013-readiness-result-distinguishes-five-states.md) | The readiness result distinguishes a cached Run from a missing one | `spec/core-lang-protocol.md` | +| [014](ADR-014-integrity-loss-blocks-rather-than-fails.md) | Integrity loss blocks the Workflow; no terminal-failure opt-in | `spec/failure-taxonomy.md` | +| [015](ADR-015-decode-failure-is-not-integrity-failure.md) | A decode failure is a separate class from an integrity failure | `spec/failure-taxonomy.md` | +| [016](ADR-016-idle-timeout-reduces-by-min.md) | The idle timeout is a Workflow-Task policy reduced by `min` | `spec/wft-lifecycle.md` | +| [017](ADR-017-rollover-is-mandatory-and-needs-its-own-timer.md) | Rollover is mandatory and needs a sink-independent timer | `spec/wft-lifecycle.md` | +| [018](ADR-018-replay-segmentation-is-reproduced.md) | Replay reproduces activation segmentation rather than collapsing it | `spec/annotation-format.md` | +| [019](ADR-019-producer-binding-is-fully-explicit.md) | Producer binding is fully explicit, and the stream name appears once | `spec/backend-contract.md` | +| [020](ADR-020-append-idempotency-is-on-identity.md) | Append is idempotent on identity, not on key alone | `spec/backend-contract.md` | +| [021](ADR-021-delivery-is-broadcast.md) | Delivery to multiple subscriptions is broadcast, not work-sharing | `spec/wft-lifecycle.md` | +| [022](ADR-022-continue-as-new-cursor-in-a-reserved-header.md) | The Continue-As-New cursor travels in a reserved internal header | `spec/annotation-format.md` | +| [023](ADR-023-park-generation-zero-is-the-unparked-wake.md) | `park_generation = 0` is the unparked wake | `spec/wake-signal.md` | +| [025](ADR-025-wake-signal-bypasses-the-data-converter.md) | The wake Signal bypasses the user's `DataConverter` | `spec/wake-signal.md` | +| [026](ADR-026-delivery-is-bounded-by-a-record-count.md) | Delivery within one activation is bounded by a record count, not by time | `spec/python-runtime.md` | +| [027](ADR-027-a-late-binding-rides-its-own-frame.md) | A subscription created after the header is bound by its own frame | `spec/annotation-format.md` | +| [028](ADR-028-decoding-is-split-across-the-boundary.md) | Decoding a record is split across the Worker/Workflow boundary | `spec/python-runtime.md` | +| [029](ADR-029-a-closed-subscription-keeps-its-state.md) | A closed subscription keeps its recorded state | `spec/python-runtime.md` | +| [030](ADR-030-cancellation-removes-its-own-park-intent.md) | Cancelling a subscription removes its park intent itself | `spec/wft-lifecycle.md` | +| [031](ADR-031-an-owed-removal-belongs-to-the-run.md) | An owed park-intent removal belongs to the Run, not to the subscription | `spec/wft-lifecycle.md` | +| [032](ADR-032-park-rollback-catches-cancellation-and-is-not-shielded.md) | Park rollback catches cancellation, and is not shielded | `spec/wft-lifecycle.md` | +| [033](ADR-033-replay-checks-bound-waits-one-directionally.md) | Replay checks that every bound wait was recreated, in one direction only | `spec/annotation-format.md` | +| [034](ADR-034-merge-rotates-its-start-position.md) | `merge()` rotates its start position across passes | `spec/python-runtime.md` | +| [035](ADR-035-both-sides-bind-the-workflows-serialization-context.md) | Both sides of a stream bind the Workflow's serialization context | `spec/python-runtime.md` | +| [036](ADR-036-cancellation-after-a-durable-append-is-an-unacknowledged-wake.md) | Cancellation after a durable append is reported as an unacknowledged wake | `spec/wake-signal.md` | +| [037](ADR-037-a-subscription-has-one-consumer.md) | A subscription has one consumer, and the refusal is on the waiter | `spec/python-runtime.md` | +| [038](ADR-038-an-append-with-no-answer-is-an-unknown-outcome.md) | An append that reports no outcome is unknown, not failed | `spec/wake-signal.md` | +| [039](ADR-039-the-continuation-carries-the-binding-and-it-is-must-understand.md) | The continuation carries the provider binding, and it is must-understand | `spec/annotation-format.md` | +| [040](ADR-040-a-fence-is-ordered-behind-data-writes-only.md) | A fence is ordered behind data writes, and behind nothing else | `spec/backend-contract.md` | +| [041](ADR-041-a-reported-task-keeps-the-readiness-promise.md) | A task on its way to the server keeps the readiness promise it made | `spec/core-lang-protocol.md` | +| [042](ADR-042-one-run-has-one-outstanding-wake-cycle.md) | One Run has one outstanding wake cycle | `spec/wake-signal.md` | +| [043](ADR-043-a-second-workflow-task-is-buffered.md) | A second Workflow Task is buffered, never substituted | `spec/wft-lifecycle.md` | +| [044](ADR-044-output-stage-tokens-and-exact-history-floors-prove-commit.md) | Output stage tokens and exact History floors prove commit | `spec/backend-contract.md` | +| [045](ADR-045-output-latency-and-input-parking-share-one-terminal-race.md) | Output latency and input parking share one terminal race | `spec/wft-lifecycle.md` | +| [046](ADR-046-output-capacity-is-logical-and-segmentation-is-shared.md) | Output capacity is logical and replay segmentation is shared with input | `spec/annotation-format.md` | +| [047](ADR-047-pending-output-is-a-topic-ordering-barrier.md) | Pending output is a topic ordering barrier | `spec/backend-contract.md` | +| [048](ADR-048-finished-output-topics-use-a-must-understand-header.md) | Finished output topics use a must-understand continuation header | `spec/annotation-format.md` | + +## The decisions that constrain the most + +If you read only a few, read these — the rest of the design leans on them: + +- **ADR-003** (structural immutability) is what makes every other size and validation claim true. +- **ADR-008** (no marker without a terminal) governs every completion path in Core. +- **ADR-011** (no I/O on the Workflow thread) is a hard property of the Python Worker, not a + preference. +- **ADR-002** (cursor as boundary) is the shape every provider operation is built around. +- **ADR-042** (one outstanding wake cycle per Run) is the liveness and deduplication boundary for + every readiness result that Core cannot accept locally. +- **ADR-043** (buffer a second Workflow Task) is the task-token ownership invariant at Core + admission. +- **ADR-044** (unique stage token plus exact History floor) is the commit-proof invariant for every + Workflow-originated output batch. +- **ADR-045** (one output/park terminal race) prevents a retained task from writing two boundaries. diff --git a/arch_docs/streaming-poc-docs/guide/README.md b/arch_docs/streaming-poc-docs/guide/README.md new file mode 100644 index 000000000..0c41d00fa --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/README.md @@ -0,0 +1,43 @@ +--- +doc_id: EWS-GUIDE-INDEX +status: explanatory-pre-production +audience: [readers, design-reviewers, operators] +normative: false +--- + +# External Workflow Streams — human guide + +This guide explains the feature as a system: what crosses each boundary, why Workflow Tasks are +sometimes retained, how wakeups work, how output becomes visible, and what replay depends on. It is +the place to build a mental model before reading protocol messages or backend operations. + +> **Audience and authority:** this guide is explanatory and intentionally omits edge cases. The +> documents in [`../spec/`](../spec/README.md) are the normative implementation reference. If a +> diagram or summary here appears to disagree with a specification, the specification wins. + +## Reading path + +1. [`architecture.md`](architecture.md) — components, data paths, control paths, and ownership. +2. [`input-lifecycle.md`](input-lifecycle.md) — the Workflow Task and wakeup state machines. +3. [`output-commit.md`](output-commit.md) — staging, commit proof, visibility, and reconciliation. +4. [`replay-and-recovery.md`](replay-and-recovery.md) — markers, exact-range replay, Worker loss, and + Continue-As-New. +5. [`operations.md`](operations.md) — failures, retention, durability boundaries, and validation. + +## The feature in one paragraph + +External Workflow Streams keep high-volume stream payloads in a pluggable external backend rather +than Temporal History. Workflows can consume external input and publish external output while compact +markers preserve deterministic replay. Signals carry wakeups, never records. Core owns Workflow Task +lifecycle and marker coordination; the language runtime owns backend I/O, payload conversion, and +replay validation. No Temporal Server protocol or persistence change is required. + +## The four invariants worth remembering + +- Stream payload bytes never enter Temporal History or a wake Signal. +- Reading or staging a record is not a commit; the corresponding marker is the durable boundary. +- Core never writes a stream marker without a terminal boundary supplied by the language runtime. +- Replay follows recorded ranges and activation segmentation, not current backend timing. + +For exact language, exceptions, and failure consequences, start at the +[`spec/` index](../spec/README.md). diff --git a/arch_docs/streaming-poc-docs/guide/architecture.md b/arch_docs/streaming-poc-docs/guide/architecture.md new file mode 100644 index 000000000..73c96ccc3 --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/architecture.md @@ -0,0 +1,172 @@ +--- +doc_id: EWS-GUIDE-ARCHITECTURE +status: explanatory-pre-production +audience: [readers, design-reviewers] +normative: false +--- + +# High-level architecture + +External Workflow Streams add a data plane beside Temporal and a small coordination plane through +the SDKs. Payloads use the data plane; Temporal History records only enough information to reproduce +the Workflow's observations. + +## Input direction + +An external producer appends payload records directly to the backend. The Python Worker reads and +decodes them, while Core decides when the open Workflow Task can accept readiness and when the task +must be completed. If no open task can accept readiness, a raw Temporal Signal creates server-visible +work; it contains only stream identity and wakeup metadata. + +```mermaid +flowchart LR + Producer[External producer or Activity] + Backend[(External stream backend)] + Runtime[Python Worker runtime] + Workflow[Workflow code] + Core[SDK Core] + Server[(Temporal Server and History)] + + Producer -->|append payload records| Backend + Backend -->|read payload records| Runtime + Runtime -->|decoded values| Workflow + Runtime -->|progress and terminal boundary| Core + Core -->|compact marker and WFT completion| Server + Producer -. wake Signal, no payload .-> Server + Server -. new Workflow Task .-> Core + Core -. resolve activation .-> Runtime +``` + +The normal low-latency path is local: a cached Run with an open Workflow Task receives readiness +directly through Core. The Signal path is the fallback when the task is parked, already completed, or +the Run is no longer cached on that Worker. + +Normative details: [`wft-lifecycle.md`](../spec/wft-lifecycle.md), +[`core-lang-protocol.md`](../spec/core-lang-protocol.md), and +[`wake-signal.md`](../spec/wake-signal.md). + +## Output direction + +Workflow code produces logical output without receiving backend offsets. The Worker stages that +output as unreadable pending data, and Core places a compact commit proof in the Workflow Task's +marker. After the server accepts the task, the stage can become visible. If the Worker disappears in +between, a reader or reconciler reaches the same decision from History. + +```mermaid +flowchart LR + Workflow[Workflow code] + Runtime[Python Worker runtime] + Backend[(External stream backend)] + Core[SDK Core] + Server[(Temporal Server and History)] + Client[External output client] + + Workflow -->|logical records| Runtime + Runtime -->|stage as PENDING| Backend + Runtime -->|stage manifest| Core + Core -->|compact marker and WFT completion| Server + Server -. accepted task proves commit .-> Runtime + Server -. History proof after Worker loss .-> Client + Runtime -->|commit stage| Backend + Client -->|reconcile pending head| Backend + Backend -->|committed records only| Client +``` + +Activities and ordinary external processes can also append already-committed output directly. That +path does not need a Workflow Task commit proof, but it cannot pass an unresolved Workflow stage on +the same topic. + +Normative details: [`backend-contract.md`](../spec/backend-contract.md), +[`annotation-format.md`](../spec/annotation-format.md), and +[`python-runtime.md`](../spec/python-runtime.md). + +## Responsibility boundaries + +| Component | Owns | Deliberately does not own | +|---|---|---| +| Workflow code | Subscriptions, consumption, logical output, deterministic control flow | Backend connections, provider offsets, wake Signals | +| Python runtime | Backend I/O, conversion, buffering, watchers, parking operations, annotation encoding and replay validation | Workflow Task admission and marker persistence | +| SDK Core | Wait generations, readiness serialization, timers, parking arbitration, marker coordination, wake interception | Stream payloads, backend types, codecs, opaque annotation contents | +| Temporal Server | Workflow Task and Signal durability, History containing compact markers | External stream storage or provider coordination | +| Backend provider | Immutable records, ordered offsets, staging, barriers, park intents, retention | Workflow determinism or History interpretation | + +This split is load-bearing. A Python-only implementation would need to place replayable payload +results in History, while teaching Core about provider records would couple every SDK to a storage +implementation. + +## Identity and lifetime + +A physical stream is scoped by namespace, Workflow ID, first execution Run ID, direction, and topic +name. The first execution Run ID keeps the identity stable across Continue-As-New while isolating a +later reuse of the same Workflow ID. Input and output directions are physically separate even when +their user-visible topic names match. + +Input subscriptions have independent cursors and broadcast delivery. Output topics remain open until +an explicit `FINISH`; Workflow completion does not synthesize one. See +[`annotation-format.md`](../spec/annotation-format.md) for the continuation encodings and +[`backend-contract.md`](../spec/backend-contract.md) for physical-key requirements. + +## Public API mental model + +- A Workflow input topic subscribes; an explicitly connected producer publishes and writes a fence. +- A Workflow output topic publishes logical records and explicitly finishes. +- A direct output producer publishes committed records; an external client subscribes from an opaque + resume boundary. +- The Worker owns the configured provider. Workflow code never carries its connection or credentials. + +Consumer and producer handles are intentionally distinct. They represent different identities, +credentials, serialization contexts, and retry obligations rather than one object passed between +processes. + +An input consumer and Workflow output publisher look like ordinary deterministic Workflow APIs: + +```python +tokens = external_stream.with_options( + idle_timeout=timedelta(seconds=1), +).topic("tokens", type=str) + +async for token in tokens.subscribe(): + process(token) + +events = external_output_stream.with_options( + max_publish_latency=timedelta(milliseconds=100), +).topic("events", type=AgentEvent) + +await events.publish(event) +await events.finish() +``` + +External processes bind explicitly to the Workflow chain and provider; external readers resume from +opaque boundaries: + +```python +producer = await ExternalStreamProducer.connect( + backend=input_backend, + workflow=WorkflowChainKey(...), + data_converter=data_converter, + session_id=retry_stable_session, +) +input_topic = producer.topic("tokens", type=str) +await input_topic.publish(token) +await input_topic.finish_writing() + +reader = await ExternalOutputStreamClient.connect( + backend=output_backend, + workflow=WorkflowChainKey(...), + client=temporal_client, +) +async for item in reader.topic("events", type=AgentEvent).subscribe(): + persist(item.data, resume_at=item.offset) +``` + +These examples show roles rather than every required option. The exact producer binding and recovery +contracts are in [`backend-contract.md`](../spec/backend-contract.md). Exact public names, handle +roles, and configuration are in [`public-surface.md`](../spec/public-surface.md). + +## Non-goals + +- Carrying payload records through Signals or Temporal History. +- Changing Temporal Server persistence or service protocols. +- Exactly-once execution of producer or application side effects. +- Global ordering across topics or work-sharing between subscriptions. +- Inferring stream termination from Workflow or producer-process closure. diff --git a/arch_docs/streaming-poc-docs/guide/input-lifecycle.md b/arch_docs/streaming-poc-docs/guide/input-lifecycle.md new file mode 100644 index 000000000..ec9c3ad69 --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/input-lifecycle.md @@ -0,0 +1,106 @@ +--- +doc_id: EWS-GUIDE-INPUT-LIFECYCLE +status: explanatory-pre-production +audience: [readers, design-reviewers] +normative: false +--- + +# Input and Workflow Task lifecycle + +The input design keeps an open Workflow Task while a Workflow is actively consuming or briefly +waiting for records. It completes that task at a recorded boundary when the wait becomes idle, every +producer reaches a fence, a command must reach the server, or the rollover deadline wins. + +## State machine + +```mermaid +stateDiagram-v2 + [*] --> NoOpenWFT + NoOpenWFT --> Active: server starts a Workflow Task + Active --> Active: records are available / resolve activation + Active --> Quiescent: Workflow blocks on complete wait set + Quiescent --> Active: local readiness is Accepted + Quiescent --> PreparingPark: idle deadline or all waits fenced + PreparingPark --> Active: backend recheck became ready + PreparingPark --> Parked: park confirmed / marker written / WFT completed + Active --> NoOpenWFT: command, rollover, or non-retainable boundary + Quiescent --> NoOpenWFT: command, rollover, or shutdown finalization + NoOpenWFT --> Active: unparked wake Signal creates a WFT + Parked --> Active: append plus parked wake Signal creates a WFT +``` + +`Active`, `Quiescent`, and `PreparingPark` all have an open Workflow Task. `Parked` means the backend +park handshake is confirmed and the task has completed. `NoOpenWFT` is also healthy: it commonly +exists after a command-producing completion or rollover while subscriptions remain registered. + +Normative transition rules: [`wft-lifecycle.md`](../spec/wft-lifecycle.md). + +## What happens when Workflow code blocks + +At each activation return, the Python runtime answers three independent questions: + +1. Did replay-visible input state change? If so, it reports an observation delta. +2. Is Workflow code blocked on streams? If so, it reports the complete quiescent wait set. +3. Is there logical Workflow output? If so, it stages it or reports it as temporarily buffered. + +Core may register a wait set without retaining the Workflow Task. This matters when the completion +also contains a timer, Activity, child Workflow, Signal, Query response, or rollover: the command must +reach the server, but later stream readiness still needs a registered wait to resolve. + +## Retention, rollover, and parking + +- One idle timer covers the complete blocked set. Different configured idle timeouts reduce to a + deterministic minimum. +- A separate rollover deadline keeps a continuously fed stream from holding one Workflow Task until + the server timeout. +- A per-activation record budget bounds one activation; rollover bounds the Workflow Task, not an + activation already running. +- Parking is an all-or-nothing handshake for the complete blocked set. Python installs provider park + intents and rechecks every member before confirming the boundary. + +The final recheck closes the classic empty-check race: a concurrent append is either found by the +recheck or paired with a wake Signal. A failed, cancelled, or superseded park attempt must remove the +intents it installed. + +## Readiness decision + +When a watcher has buffered a record, it reports readiness to Core. The acknowledgement determines +whether local delivery was accepted or a server-visible wake is still owed. + +```mermaid +flowchart TD + Ready[Watcher buffered a record] --> Notify[notify Core with Run, wait, generation] + Notify -->|Accepted| Local[Core queues or augments a resolve activation] + Notify -->|Stale| Reprobe[Re-probe and report the current generation] + Reprobe --> Notify + Notify -->|Parked| ParkWake[Send parked wake Signal] + Notify -->|No open WFT| UnparkWake[Send unparked wake Signal; keep watcher] + Notify -->|Run not found| EvictWake[Send unparked wake Signal; then tear watcher down] +``` + +Only `Accepted` transfers the delivery obligation completely to Core. A stale answer is retried and +the retry's answer controls the action. The other three terminal answers send the reserved Signal; +they differ in generation and watcher cleanup. + +Normative result meanings: [`core-lang-protocol.md`](../spec/core-lang-protocol.md). Signal encoding, +deduplication, and retry obligations: [`wake-signal.md`](../spec/wake-signal.md). + +## Three generations + +| Generation | Scope | Purpose | +|---|---|---| +| Wait generation | One subscription | Rejects readiness for an earlier blocked episode | +| Quiescence generation | One complete blocked snapshot | Correlates parking and finalization with the current wait set | +| Park generation | One confirmed park | Correlates provider intents and parked wake Signals | + +The park generation uses the quiescence generation that was confirmed. Generation zero is reserved +for an unparked recheck Signal. + +## Why Signals never carry records + +The backend remains the only stream data store. A Signal merely asks Temporal to create work when +the local Worker cannot deliver readiness into an open task. Once activated, Python rechecks all +active subscriptions rather than trusting the named topic as a complete availability statement. + +This keeps History small and makes duplicate or stale wake Signals safe: they may cost an empty +Workflow Task, but they cannot introduce a duplicate payload into History. diff --git a/arch_docs/streaming-poc-docs/guide/operations.md b/arch_docs/streaming-poc-docs/guide/operations.md new file mode 100644 index 000000000..dd6a855f6 --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/operations.md @@ -0,0 +1,108 @@ +--- +doc_id: EWS-GUIDE-OPERATIONS +status: explanatory-pre-production +audience: [readers, operators, test-authors] +normative: false +--- + +# Operations and validation + +External Workflow Streams depend on Temporal plus an external payload store. The feature remains +correct under ordinary task retry and Worker loss, but operators must preserve the records referenced +by retained History and must understand which wakeup guarantees come from producers rather than the +consumer SDK. + +## Failure classes + +| Symptom | Meaning | Expected response | +|---|---|---| +| Backend unavailable or timing out | Transient storage failure | Restore service; the Workflow Task retries | +| Recorded range, stage, or deciding History missing | Integrity loss | Restore/repair retained data or terminate the Run | +| Bytes present but payload conversion fails | Producer/consumer serialization mismatch | Align converter, codec, and serialization context | +| Replayed subscriptions differ from the marker | Workflow nondeterminism | Restore compatible code or use Workflow versioning | + +The SDK reports these separately because retrying helps only the first class. Exact error types, +completion behavior, and metrics are in [`failure-taxonomy.md`](../spec/failure-taxonomy.md). + +## Retention rule + +The external backend must retain every input record referenced by retained Workflow History and every +pending output stage whose deciding History can still be consulted. Temporal retention alone does not +preserve external payloads. + +Before deleting or trimming data, an implementation must prove that no retained marker or +Continue-As-New chain can refer to it. Provider-specific retention and garbage collection therefore +belong to the backend contract, not to Workflow code. + +## Wake durability boundary + +A producer appends first and then sends an acknowledged, idempotent wake when local delivery is not +possible. If a plain process dies after append and before the wake enters Temporal, a parked Workflow +can remain asleep even though its data is durable. + +Deployments choose one explicit mitigation: + +- Use a durable producer such as a retried Temporal Activity and treat wake acknowledgement as part + of successful publication. +- Implement a durable provider outbox and relay. +- Add a Workflow Timer sweep that periodically rechecks, accepting its History and latency cost. + +The SDK does not silently claim to close this distributed crash window. See the durability boundary +in [`wft-lifecycle.md`](../spec/wft-lifecycle.md). + +## Provider requirements at a glance + +- Immutable append-only records and stable, totally ordered opaque offsets. +- Exclusive live reads and inclusive exact-range replay reads. +- Idempotent append by stable record identity, with conflicts detected. +- Injective physical-key derivation across every stream identity component and direction. +- Race-free park-intent operations with conditional removal. +- Output stages with irreversible idempotent commit/abort and a readable-prefix barrier. +- Retention sufficient for replay and pending-stage reconciliation. + +These bullets are only a checklist. The normative operations and conformance cases are in +[`backend-contract.md`](../spec/backend-contract.md). + +## Trusting test results + +The required-test lists are executable inputs to the Python test gate, not informal documentation. +They map each declared behavior to a real test and check the expected case count: + +- [`tests-m1.md`](../required-tests/tests-m1.md) covers the implemented milestone behavior. +- [`tests-m2.md`](../required-tests/tests-m2.md) reserves the next milestone's required behavior. + +Before acting on a test result, follow [`verification-hazards.md`](../verification-hazards.md). In +particular, verify the native extension is fresh, repositories and submodules are aligned, the gate +is reading the intended test list, and fault-injection tests actually observed their control +stimulus. + +## History cost model + +History cost follows Workflow Task boundaries rather than stream record count. On the input side, +many records and several retained-task activations can collapse into one marker; additional History +events come from completed input-consumption batches, wake Signals that move an idle Run back to +active work, and forced rollovers. On the output side, every latency- or capacity-driven flush adds a +marker and the surrounding Workflow Task lifecycle, so a larger publication window trades fresher +external reads for fewer History events. + +Marker bytes are separately hard-bounded. A single-stream input run costs two offsets, a count, and +sparse control positions whether it covers ten records or 100,000. Alternating streams add schedule +transitions and may force extra rollovers. Output manifests scale with topics and activation segments, +not payload bytes; counts and fingerprints are aggregate fields. An individual frame that cannot fit +is rejected rather than allowing an unbounded marker. + +Exact input and output encodings and budget behavior are defined in +[`annotation-format.md`](../spec/annotation-format.md). Flush and rollover boundaries are defined in +[`wft-lifecycle.md`](../spec/wft-lifecycle.md). + +## Operational questions to answer before production + +- Which component owns acknowledged wake retries? +- What retention policy protects replay ranges and pending stages? +- How are storage, integrity, decode, and shutdown-wake metrics alerted differently? +- Which producer owns each output topic's explicit terminal? +- How are provider credentials and Workflow-bound serialization contexts distributed? +- What publication latency is acceptable relative to added Workflow Task and History-event cost? + +The feature is pre-production until its required validation is complete. Candidate extensions must +not be treated as implemented behavior; their status is maintained under [`../proposals/`](../proposals/). diff --git a/arch_docs/streaming-poc-docs/guide/output-commit.md b/arch_docs/streaming-poc-docs/guide/output-commit.md new file mode 100644 index 000000000..0be7d6504 --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/output-commit.md @@ -0,0 +1,105 @@ +--- +doc_id: EWS-GUIDE-OUTPUT-COMMIT +status: explanatory-pre-production +audience: [readers, design-reviewers] +normative: false +--- + +# Output commit and visibility + +Workflow-originated output has one central guarantee: + +> An external client may observe a batch only after the Workflow Task that produced it is durably +> committed in Temporal History. + +Writing visibly before Workflow Task completion could leak output from a rejected task. Writing only +after completion could lose output when a Worker dies between the server acknowledgement and the +backend write. Staging plus a compact History proof closes both windows. + +## Successful path + +```mermaid +sequenceDiagram + participant W as Workflow code + participant P as Python runtime + participant B as Backend + participant C as SDK Core + participant T as Temporal Server + participant R as External reader + + W->>P: publish logical records + P->>B: stage immutable PENDING batch + B-->>P: stage token and placed offsets + P->>C: output manifest on completion + C->>T: Workflow Task completion plus compact marker + T-->>C: task accepted + P->>B: commit stage + R->>B: read after resume boundary + B-->>R: committed records +``` + +The manifest contains identities, counts, logical-byte totals, fingerprints, activation segmentation, +and an exact History floor—not payload bytes or provider offsets exposed to Workflow code. + +## Stage state machine + +```mermaid +stateDiagram-v2 + [*] --> Buffered: Workflow publish + Buffered --> Pending: flush boundary stages batch + Pending --> Committed: exact marker token found in History + Pending --> Aborted: later WFT close or Workflow closure proves rejection + Pending --> Pending: History is not yet decisive + Committed --> Visible: provider advances readable prefix + Aborted --> Skipped: provider advances past rejected records + Visible --> [*] + Skipped --> [*] +``` + +Commit and abort are idempotent and irreversible. A pending batch is a per-topic ordering barrier: +later direct or Workflow output may already exist physically, but readers cannot pass it until the +pending predecessor is committed or aborted. + +Normative staging and barrier rules: [`backend-contract.md`](../spec/backend-contract.md). + +## What causes a flush + +Workflow output is normally staged when: + +- the activation already has a command that must complete the Workflow Task; +- the output visibility deadline expires; +- logical record, byte, or manifest capacity requires rollover; +- a park is confirmed; or +- another task boundary such as rollover or shutdown must be finalized. + +When the Workflow is quiescent and the task can be retained, output may remain in language-runtime +memory until the earliest configured publication deadline. The output deadline, input readiness, +parking, and ordinary rollover arbitrate on Core's serialized state so only one terminal boundary and +one marker win. + +Exact race behavior: [`wft-lifecycle.md`](../spec/wft-lifecycle.md). + +## Recovery after Worker loss + +A Worker normally commits its stage after the server accepts the Workflow Task. Recovery does not +depend on that Worker surviving: + +1. A reader encounters the pending head and reads the producing Run's History strictly above the + manifest's exact floor. +2. The exact stage token in a marker proves commit. +3. A later durable Workflow Task closing boundary or Workflow closure without that token proves + abort. +4. If History is not decisive yet, the batch remains pending and the reader does not guess. + +Unavailable Temporal or backend services are transient. History that is required for a decision but +has expired is an integrity failure, because neither visibility outcome can be proven safely. + +## Direct output + +Activities and external processes append output as immediately committed records. This is the better +path for token-rate or other very high-frequency output because it does not trade every latency +window for a Workflow Task lifecycle. Direct writes still obey topic ordering barriers and explicit +`FINISH` semantics. + +The Workflow path and direct path have no promised global order across topics. When both publish to +one topic, applications must also designate who owns the terminal. diff --git a/arch_docs/streaming-poc-docs/guide/replay-and-recovery.md b/arch_docs/streaming-poc-docs/guide/replay-and-recovery.md new file mode 100644 index 000000000..442161047 --- /dev/null +++ b/arch_docs/streaming-poc-docs/guide/replay-and-recovery.md @@ -0,0 +1,117 @@ +--- +doc_id: EWS-GUIDE-REPLAY-RECOVERY +status: explanatory-pre-production +audience: [readers, design-reviewers] +normative: false +--- + +# Replay, continuation, and recovery + +History does not contain stream payloads. Instead, each external-stream marker records enough +structure to fetch the exact external records again, validate them, and reproduce the order and +activation boundaries that Workflow code observed live. + +## Replay path + +```mermaid +flowchart LR + History[(Temporal History marker)] + Core[SDK Core] + Runtime[Python runtime] + Backend[(External backend)] + Validate[Range and binding validation] + Workflow[Workflow code] + + History -->|opaque annotation and manifest| Core + Core -->|ReplayExternalStreams| Runtime + Runtime -->|inclusive reads of recorded ranges| Backend + Backend -->|stored immutable bytes| Validate + Runtime --> Validate + Validate -->|recorded segment order| Workflow +``` + +Replay never waits for new records, starts readiness watchers, runs the idle timer, or performs the +park handshake. Backend reads still occur because payloads are external; an unavailable backend is a +storage failure rather than a nondeterminism error. + +Normative annotation grammar and validation: [`annotation-format.md`](../spec/annotation-format.md). + +## What the marker records + +For input, the opaque annotation records: + +- subscription bindings and starting cursor boundaries; +- inclusive offset ranges that were delivered; +- the cross-stream delivery schedule grouped into activation segments; and +- the terminal blocked-cursor snapshot. + +For Workflow output, the marker also carries a compact logical manifest identifying the staged batch +and its activation segments. It carries counts and fingerprints, not output payloads. + +Core stores and returns the input annotation as opaque bytes. Python owns its encoding and validation, +which keeps Core independent of provider offset formats and payload codecs. + +## Validation model + +Replay checks both external storage and Workflow code: + +- Recorded ranges must still exist exactly at their first and last offsets and contain no gaps. +- Provider identity and format must match the binding under which the cursor was recorded. +- Record bytes are assumed immutable because providers must guarantee structural immutability. +- Workflow code must recreate every recorded subscription binding and consume the recorded schedule. + +Missing or trimmed records are integrity loss. Present bytes that cannot be decoded are a converter +or codec mismatch. A changed subscription layout is Workflow nondeterminism. These outcomes have +different operator responses; see [`failure-taxonomy.md`](../spec/failure-taxonomy.md). + +## Commit positions + +It helps to distinguish four positions in the live runtime: + +| Position | Meaning | +|---|---| +| Committed | Cursor recorded by an accepted marker; survives eviction | +| Consumption | Last record handed to Workflow code; feeds Continue-As-New state | +| Delivery | Last record drained into an activation; feeds the marker delta | +| Prefetch | Last record buffered by the manager; disposable on retry or eviction | + +Reading, buffering, and delivery do not durably advance the stream. If a Workflow Task fails, the +runtime discards uncommitted work and reconstructs from the last accepted marker. + +Normative ownership and movement of these positions: +[`python-runtime.md`](../spec/python-runtime.md). + +## Continue-As-New + +```mermaid +sequenceDiagram + participant Old as Predecessor Run + participant H as Continue-As-New header + participant New as Successor Run + participant B as Backend + + Old->>Old: finish activation and freeze consumption boundary + Old->>H: write cursor plus stream and provider binding + H->>New: restore before Workflow code starts + New->>New: recreate subscription with same wait identity + New->>B: live read strictly after restored boundary +``` + +The continuation carries the whole binding, not just an opaque cursor. An offset has meaning only in +the provider and stream that produced it. Input cursor state and finished-output-topic state use +separate must-understand headers; an unsupported or mismatched format fails before Workflow code or +backend reads begin. + +## Recovery boundaries + +- A Workflow Task retry or Worker eviction repeats work from the last marker; it does not lose a + committed record. +- A pending Workflow output stage is resolved from its exact History proof, independently of the + original Worker. +- A parked input wake is only as durable as the component responsible for sending it. Applications + must use a durable producer, a provider outbox, or an explicit consumer sweep where the plain + producer crash window matters. + +That last point is an intentional availability boundary, not a replay limitation. The detailed +contract is in [`wft-lifecycle.md`](../spec/wft-lifecycle.md) and +[`wake-signal.md`](../spec/wake-signal.md). diff --git a/arch_docs/streaming-poc-docs/overview.md b/arch_docs/streaming-poc-docs/overview.md new file mode 100644 index 000000000..e4cf83e33 --- /dev/null +++ b/arch_docs/streaming-poc-docs/overview.md @@ -0,0 +1,43 @@ +--- +doc_id: EWS-OVERVIEW-COMPAT +status: compatibility-landing-page +audience: [all] +normative: false +--- + +# External Workflow Streams — overview + +External Workflow Streams move high-volume stream payloads out of Temporal History and into a +pluggable backend while compact marker events preserve deterministic replay. Workflows can consume +external input and publish externally readable output. Signals provide wakeups when local readiness +cannot reach an open Workflow Task; they never carry payload records. No Temporal Server change is +required. + +The documentation now separates conceptual explanation from exact implementation contracts: + +- Start with the [`human guide`](guide/README.md) for architecture, state machines, replay, output + visibility, and operational boundaries. +- Use the [`normative reference`](spec/README.md) for protocol messages, state and transition rules, + provider operations, algorithms, errors, and edge cases. +- Read [`decisions/`](decisions/README.md) for rejected alternatives and + [`proposals/`](proposals/README.md) for candidate behavior. + +## Essential model + +```text +external producer -> external backend -> Workflow (input) +Workflow/Activity -> external backend -> external client (output) + | + +-> payload bytes stay outside History + +Temporal History <- compact replay and output-commit markers +Temporal Signal <- wake metadata only, never stream data +``` + +The core lifecycle and ownership diagram is in +[`guide/architecture.md`](guide/architecture.md). Workflow Task and wake states are in +[`guide/input-lifecycle.md`](guide/input-lifecycle.md), and the output stage state machine is in +[`guide/output-commit.md`](guide/output-commit.md). + +> This page is a compatibility landing page and is not normative. Exact behavior belongs to the +> specifications indexed by stable document IDs in [`spec/README.md`](spec/README.md). diff --git a/arch_docs/streaming-poc-docs/proposals/README.md b/arch_docs/streaming-poc-docs/proposals/README.md new file mode 100644 index 000000000..f4730fa91 --- /dev/null +++ b/arch_docs/streaming-poc-docs/proposals/README.md @@ -0,0 +1,22 @@ +--- +doc_id: EWS-PROPOSAL-INDEX +status: mixed +audience: [design-reviewers, implementers, coding-agents] +--- + +# Proposals and promotion records + +Candidate extensions and promotion records live here. A candidate is not part of the accepted +External Workflow Streams specification. A promotion record may retain the rationale for an +implemented design, but current normative behavior must be cited from `spec/` and +`decisions/`, not from this directory. + +| Proposal | Status | Short overview | Detailed design | +|---|---|---|---| +| Workflow-originated external output streams | Implemented; feature-wide pre-production validation continues | [Overview](workflow-originated-external-streams-overview.md) | [Promotion design and rationale](workflow-originated-external-streams.md) | +| Workflow-to-Workflow external stream subscriptions | Future enhancement; not implemented | [Overview](workflow-to-workflow-external-streams-overview.md) | [Full candidate design](workflow-to-workflow-external-streams.md) | + +Promotion moves normative behavior into `spec/`, records each non-obvious choice in `decisions/`, +and adds required-test cases. Open cases in the executable mapping are the validation backlog for +the pre-production feature as a whole; they are not a separate maturity classification for the +output direction. diff --git a/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams-overview.md b/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams-overview.md new file mode 100644 index 000000000..74cd61303 --- /dev/null +++ b/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams-overview.md @@ -0,0 +1,65 @@ +--- +doc_id: EWS-PROPOSAL-OUTPUT-OVERVIEW +status: implemented-pre-production +audience: [design-reviewers, product-reviewers] +normative: false +--- + +# Workflow-originated external output streams — overview + +Workflow-originated output is implemented as the output half of External Workflow Streams. Workflow +code or its Activities publish payloads to an external backend, and external clients subscribe there +without making Temporal History the data store. + +> **Current authority:** this is a non-normative promotion summary. Accepted behavior lives in +> [`../spec/`](../spec/README.md) and ADR-044 through ADR-048. The feature remains pre-production +> while the required validation gate is incomplete. + +## The problem + +`temporalio.contrib.workflow_streams` stores its append-only log in Workflow state, so payload size +and replay work grow with History. That is unsuitable for agent token deltas, progress events, traces, +and other high-volume client-facing output. + +The external output direction is: + +```text +Workflow or Activity -> external backend -> external client +``` + +## Central guarantee + +An external client may observe a Workflow-produced batch only after the Workflow Task that produced +it commits in Temporal History. + +The Worker stages an unreadable batch before reporting the task, writes a compact stage proof in the +task's external-stream marker, and promotes the stage after acceptance. If that Worker disappears, a +reader or reconciler proves commit or abort from History. Direct Activity or process output is +already committed but cannot overtake an unresolved Workflow stage on the same topic. + +See the human explanation and state machine in +[`../guide/output-commit.md`](../guide/output-commit.md). + +## User-visible shape + +- Workflow topics expose deterministic `publish()` and explicit `finish()` operations. +- Activities and external processes use explicitly connected direct-output producer handles. +- External clients subscribe with opaque resumable cursor boundaries. +- Input and output handles, records, coordination metadata, and physical keys remain distinct. +- Workflow-originated batches flush on commands, visibility deadlines, capacity, parking, rollover, + or shutdown boundaries. + +Fine-grained, high-rate output should normally originate in Activities: each Workflow output flush +adds a marker and Workflow Task lifecycle to History. + +## Scope decisions retained by promotion + +- Payload bytes never enter History. +- Staged data is invisible until its exact marker proves commit. +- Pending data is a per-topic ordering barrier. +- Output capacity is measured on deterministic logical frames. +- `FINISH` is explicit and survives Continue-As-New through a must-understand header. +- No global ordering is promised across topics or between concurrent Workflow and direct producers. + +For the original end-to-end rationale, API sketch, rejected alternatives, and promotion record, read +the [`detailed design`](workflow-originated-external-streams.md). diff --git a/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams.md b/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams.md new file mode 100644 index 000000000..1ed27fdc9 --- /dev/null +++ b/arch_docs/streaming-poc-docs/proposals/workflow-originated-external-streams.md @@ -0,0 +1,746 @@ +--- +doc_id: EWS-PROPOSAL-OUTPUT-DESIGN +status: implemented-promotion-record +audience: [implementers, coding-agents, design-reviewers] +normative: false +--- + +# Detailed promotion design — Workflow-originated external output streams + +**Status:** Implemented as part of External Workflow Streams. The feature as a whole remains +pre-production. The normative contracts now live in `spec/` and ADR-044 through ADR-048; this +document retains the end-to-end rationale and API overview. `required-tests/` records remaining +feature-wide validation coverage explicitly. + +For a short decision overview, read +[`workflow-originated-external-streams-overview.md`](workflow-originated-external-streams-overview.md). + +## Summary + +External Workflow Streams include the complementary stream direction: + +```text +Workflow or its Activities -> external backend -> external client +``` + +The input direction moves externally produced records into a Workflow without putting their +payloads in History. The output direction lets a Workflow publish externally observable records under the +same constraint. It is intended for agent token deltas, progress events, traces, and other ordered +output that clients must be able to resume without making Workflow History the data store. + +This is not a storage-mode switch for `temporalio.contrib.workflow_streams`. It extends +`temporalio.contrib.external_workflow_streams` with a distinct output direction, distinct handle +types, and a client-side subscriber. The existing input direction and its accepted contracts remain +unchanged. + +The central rule is: + +> A client may observe an output batch only after the Workflow Task that produced it is durably +> committed in Temporal History. + +Writing to the backend before Workflow Task completion without such a gate leaks output from a +failed or rejected task. Writing only after completion loses output if the Worker dies after the +server accepts the task but before the backend write. The implemented protocol therefore stages output +before completing the Workflow Task, records only a compact commit proof in History, and exposes the +staged records after the server accepts that proof. + +Workflow-originated output has a **Workflow Task visibility quantum**. The first buffered publish +arms an output-flush deadline below the ordinary Workflow Task rollover deadline; reaching it +completes and rolls over a retained task even when its input subscriptions remain active. Records +therefore begin flushing no later than that configured quantum and become visible after the staging +and Workflow Task completion round trips under healthy server, Worker, and backend conditions. Each +such flush writes a marker and a Workflow Task lifecycle into History. Fine-grained, genuinely +high-rate output should consequently originate in Activities, whose direct output publishes do not +require one Workflow Task per latency window. + +## Motivation + +`temporalio.contrib.workflow_streams` serves the correct direction for UI and agent output, but its +append-only log is Workflow state and its payloads consequently become History payloads. High-volume +token streams then scale History bytes and replay work with token count. + +The current External Workflow Streams feature cannot replace it because its direction is the mirror +image: + +| Capability | External input stream | External output stream | +|---|---|---| +| Workflow role | Consumer | Producer | +| Other endpoint | External producer | External client subscriber | +| Workflow API | `subscribe()` | `publish()` | +| Wakeup | Signal wakes the Workflow | Backend watch wakes the client | +| Replay fact | Which records the Workflow consumed | Which output batch the Workflow committed | + +The Temporal Agent Harness makes the gap concrete. Activities and Workflow code publish a single +ordered event log, clients resume it by offset, and the log is the live UI protocol. Moving only the +Activity-to-Workflow token ingress to the current feature would be a useful hybrid, but it would not +externalize the client-facing log. + +## Goals + +- Keep output payload bytes out of Temporal History. +- Never expose output from a Workflow Task that did not commit. +- Recover output whose Workflow Task committed when the producing Worker died before making it + visible. +- Give clients opaque resumable cursors and direct backend reads. +- Preserve publish order within each committed Workflow batch, and prevent later committed records + from overtaking an unresolved staged predecessor on the same topic. +- Keep Workflow code deterministic and free of backend connections or credentials. +- Span a Continue-As-New chain and remain isolated from later reuse of the same Workflow ID. +- Reuse the configured backend, `DataConverter`, serialization context, immutable-record contract, + and producer idempotency rules where their semantics match. + +## Non-goals + +- A transaction spanning Temporal and an arbitrary external database. The protocol supplies a + recoverable commit proof, not distributed atomic commit. +- Making output visible before its producing Workflow Task commits. +- Returning a provider offset synchronously to Workflow code. That offset is not durable until the + task commits and must not influence deterministic Workflow control flow. +- A global order or atomic snapshot across different topics. Ordering and batch visibility are + per-topic. +- A deterministic total order between a concurrently running Activity and Workflow Task retry. The + backend barrier prevents overtaking an unresolved predecessor, but an aborted attempt contributes + no records and its retry may land after Activity output appended in between. +- Letting browser code hold backend credentials. A web service may proxy the client subscriber to + SSE or WebSocket clients. +- Replacing the existing input direction or deprecating `contrib.workflow_streams` in the first + release. +- Token-rate publishing from Workflow code at no History-event cost. Output-flush rollover bounds + latency by trading one marker plus Workflow Task lifecycle events for each flush window. + +## Public API + +These names are the implemented Python surface. Workflow code uses +`ExternalOutputStreamOptions` and `ExternalOutputStreamTopic`; direct producers use +`ExternalOutputStreamProducer` and `ExternalOutputStreamProducerTopic`; readers use +`ExternalOutputStreamClient`, `ExternalOutputStreamClientTopic`, and `ExternalOutputStreamItem`. + +### Workflow publisher + +```python +from datetime import timedelta + +from temporalio.contrib.external_workflow_streams import external_output_stream + +events = external_output_stream.with_options( + max_publish_latency=timedelta(milliseconds=100), +).topic("agent-events", type=AgentEvent) + +await events.publish(AgentEvent(...)) +await events.finish() +``` + +`publish()` ordinarily accepts the value into the current Workflow Task's output batch without a +backend round trip and returns no external offset. Making it awaitable gives the runtime a +deterministic backpressure point: when the batch's record or logical-byte budget is exhausted, the +await yields so the current batch can be staged and a new Workflow Task can continue publishing. +The first retained completion carrying accepted output arms `max_publish_latency`; if that deadline +wins first, Core asks the out-of-sandbox Python runtime to finalize, Python stages the accumulated +batch, and Core rolls over the retained Workflow Task. + +`finish()` appends an ordered terminal control record for this topic. It is explicit because a +Workflow failure or termination cannot reliably execute cleanup code. A client may also stop by +observing the Workflow's terminal status through Temporal; it must not infer completion from an idle +backend watch. + +The Workflow-facing handle has `publish()` and no `subscribe()`. The current Workflow-facing input +handle keeps `subscribe()` and no `publish()`. + +### Activity or external output producer + +```python +producer = await ExternalOutputStreamProducer.connect( + backend=RedisStreamBackend(url=...), + workflow=WorkflowChainKey( + namespace=..., + workflow_id=..., + first_execution_run_id=..., + ), + client=client, + session_id=..., # stable across Activity retry +) +events = producer.topic("agent-events", type=AgentEvent) + +await events.publish(AgentEvent(...)) +await events.finish_writing() +``` + +Activities need this path for live token and tool events. The chain key is explicit for the same +reason it is on the current producer: Activity info does not contain the first execution Run ID. +Workflow code threads the key through an Activity input or an opaque application context. + +Unlike the current input producer, an output producer sends no wake Signal to the Workflow. Its +consumer watches the backend directly. Its records are immediately committed singleton batches, +idempotent on `(session_id, sequence)`. `finish_writing()` closes that producer's topic handle; when +direct and Workflow publishers share a topic, the application chooses one terminal owner because a +direct producer cannot mutate the Workflow's continuation state and clients stop at the first +`FINISH`. + +### External client subscriber + +```python +reader = await ExternalOutputStreamClient.connect( + backend=RedisStreamBackend(url=...), + workflow=WorkflowChainKey(...), + client=client, +) +events = reader.topic("agent-events", type=AgentEvent) + +async for item in events.subscribe(after=cursor): + render(item.data) + cursor = item.offset +``` + +The yielded item carries both the decoded value and an opaque provider offset. `BEGINNING` and +`AFTER(offset)` retain their current boundary meanings. A topic also exposes a committed `tail()` +boundary so a caller can establish a safe resume point before sending an Update that causes new +output. + +The Temporal client is required, not optional. It verifies the Workflow chain binding and is the +fallback authority for resolving a staged batch when a Worker died in the acknowledgement window. +The common path remains a backend-only read after connection. + +`tail()` is only a position boundary; it does not correlate a later Update with a particular output +record. A protocol that supports concurrent Updates must also return a stable operation identifier +such as `turn_id`, and the subscriber must scan for the matching `turn_started(turn_id)` rather than +adopting the first turn marker after the boundary. + +## Visibility and History cost + +With no retained input wait, a normal Workflow Task completes after its activation and output is +staged on that completion. With retained external-input waits, three conditions can flush output: + +1. the activation produces another server-bound command, so the task already has to complete; +2. the output batch reaches its record or logical-byte budget; or +3. the output-flush deadline expires. + +The third is a distinct Core timer, armed by the first output in an empty batch and clamped below the +ordinary rollover deadline. It completes with an output-flush terminal, writes the output marker, +and requests a new Workflow Task so input consumption continues. Replay follows the recorded +terminal and never arms a wall-clock output deadline. + +### Output flush and input parking use one terminal race + +The output deadline does not run beside the input direction's park handshake without coordination. +Core serializes the output-flush, idle/all-fenced park, readiness, ordinary rollover, and +server-command completion transitions for the open Workflow Task; exactly one terminal transition +wins. + +- If `ParkSetConfirmed` wins, the output batch is staged and its marker is included before that park + completion is sent to the server. The output timer is cancelled; parking already provides an + earlier visibility boundary. +- If the backend recheck becomes ready, Python returns `WorkflowOutputStreamBuffered` before + `StreamSetBecameReady`; the current park preparation is rolled back and the original output + deadline remains armed for the still-open Workflow Task. +- If readiness wins after Python already staged output for a confirmation, Core discards the stale + park terminal, issues resolve to retire every intent, then finalizes with `TASK_COMPLETED`, records + the carried commit once, and forces a replacement Workflow Task. +- If the output-flush deadline wins while `PrepareExternalStreamPark` is outstanding, Core aborts + that quiescence generation. When the outstanding result returns, Core ignores its park terminal, + issues resolve to remove or record every owed intent removal, then finalizes with + `OUTPUT_LATENCY`. A commit already carried by the losing result is retained; otherwise Python + stages during finalization. Core records it once and completes with `force_new_wft = true`. The + wait generations remain active and are reconstructed in the replacement task; the output flush + never confirms a park generation. +- A late `ParkSetConfirmed`, readiness result, or deadline result for the losing transition is stale + and cannot produce a second marker or completion. + +The output-flush deadline is reduced by `min` across non-empty output topics and can only move +earlier as a shorter-latency topic joins the batch. It is clamped below ordinary rollover; an idle or +all-fenced park may still finish earlier and flush the same batch. These precedence and generation +rules are part of the required latency-flush ADR, not an implementation detail left to timer order. + +The visibility guarantee is conditional, not a delivery SLA: under healthy dependencies a +Workflow-originated record is visible after at most `max_publish_latency` plus staging and Workflow +Task completion latency. A backend outage blocks that completion, and an ambiguous server outcome +can hold the client at a pending barrier until History decides it. + +Marker **bytes** scale with topic/segment metadata rather than record payload bytes. History +**events** scale with output flushes: each latency- or capacity-driven flush adds an output marker +and the surrounding Workflow Task lifecycle events. Increasing the latency window batches more +output at the cost of staler clients. High-rate token deltas should be emitted by an Activity-side +`ExternalOutputStreamProducer`; Workflow publishes are for coarser deterministic lifecycle and +protocol events. + +## Stream identity and isolation + +Direction is part of the physical identity: + +```text +(namespace, workflow ID, first execution Run ID, direction, stream name) +``` + +It is not a user-controlled prefix added to `stream name`. A topic named `events` may exist in both +directions without either side consuming the other's records, idempotency keys, or coordination +metadata. Providers may represent this with distinct key types or an internal direction component, +but key derivation must remain injective. + +The first execution Run ID keeps the output stable across Continue-As-New and prevents a later +Workflow using the same Workflow ID from inheriting the earlier chain's output. + +## Commit protocol + +### Stage-attempt identity + +The out-of-sandbox Worker runtime mints a cryptographically unique, opaque `stage_token` for every +attempt to stage one Workflow Task's output. The token is not derived from a +`WorkflowTaskScheduled` or `WorkflowTaskStarted` event ID: speculative Workflow Tasks may discard +those events and later reuse their IDs. It is never exposed to Workflow code. + +One stage token covers the flush, with `(stage_token, topic, publish_index)` identifying each record +inside its topic sub-batch. Repeating a stage after losing the backend acknowledgement reuses the +same token and manifest and is idempotent. Re-executing the Workflow Task after rejection, timeout, +or Worker loss mints a fresh token, even if the server reuses every speculative event ID and the +logical output is byte-identical. + +The staged metadata also carries a `history_floor_event_id`. Core derives it by locating this +Workflow Task's own `WorkflowTaskScheduled` in the ordered History view used to build the activation +and taking the event immediately preceding that Scheduled event. It does **not** try to decide +whether the Scheduled or Started events it was handed are persisted. For a speculative task those +events may disappear, but their predecessor is still the durable boundary before the speculative +transaction. + +Reconciliation scans strictly above the floor. The derivation is mandatory because the safety +direction is asymmetric: a floor moved backward across a prior Workflow Task closing event lets a +resolver see that old close while this task is still in flight and abort output which later commits. +A floor moved forward, while still below this task's possible outcome, can only postpone resolution. +Core refuses to stage if it cannot identify the exact predecessor; an implementation must never +substitute a conservative lower event ID. The marker carries the unique token; the floor only says +where a resolver may begin looking for the first outcome capable of deciding that token. + +An immutable manifest accompanies the staged batch: + +- provider and format version; +- stage token, History floor, topic, and sub-batch identity; +- record count; +- a rolling fingerprint of the logical, pre-codec payloads and control records; and +- the logical, pre-codec byte count used for capacity enforcement. + +The fingerprint detects a retry of the *same stage operation* trying to reuse its token for +different logical output without requiring encoded bytes to be deterministic. A payload codec may +use randomness; the first successfully staged encoded bytes remain authoritative while the logical +fingerprint is what a repeated stage compares. A new Workflow Task execution uses a new token and +cannot conflict with an orphaned speculative attempt. + +"Logical" has one exact boundary: the Workflow sandbox runs the deterministic `PayloadConverter`, +then constructs a versioned canonical frame without serializing the `Payload` protobuf itself. For +fingerprint version 1, each record frame contains unambiguous unsigned length prefixes around: + +1. the UTF-8 topic bytes and record kind; +2. every `Payload.metadata` key/value pair, keys sorted lexicographically by their UTF-8 bytes; and +3. the raw `Payload.data` bytes. + +The logical byte count is the sum of those canonical frame lengths. The batch fingerprint is +SHA-256 over the ordered, length-prefixed frames. Sorting the metadata is load-bearing: it is a +protobuf map, and default protobuf serialization does not promise map-entry order. The fingerprint +format version rides in the manifest so a later canonicalization cannot be mistaken for the current +one. + +The out-of-sandbox runtime subsequently applies the `PayloadCodec` and any external payload-storage +transform. A custom `PayloadConverter` used by Workflow code remains subject to Temporal's ordinary +determinism requirement; a randomized or version-changed `PayloadCodec` may alter stored bytes and +length without altering the logical identity or segmentation. + +### Successful path + +```text +Workflow activation + -> publish values into a bounded language-side batch + -> Worker runtime mints a unique stage token + -> Core asks the out-of-sandbox runtime to stage the batch under that token + -> backend durably stores immutable records as PENDING + -> runtime acknowledges the stage + -> Core completes the Workflow Task with a compact output marker + -> Temporal Server commits the marker + -> Worker marks the staged batch COMMITTED + -> client watch may yield the records +``` + +The marker contains the stage token and compact manifest, never the payloads. Marker size is +therefore proportional to topic transitions and retained-task activation segments rather than +record payload bytes. A hard manifest byte budget forces Workflow Task rollover before the marker +can approach the server event-size limit. + +Core must not send the server completion until staging is durable. The backend must not let a reader +pass a pending batch. Together those rules prevent both missing committed output and speculative +output reordering. + +### Failed and ambiguous paths + +| Failure point | Required outcome | +|---|---| +| Before staging | No backend records and no marker | +| During staging, outcome unknown | The same token is retried while its runtime exists; otherwise the batch remains pending for reconciliation | +| Stage succeeds, Workflow Task is explicitly rejected or fails | Batch is marked aborted and readers skip it | +| Server accepts the marker, Worker dies before commit | Batch remains pending; a reader or reconciler proves the marker from History and commits it | +| Commit acknowledgement is lost | Repeating commit is an idempotent success | +| Pending batch has no provable History outcome yet | Readers stop at it; they do not guess or pass it | +| History contains a later Workflow Task closing event or Workflow closure without this token | Batch is idempotently aborted | +| History needed to decide has expired | Integrity failure; never expose or silently discard the batch | + +A pending batch is a barrier in topic order. Later Activity records may already be durable, but a +client cannot yield them until every preceding batch is committed or aborted. Without that barrier, +an Activity scheduled by the producing Workflow Task could overtake the Workflow events that +logically preceded it. + +### Reconciliation is part of correctness + +The acknowledgement window exists even when the Worker performs the normal post-commit transition: +the process can always die after the server commit and before the backend commit. Recovery cannot +depend on another Workflow Task because the committed task may have completed the Workflow. + +Therefore both are required: + +1. Workers opportunistically reconcile pending batches for the chains they service. +2. `ExternalOutputStreamClient` resolves a pending head lazily by reading the relevant History and + matching the compact marker. + +An optional standalone reconciler can keep the slow path away from readers, but it is an operational +optimization rather than the only recovery mechanism. Backend credentials alone are insufficient +to resolve pending state; the resolver also needs permission to read the Workflow's History. + +Resolution is a positive predicate over History, never a timeout and never absence alone. The +resolver fetches a complete, strongly consistent History response from the staged +`history_floor_event_id` through the current durable end; it does not decide from a partial page or +an eventually consistent cache. It then applies these rules in order: + +1. If an external-output marker carrying this exact `stage_token` exists, commit the batch. +2. Otherwise, if History contains a `WorkflowTaskCompleted`, `WorkflowTaskFailed`, or + `WorkflowTaskTimedOut` event above the floor, or the Workflow has closed, abort the batch. A + committed task's marker and closing event belong to the same durable server transaction; once a + complete response contains that closing boundary without the token, the omitted token cannot be + appended retroactively. +3. Otherwise the outcome is still undecidable and the batch remains pending. + +This rule survives speculative Workflow Tasks whose Scheduled and Started events were never +persisted. A discarded attempt's unique token cannot match the retry's marker, and the first durable +Workflow Task closing boundary above its floor proves the orphan absent. Reused event IDs and +byte-identical manifests are irrelevant because neither is the identity. + +The pending record remains a head-of-line barrier until that proof arrives. There is deliberately no +finite timeout guarantee for an abandoned speculative task. If its Worker dies, its Update caller +disconnects so the server never redelivers it, and nothing else touches the Workflow, the +speculative Scheduled/Started/TimedOut events may all remain absent and neither rule can fire. The +batch stays pending until a later durable Workflow Task outcome or Workflow closure provides the +positive boundary; if History retention expires first, it becomes an integrity failure. + +The usual harness path resolves sooner because the stream reader is also waiting for the Update and +keeps that operation alive, but that is an application property rather than a stream guarantee. +Worker unavailability, server outage, History read failure, caller disconnect, or retry backoff can +extend the wall-clock delay without bound. Readers report that they are waiting on reconciliation +rather than presenting it as ordinary backend idleness. A future resolver may add another positive +proof source, but it may not turn elapsed time or a missing token alone into abort. + +## Replay and determinism + +On replay, Workflow code executes the same `publish()` calls. Core matches their topic, order, +control kind, logical size, and logical fingerprint against the recorded output marker. It does not +mint a new token, stage records, commit records, arm an output-flush timer, or recompute a batch +boundary. A mismatch is nondeterminism. + +Batch segmentation is recorded and reproduced, not inferred from the current codec or current +runtime limits. The marker groups output commands into the same activation segments the live +retained Workflow Task observed, including an empty segment where a live activation performed a +drain but published nothing. Replay performs exactly the same number of event-loop drains as live +execution. As in ADR-018, a marker with *k* segments performs the first *k - 1* drains in the replay +driver and leaves the final drain to the activation's ordinary trailing `_run_once`. + +Input and output annotations for one retained Workflow Task share this segment schedule. Their +drivers must not each perform *k* drains and double the live schedule; they attach their deliveries +and output-command expectations to one ordered sequence of segment frames. Capacity- and +latency-driven output terminals are part of that recorded schedule. The live budget check and +wall-clock deadline are disabled on replay, so a changed compression ratio or Worker option cannot +move the suspension boundary. + +The marker is the durable fact that a batch belongs to the execution. Backend committed state is +not accepted as a substitute: backend state can outlive History, can be copied incorrectly, and is +not part of the deterministic event sequence. + +Workflow code never observes: + +- provider offsets; +- whether a client is connected; +- when post-commit promotion completes; or +- backend retry timing. + +None of those facts may affect Workflow control flow. An application that needs a durable client +acknowledgement sends it back through a normal Update, Signal, or the existing external input +stream; output-stream delivery itself is not an acknowledgement channel. + +## Backend contract additions + +The exact interface is provider work, but its semantics must include: + +- atomically stage an immutable, ordered batch as pending; +- idempotently commit or abort that exact batch; +- reject reuse of one stage token with a different manifest; +- read committed records strictly after a cursor while stopping at unresolved pending data; +- inspect a pending stage token, History floor, Run identity, and manifest for reconciliation; +- retain resolution metadata as long as a cursor or retained History may refer to it; and +- garbage-collect abandoned staging only after History proves abort or after an operator resolves an + integrity incident. + +Record immutability remains mandatory. Commit and abort are separate coordination metadata; they do +not rewrite record bytes. A Redis provider can use an append-only record log plus a status index and +atomic scripts. Providers that cannot stop a read at unresolved data do not satisfy the output +contract even if they satisfy the current input contract. + +Input and output conformance suites and capability ABCs are separate. An input-only +`StreamBackend` remains valid, and direct output producers and clients can use an output-only +`OutputStreamBackend`. The first-release Worker option is narrower: it is validated as a +`StreamBackend`, so Workflow-originated output requires that configured provider to implement both +contracts. Requiring every existing input provider to support transactional output staging would +turn an additive feature into a breaking change. + +## Ordering guarantees + +For one topic, clients observe: + +1. Workflow records in publish order within their committed Workflow Task; +2. committed batches in backend order; and +3. directly produced Activity records at their assigned backend offsets, without passing an + unresolved predecessor during reads. + +Retries do not create a second visible record for the same publish identity. Aborted batches occupy +coordination positions but yield no data. + +This is a barrier guarantee, not one deterministic total order between independently running +producers. An Activity may append while a Workflow stage is pending. If that stage commits, readers +observe its records before the Activity because the pending barrier was already ahead of it. If the +stage aborts, it yields nothing and a newly executed Workflow Task's replacement batch may append +after the Activity. That relative order can therefore change across a Workflow Task retry, and no +Workflow code may depend on it. + +There is no cross-topic transaction. If one Workflow Task publishes to topics A and B, both markers +commit in Temporal together, but backend promotion and client observation may occur at different +times. A consumer needing one total order should publish envelopes to one topic and put the logical +channel in the envelope. + +## Capacity and backpressure + +Output payloads do not go to the server, but they still cross the Workflow sandbox and the +language/Core boundary. The runtime enforces record and **logical pre-codec byte** budgets per +output batch, which means per Workflow Task rather than per activation. A retained Workflow Task +may span many activations; all of their accepted output counts against the same batch bound. When +accepting another record would cross a limit, `await publish()` yields and requests Workflow Task +rollover; the next Workflow Task resumes after the preceding batch is staged. + +Logical size is the deterministic canonical frame size defined above, before any `PayloadCodec`. +Encoded size is provider/transport cost and never chooses a Workflow-visible boundary. A provider +may stream one logical stage through several idempotent +transport writes under the same pending token, then seal its manifest; it must not force a new +Workflow segment merely because a codec changed compression ratio. + +Replay takes segment boundaries from the marker and skips both capacity checks and the +output-flush deadline. It still validates the recorded logical counts and fingerprints. A Worker +with a lower current live limit can replay an older larger batch because the bytes are already +staged; applying the new limit would create nondeterminism instead of providing protection. + +A single record larger than the configured maximum is rejected before it enters the batch. The SDK +must not split a user value into transport fragments unless the public codec contract explicitly +supports it. + +Client-side subscribers have independent record and byte prefetch limits. A slow client affects its +own memory and retention position, not Workflow Task progress. + +## Continue-As-New, completion, and retention + +The output key uses the first execution Run ID, so Continue-As-New keeps the same topic. Staged +metadata names the current Run, and the independently unique stage token prevents retries and +successive Runs from colliding even when speculative Workflow Task event IDs are reused. + +Continue-As-New is not a terminal record. An explicit Workflow `finish()` remains visible across the chain +and prevents later publishes to that topic. The finished-topic set plus the configured provider ID +and format version travels in a reserved must-understand Continue-As-New header. The +successor restores it before Workflow code runs and deterministically rejects a later `publish()`; +it never asks mutable backend state whether the topic was finished. This is the output analogue of +ADR-022 and ADR-039. + +Normal Workflow completion does **not** synthesize terminal records. Only explicit `finish()` from +Workflow code or `finish_writing()` from a direct producer appends `FINISH`. Failure, cancellation, +termination, and successful completion therefore do not close still-open topics. A client that +needs to stop automatically combines backend watches with Temporal execution status. + +Committed records and their resolution metadata must remain available for at least the advertised +client resume window. Pending resolution requires the associated History to remain available. +Operators must configure backend retention and Temporal retention together; deleting either side +first can turn a recoverable acknowledgement window into an integrity failure. + +## Availability and failure classification + +Staging is on the Workflow Task commit path. If the output backend is unreachable, times out, or +rejects a transient operation, Core does not send the server completion: the Workflow Task fails +with the external-storage cause and normal Temporal retry, and the Workflow makes no progress until +staging succeeds. This maps to the transient `StreamStorageError` row in +`spec/failure-taxonomy.md`, with a distinct output-stage operation label on metrics. + +That cost is deliberate. Completing the Workflow Task without a durable stage can permanently lose +committed output. Applications that prefer Workflow progress over output durability should use a +separately specified best-effort telemetry sink, not weaken this stream's contract. + +A missing staged record, conflicting immutable manifest, or History that expired before a pending +token could be resolved is `StreamIntegrityError`: the reader or reconciler blocks rather than +choosing commit or abort. Decode failures remain `StreamDecodeError`. Backend or Temporal outages +during client reconciliation are transient storage failures and leave the barrier pending. + +## Security and authorization + +Output records are application data and inherit the `DataConverter` and codec configured for the +Workflow's serialization context. The output producer and client bind the converter to the same +namespace and Workflow ID context before encoding or decoding. + +Backend possession is not proof that a caller may read every Workflow stream. A production provider +needs key-level authorization or a trusted service boundary. The SDK API should not imply that a +raw Redis URL is a suitable browser credential. + +History access used for reconciliation is also privileged. A service may split ordinary committed +reads from reconciliation so most readers hold only backend read permission and delegate pending +resolution to a trusted component. + +## Temporal Agent Harness migration + +The harness can migrate without changing its external event +vocabulary: + +1. `AgentWorkflowRunner` publishes `AgentEvent` envelopes to one external output topic. +2. model and tool Activities use `ExternalOutputStreamProducer` for live deltas and lifecycle + events on that same topic; +3. `AgentClient` and the web service subscribe with `ExternalOutputStreamClient` and persist opaque + cursors instead of dense integer offsets; +4. the client obtains `tail()` before sending an agent-message Update, uses that earlier boundary + as the merge start hint, then scans for the exact `turn_id` returned by the Update rather than + accepting another concurrent turn's marker; and +5. the Nexus adapter carries opaque cursor tokens and reads the backend instead of calling the + private WorkflowStream poll Update. + +Taking `tail()` before the Update admits harmless earlier events but cannot miss the subsequent +`turn_started`. Matching `turn_started.turn_id` to the Update result makes the scan safe when another +caller submits a concurrent Update after the same boundary. This removes the harness's dependency on +the private in-Workflow `_on_offset()` head query, which an externally assigned opaque offset cannot +reproduce deterministically inside the Workflow. + +A rollout may dual-publish to `contrib.workflow_streams` and the external output topic for +comparison, but consumers choose exactly one authoritative log. Merging both would duplicate every +event and give two incomparable cursor spaces. + +## Required validation + +An implementation is not complete without tests that force every acknowledgement window: + +- no record becomes visible before the Workflow Task marker commits; +- an explicitly failed or rejected task leaves no visible output; +- a discarded speculative Workflow Task stages one token and its redelivery stages another when + event IDs are reused, both when logical output is identical and when its manifest changes, and + History commits only the token in its marker; +- a pending speculative token aborts after the first durable Workflow Task closing boundary above + its History floor, without waiting for nonexistent Scheduled or Started events; +- the History floor is the event immediately preceding the producing task's Scheduled event; a + previous Workflow Task close is excluded from the scan, and mutating the floor below it makes the + test fail by demonstrating the otherwise-possible false abort; +- a disconnected speculative Update with no later durable event remains pending rather than being + aborted by elapsed time, eventually classifying retention loss as integrity failure; +- crash after stage and before server completion resolves to abort or retry without a phantom; +- crash after server completion and before backend commit is repaired by a cold client after the + Workflow has completed; +- repeated stage, commit, abort, and reconciliation calls are idempotent; +- identity reuse with a different logical fingerprint is rejected; +- replay validates output commands but performs no backend write, token mint, capacity split, or + wall-clock flush; +- a codec whose encoded size changes between live execution and replay preserves the marker's batch + and activation segmentation; +- a multi-activation retained Workflow Task reproduces exactly the live drain count when both input + and output annotations are present, rather than summing both drivers' drains; +- a Workflow-originated publish made during a retained Workflow Task becomes visible within the + configured output-flush quantum under healthy dependencies; +- several publishes inside one flush window produce one output marker, while crossing *n* latency + windows produces *n* markers and the corresponding Workflow Task lifecycle cost; +- a pending batch blocks later Activity output from overtaking it; +- Activity retry produces one visible record per idempotency identity; +- cursor resume has neither gaps nor duplicates across Workflow Task rollover and + Continue-As-New; +- input and output topics with the same user name remain isolated; +- randomized payload codecs do not make a logical retry conflict; +- two logically identical Payloads whose metadata maps were inserted in opposite orders have the + same versioned canonical frame, byte count, and SHA-256 fingerprint; +- output-flush/park races in both orders produce exactly one Workflow Task terminal and one output + marker; an output winner invalidates the quiescence generation, rolls back every installed intent, + and forces a replacement Workflow Task, while a park winner stages the output before completing; +- concurrent message Updates scan from their boundaries to their own returned `turn_id` and cannot + adopt each other's `turn_started`; +- a finished topic survives Continue-As-New through the reserved header and rejects a successor + publish without reading backend state; +- an output-stage outage blocks Workflow Task progress and is classified as transient external + storage failure; +- a missing staged record or expired History produces an integrity failure; and +- marker and activation batch budgets are hard bounds, including one oversized topic name and one + oversized record. + +Every crash test must prove that its injected failure reached the intended boundary before its +result is accepted; the existing verification-hazard rules apply unchanged. + +## Normative promotion + +The durable rules are promoted into their single normative homes: + +- ADR-044 and `spec/backend-contract.md`: stage tokens and exact History-floor reconciliation; +- ADR-045 and `spec/wft-lifecycle.md`: latency flush and the shared output/park terminal race; +- ADR-046 and `spec/annotation-format.md`: logical capacity and shared replay segmentation; +- ADR-047 and `spec/backend-contract.md`: pending ordering barriers and mixed producers; and +- ADR-048 and `spec/annotation-format.md`: finished-topic Continue-As-New state. + +`spec/core-lang-protocol.md`, `spec/python-runtime.md`, and `spec/failure-taxonomy.md` own the +cross-boundary mechanics and failure reporting. The 27 validation cases are promoted into +`required-tests/`; this document is not a second normative copy of those contracts. + +## Alternatives considered + +### Publish through an Activity + +An Activity provides durable retries, but its argument or result payload is recorded in History. +Passing the output value as the Activity input therefore defeats the feature. Passing only a key +requires the value to have been written externally already and returns to the same commit problem. + +### Write after Workflow Task completion + +This prevents speculative output but loses a committed batch if the Worker dies in the +server-accepted/backend-not-written window. There is no later Workflow Task to repair it when the +task completed the Workflow. + +### Write visibly before Workflow Task completion + +This survives a Worker crash but leaks events from rejected tasks. A client can render an answer the +Workflow never committed, and later replay can produce a different answer under the same turn. + +### Keep using `contrib.workflow_streams` + +This remains valid for moderate-volume streams and requires none of the protocol above. It does not +meet the goal of keeping payload bytes out of History. + +### Deliver Activity output directly and Workflow output through the old stream + +This hybrid reduces token Signal volume and may be a useful intermediate harness migration. It +creates two logs, two cursor spaces, and no single ordering between Activity and Workflow events, so +it is not the implemented end state recorded here. + +## Resolved public and operational decisions + +- The public names are `external_output_stream`, `ExternalOutputStreamOptions`, + `ExternalOutputStreamTopic`, `ExternalOutputStreamProducer`, + `ExternalOutputStreamProducerTopic`, `ExternalOutputStreamClient`, + `ExternalOutputStreamClientTopic`, and `ExternalOutputStreamItem`. +- Lazy History reconciliation is built into every `ExternalOutputStreamClient`; it is not a + pluggable optional resolver. Workers also reconcile opportunistically after reporting. +- Topic termination is explicit only: `finish()` and `finish_writing()` append it; Workflow + completion does not. +- The default `max_publish_latency` is 100 milliseconds. The minimum configured latency among the + non-empty topics in one batch controls its deadline. +- Input and output provider capabilities are independent. Input-only providers remain valid; + output-only providers serve direct producers and clients; Workflow-originated output on a Worker + currently requires one provider implementing both contracts. +- The SDK sets no default committed-output retention and exposes no force-resolve operator API. + Providers retain records for their advertised resume window and resolution metadata for at least + the matching History window. Missing proof is an integrity incident to repair or terminate, never + a timeout-based guess. +- Activity/external direct producers and Workflow publishers may share a topic. Pending Workflow + stages are ordering barriers, but the SDK promises neither a global cross-producer order nor a + cross-topic transaction. diff --git a/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams-overview.md b/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams-overview.md new file mode 100644 index 000000000..a5b537a0d --- /dev/null +++ b/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams-overview.md @@ -0,0 +1,65 @@ +--- +doc_id: EWS-PROPOSAL-W2W-OVERVIEW +status: future-not-implemented +audience: [design-reviewers, product-reviewers] +normative: false +--- + +# Workflow-to-Workflow external stream subscriptions — overview + +This candidate extension would allow one Workflow to consume another Workflow's committed external +output directly, without copying payloads through either Workflow's History or through a second +external stream. + +> **Status:** future enhancement; not implemented and not part of the accepted specification. + +## Proposed data and wake path + +```mermaid +flowchart LR + A[Source Workflow A] + Output[(A output topic)] + Outbox[(Durable notification outbox)] + Signal[Raw wake Signal] + B[Consumer Workflow B] + + A -->|staged then committed output| Output + Output -->|atomic visibility obligation| Outbox + Outbox -. idempotent dispatch .-> Signal + Signal -. wake only, no payload .-> B + Output -->|committed payload read in place| B +``` + +B would record the exact source binding and consumed ranges in its own replay marker. A's output +records would remain under A's existing output key; the design deliberately does not alias or copy +them into B's input key. + +## Why existing APIs are insufficient + +- A's output key identifies A, but a wake must target B. +- An evicted B has no live backend watcher, so subscriptions and notifications must be durable. +- B's current input annotation implies B's own input key and serialization context. +- B must stop at unresolved output stages until A's History proves commit or abort. + +The hard part is therefore durable discovery and wakeup, not reading the payload bytes. + +## Proposed additions + +- An explicit source Workflow/output-topic reference and distinct Workflow-side handle. +- A provider capability for durable foreign-output registrations. +- Atomic visibility notifications plus reconciliation and notification outboxes. +- A versioned cross-Workflow wake envelope. +- Source bindings in replay annotations and Continue-As-New state. +- Replay-retention leases and a default-deny authorizer. + +The first version would be same-namespace, broadcast-only, explicitly authorized, and limited to +providers capable of atomic visibility notifications. + +## Decision boundary + +Adopting this proposal would expand the provider, wake, replay, authorization, retention, and Core ↔ +language protocols. None of those additions should appear in accepted `spec/` documents until the +proposal is approved and its required tests are defined. + +For exact APIs, races, outbox semantics, failure handling, rollout, and the proposed invariant set, +read the [`detailed proposal`](workflow-to-workflow-external-streams.md). diff --git a/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams.md b/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams.md new file mode 100644 index 000000000..046f326bc --- /dev/null +++ b/arch_docs/streaming-poc-docs/proposals/workflow-to-workflow-external-streams.md @@ -0,0 +1,818 @@ +--- +doc_id: EWS-PROPOSAL-W2W-DESIGN +status: future-not-implemented +audience: [implementers, coding-agents, design-reviewers] +normative: false +--- + +# Detailed proposal — Workflow-to-Workflow External Stream Subscriptions + +**Status:** Future enhancement; not implemented + +**Scope:** Temporal Python SDK, SDK Core, and External Workflow Streams providers + +**Depends on:** External input streams, Workflow-originated external output streams, and their +existing replay, output-staging, and wake-signal protocols + +For a short decision overview, read +[`workflow-to-workflow-external-streams-overview.md`](workflow-to-workflow-external-streams-overview.md). + +## Summary + +Allow one Workflow to consume a committed external output stream owned by another Workflow without +copying payloads through Temporal History or through a second external stream: + +```text +source Workflow A + -> OUTPUT stream in the external backend + -> durable visibility notification + -> wake Signal carrying no payload + -> consumer Workflow B reads A's committed OUTPUT records directly +``` + +The data path is zero-copy: A's output records remain under A's existing `OUTPUT` `StreamKey`, and B +reads those records in place. B does **not** alias A's output as one of B's input streams. Its replay +marker records the exact source binding and committed output ranges that B observed. + +The hard part is wakeup, not reading. An output key identifies A but does not identify the workflows +that consume it. The design therefore adds durable consumer registrations and provider-backed +reconciliation and notification outboxes. Making output visible and creating notification +obligations are one atomic provider transition. Background dispatchers resolve stranded pending +stages and turn visibility obligations into idempotent raw Temporal Signals addressed to B. This +remains correct when A, B, or either Worker restarts. + +The first version is same-namespace only, broadcast-only, explicitly authorized, and available only +for providers that implement a new optional cross-Workflow subscription capability. + +## Decision + +Introduce a distinct Workflow-side subscription handle for reading another Workflow's external +output. Reuse the existing: + +- source output physical records and pending-stage barrier; +- Workflow Task marker and activation-segment schedule; +- global wait-set quiescence and parking protocol; +- raw, codec-bypassing wake Signal transport; and +- opaque provider cursors. + +Add: + +- an exact `WorkflowOutputStreamReference` containing the source chain and topic; +- an optional `WorkflowOutputSubscriptionBackend` provider capability; +- a durable registration for every foreign output subscription; +- atomic output-visibility notifications plus leased reconciliation and notification outboxes; +- a versioned cross-Workflow wake envelope; +- source-output range bindings in B's replay annotation and Continue-As-New state; and +- replay-retention leases that outlive active registrations; and +- a default-deny Worker authorizer for cross-Workflow reads. + +Do not copy records from A's `OUTPUT` key to B's `INPUT` key. A relay that performs that copy remains +a valid application-level alternative, but it is not the native feature designed here. + +## Terminology + +| Term | Meaning | +|---|---| +| Source | The Workflow chain whose external `OUTPUT` topic holds the records; A in the examples | +| Consumer | The Workflow chain whose code subscribes to the source; B in the examples | +| Source binding | Source `WorkflowChainKey`, `OUTPUT` direction, topic, provider binding, and initial cursor | +| Registration | Durable provider state saying that one B wait consumes one A output topic | +| Visibility transition | A commit or abort that advances the readable committed prefix of an output topic | +| Notification | A durable obligation to announce a visibility transition to one consumer registration | +| Replay-retention lease | Provider state preventing GC while B's retained History can reference source records | +| Binding ID | A protocol hash of the immutable source binding and consumer wait identity | + +## Goals + +- Let B consume A's committed external output directly from Workflow code. +- Keep every stream payload out of both Workflows' Histories. +- Never expose output from an A Workflow Task that did not commit. +- Preserve B's deterministic replay, including ordering across several local and foreign streams. +- Guarantee that a committed, readable record eventually wakes a parked B while a dispatcher and + Temporal are available, including after Worker restart. +- Preserve source output order, opaque resume cursors, explicit `FINISH`, and Continue-As-New chain + identity. +- Support broadcast fan-out: each B has an independent cursor and replay history. +- Make authorization explicit and replay-safe. +- Keep existing input-only and output-only providers valid. + +## Non-goals + +- Work-sharing or consumer groups. Every subscription receives the complete source stream. +- Cross-namespace consumption in the first version. +- Reading another Workflow's external **input** stream. +- Inferring completion from A closing. Only an ordered output `FINISH` ends iteration. +- Exactly-once execution of B's application side effects. Temporal replay semantics still apply. +- Global order across source topics or across different source Workflows. +- Detecting cycles or deadlocks between Workflows that wait on one another. +- Revoking a source binding after an authorization decision has committed. Revocation is future work. +- Browser access to backend credentials. +- Making cross-Workflow consumption work on a provider that cannot atomically create visibility + notifications. + +## Why the current APIs cannot be composed directly + +Today, `external_stream.topic(name).subscribe()` derives an `INPUT` key from the running Workflow's +own namespace, Workflow ID, and first execution Run ID. It accepts no foreign chain key. +`ExternalOutputStreamClient` can read A's `OUTPUT` key, but it performs backend and Temporal service +I/O and therefore cannot run inside deterministic Workflow code. + +Simply permitting B to name A's key leaves four correctness gaps: + +1. **No wake target.** A's output key identifies A, while a wake must be sent to B. +2. **No durable subscriber registry.** If B is evicted, a backend watch in B's old Worker no longer + exists, and A has no way to discover B. +3. **Wrong replay binding.** B's current input annotation implies B's own `INPUT` key and B's + serialization context, not A's `OUTPUT` key and A's context. +4. **Pending output.** B must not read a staged A batch until A's producing Workflow Task has a + durable marker proving commit. + +Direction must remain part of physical identity. Treating A's output and B's input as the same key +would violate provider isolation, mix record formats and coordination metadata, and route wakeups to +the wrong chain. + +## Public API + +The proposed names are intentionally explicit. The source reference is data and may be passed in a +Workflow input, Signal, Update, result, or Activity argument. + +```python +from temporalio.contrib.external_workflow_streams import ( + BEGINNING, + WorkflowChainKey, + WorkflowOutputStreamReference, + external_stream, +) + +source = WorkflowOutputStreamReference( + workflow=WorkflowChainKey( + namespace="payments", + workflow_id="agent-A", + first_execution_run_id="source-chain-run-id", + ), + topic="events", +) + +events = external_stream.from_workflow_output( + source, + type=AgentEvent, + after=BEGINNING, +) + +async for event in events.subscribe(): + process(event) +``` + +Proposed types: + +```python +@dataclass(frozen=True) +class WorkflowOutputStreamReference: + workflow: WorkflowChainKey + topic: str + +class ExternalWorkflowOutputTopic(Generic[T]): + source: WorkflowOutputStreamReference + value_type: type[T] | None + after: Cursor + + def subscribe(self) -> ExternalWorkflowOutputSubscription[T]: ... +``` + +Rules: + +- `WorkflowChainKey` is mandatory. Resolving a Workflow ID to whichever chain happens to own it at + subscription time is not supported in v1; it races with Workflow ID reuse. +- The source namespace must equal B's namespace in v1. +- `after` accepts `BEGINNING`, `AFTER(offset)`, or an `Offset` with the same meaning as + `ExternalOutputStreamClient.subscribe`. There is no implicit `TAIL` option in v1. A caller that + needs tail semantics obtains a committed tail outside Workflow code and passes the cursor in. +- On B Continue-As-New, the restored durable cursor overrides the first-Run starting cursor. The + recreated API call must name the same source and topic or replay fails as nondeterministic. +- The handle yields decoded values, not backend offsets. Cursor ownership remains inside the + Workflow runtime after the initial boundary, matching existing Workflow input subscriptions. +- A source must exist and its first execution Run ID must verify before the first registration. +- `FINISH` ends iteration. Source completion, failure, cancellation, or termination does not + synthesize `FINISH`. + +An external convenience function may resolve and verify a source reference from a client handle, +but the Workflow-side API always receives the exact resolved reference. + +## Stream identity and serialization + +The physical source key remains: + +```text +(source namespace, + source Workflow ID, + source first execution Run ID, + OUTPUT, + source topic) +``` + +B's identity is not added to that key. It belongs in the registration, because adding it to the +stream key would create one physical copy per consumer rather than broadcast one source log. + +Output payloads were encoded using A's Workflow serialization context. B therefore decodes with its +configured `DataConverter` bound to A's namespace and Workflow ID, not B's. The provider ID, provider +format version, source chain, topic, and source serialization context identity are recorded in B's +first binding annotation. A codec or key configuration that cannot decode A's immutable bytes raises +`StreamDecodeError`, not an integrity error. + +The source and consumer Workers must address the same logical provider data and agree on provider ID +and format version. They do not need to share a provider connection object or credentials. + +The provider format advertises whether visibility notification is atomic. Registering a foreign +consumer against a source topic without that capability is rejected. All Workflow and direct +publishers writing a workflow-consumable provider prefix must use the capability-aware format; +provider-side version metadata rejects an older publisher instead of accepting a write that cannot +notify existing registrations. This is a deployment compatibility boundary, not a best-effort check. + +## Authorization + +Cross-Workflow reads are default-deny. Enabling the provider capability alone does not grant every +Workflow on a Worker access to every output stream visible to its backend credentials. + +The Worker accepts an out-of-sandbox authorizer: + +```python +Worker( + ..., + external_stream_backend=backend, + external_stream_authorizer=authorizer, +) +``` + +The authorizer receives the source key, consumer chain, current consumer Run, topic, requested start +cursor, and operation `SUBSCRIBE_WORKFLOW_OUTPUT`. It returns `ALLOW(decision_id)` or `DENY(reason, +decision_id)`. The decision ID is non-secret and stable enough for diagnostics; credentials and +policy documents never enter History. + +Authorization is an external observation and therefore part of deterministic state: + +- an allowed binding and its decision ID are recorded in B's first stream marker; +- a denial caught by Workflow code is recorded as a denied binding outcome if that Workflow Task + commits; and +- replay uses the recorded outcome and never calls the authorizer. + +An uncommitted Workflow Task attempt may be authorized again on retry, just as it may observe newer +external records; only the accepted attempt becomes replay history. Once an allowed binding commits, +it remains authorized for that Workflow chain. Operators requiring immediate revocation must stop B +or deny access in the backend as an operational intervention; a deterministic revocation protocol is +outside v1. + +## Provider capability + +Add an optional capability independent of `StreamBackend` and `OutputStreamBackend`: + +```python +class WorkflowOutputSubscriptionBackend(OutputStreamBackend): + ... +``` + +An output-only provider remains valid for external clients and direct producers. A provider must +implement this additional capability only when a Workflow calls `from_workflow_output`. + +### Durable registration + +The registration key is injectively encoded from: + +```text +(source OUTPUT StreamKey, + consumer namespace, + consumer Workflow ID, + consumer first execution Run ID, + consumer wait ID) +``` + +Its value contains at least: + +- current consumer Run ID; +- binding ID and authorization decision ID; +- consumer cursor used for recheck; +- current quiescence/park generation, where zero means unparked; +- monotonically increasing registration version; and +- creation/update metadata for diagnostics and garbage collection. + +Unlike the existing input park intent, this registration exists while the subscription is active, +not only while B is parked. That persistent identity is necessary because A's output key cannot imply +which workflows should receive an unparked wake. + +Continue-As-New updates the value to B's successor Run while retaining the same consumer chain and +logical wait binding. Cancellation or subscription close conditionally removes the registration by +consumer Run ID and registration version so stale cleanup cannot delete a successor's registration. +An unconfirmed removal is an owed cleanup operation and is retried. + +### Required operations + +Names are illustrative; their transactional properties are normative. + +- `register_output_subscription(registration, after)` atomically installs or updates the + registration and rechecks the readable committed prefix after `after`. +- `update_output_subscription(registration_version, cursor, park_generation)` conditionally updates + the current Run, cursor, and generation. +- `remove_output_subscription_if_matches(...)` returns removed, absent, or mismatch. +- `read_committed_output_range(key, first, last)` returns exactly the committed, readable records in + an inclusive provider range and reports any unresolved barrier at or before `last`. +- `make_output_stage_visible(manifest, decision)` atomically commits or aborts a stage, advances the + readable prefix, and creates/coalesces notifications for every active registration whose readable + view advanced. +- `stage_output_for_subscribers(manifest, records)` preserves existing pending-stage semantics and + creates/coalesces a durable stage-reconciliation item when the source currently has registrations. +- `append_committed_output_and_notify(key, record)` atomically performs a direct output append and + creates/coalesces the corresponding notifications. +- `claim_stage_reconciliations` and its renew/ack/release operations implement durable recovery of + pending heads. A registration installed behind an existing pending head creates the same work. +- `claim_notifications`, `renew_notification_claim`, `ack_notification_if_current`, and + `release_notification_claim` implement the leased wake outbox. +- `advance_replay_retention` monotonically widens a consumer binding's referenced output range, and + `close_replay_retention` sets its safe release boundary without deleting it early. + +Existing `commit_output`, `abort_output`, and `append_output` may delegate to these operations when +the capability is present. Providers must not advertise cross-Workflow subscriptions if output +visibility and notification creation can be separated by a crash. + +### Atomic registration/recheck race + +Registration and visibility use one provider serialization point: + +```text +commit before registration -> registration recheck returns the new committed records +registration before commit -> commit creates a durable notification +``` + +There is no third interleaving in which the records become visible but neither path observes them. + +### Commit and abort both notify + +A notification is tied to advancement of the **readable prefix**, not merely to appending a record. +This distinction is required for output barriers: + +```text +offset 10 PENDING stage S +offset 11 committed direct output, hidden behind S +``` + +Committing S makes offsets 10 and 11 readable. Aborting S also makes offset 11 readable. Both +terminal decisions must create notifications. A design that notifies only on append or commit loses +the abort case permanently. + +## Reconciliation and notification outboxes + +There are two kinds of durable coordination work: + +- **Stage reconciliation** proves whether an unresolved pending A stage committed or aborted. It is + keyed by the exact stage manifest and coalesced across all consumers of that source topic. +- **Visibility notification** announces an already readable prefix to one B registration. It carries + no stage decision and no payload. + +Creating a pending stage while registrations exist creates reconciliation work. Registering behind +an already pending head does the same. This closes the case in which A's Worker dies after its +Workflow Task commits, B's Worker also restarts, and no cached process remains to perform the lazy +client reconciliation that current external output clients rely on. + +Dispatchers process reconciliation before the affected visibility notification: they read A's exact +deciding History, apply commit or abort through `make_output_stage_visible`, and let that atomic +transition create the notification obligations. An undecided stage is retried with backoff. A stage +with no registrations may continue to use today's source-Worker and external-client lazy +reconciliation; a later registration makes its reconciliation durable. + +### Notification wake dispatch + +Each notification identifies the source binding, consumer registration, latest newly visible tail, +registration version/generation, and notification revision against which it was created. The +provider may coalesce several visibility transitions for the same registration by increasing that +revision and moving the tail forward; payload records are never copied into the outbox. + +Every Worker configured with a capable backend may run a lightweight global dispatcher. Dispatchers +do not require B to be cached and do not reconstruct B's Workflow state. They: + +1. lease-claim a notification; +2. read the current registration; +3. build a raw wake request addressed to B; +4. send the Signal idempotently; +5. conditionally acknowledge the notification only if the registration version, generation, and + claimed notification revision still match; and +6. release/recompose when B changed generation during the send. + +Claims are renewable and expiring. Losing a claim is not proof that another dispatcher sent the +Signal, so racing dispatchers may send. Stable request IDs make that safe. + +The global scan is essential. A per-run backend watch is a useful low-latency fast path while B is +cached, but it disappears on Worker restart. Likewise, relying only on A's Worker to reconcile and +wake fails if A completes and that Worker never returns. Durable reconciliation and notification +outboxes plus any live dispatcher remove both dependencies. + +### Wake envelope v2 + +Use the existing reserved Signal name with a new envelope version. It still bypasses the user's +`DataConverter` and contains no stream payload. + +Conceptually: + +```protobuf +message WakeSignalV2 { + uint32 envelope_version = 1; // 2 + string consumer_first_execution_run_id = 2; + uint32 wait_id = 3; + uint64 park_generation = 4; // 0 means unparked + bytes binding_id = 5; // source binding + consumer wait + string notification_id = 6; // diagnostics/deduplication + string dispatcher_identity = 7; // diagnostics only +} +``` + +Core stores the binding ID beside the registered wait. It accepts a wake only when the target +consumer chain, wait ID, binding ID, and non-zero generation match. Generation zero requests a +recheck of B's complete active wait set. Core suppresses every version of the reserved Signal from +user Signal handlers, including malformed or stale envelopes. + +Request IDs are derived as follows: + +- parked: `(consumer chain, binding ID, wait ID, park generation)` so every notification for the + same parked generation collapses to one server wake; +- unparked: the same fields plus the durable notification ID and claimed revision, so later + visibility transitions do not reuse a request ID that the server already accepted. + +After Signal acknowledgement, the dispatcher conditionally acknowledges the outbox item. If B +changed its registration version or park generation, or if newly visible output advanced the +notification after it was claimed, the item remains pending and is recomposed. This prevents a stale +but successfully delivered Signal from consuming the only wake owed to B's new park or to output +that became visible after the signaled Workflow Task had already rechecked. + +Closed or reused target chains are terminal for that registration. After the Temporal service proves +the target closed or the first execution Run ID mismatched, the dispatcher conditionally removes the +stale registration and acknowledges its notifications. Transient service failures remain retryable. + +## Live consumption + +When B creates the subscription, its out-of-sandbox runtime: + +1. validates the full source reference and same-namespace rule; +2. verifies A's chain against Temporal; +3. verifies provider capability and provider binding; +4. obtains an authorization decision; +5. binds B's decoder to A's serialization context; +6. atomically registers and rechecks from the requested cursor; and +7. returns readiness to B through the existing external-stream activation path. + +For later records, either B's local backend watch reports readiness directly to Core or an outbox +dispatcher Signals B. A valid Signal causes `ResolveExternalStreamWaits`; Python rechecks all active +waits, including local input and foreign output subscriptions. + +A foreign output wait participates in the same global quiescent set as existing input waits. Mixed +waits therefore retain one Workflow Task, share one idle-timeout reduction, and use one terminal race. +Parking conditionally writes the current generation to every foreign registration before Core +confirms the park. Readiness concurrent with parking is resolved by the provider's atomic +registration/recheck transition and the existing Core park handshake. + +Slow B consumers never block A or external clients. They increase retained provider data and their +own replay ranges, but source publishing does not wait for their application code. + +## Output staging and pending barriers + +B reads only the committed readable prefix. An unresolved A stage remains a hard ordering barrier; +B may consume committed records before it but never at or after it. + +The consumer runtime may reconcile a pending head exactly as `ExternalOutputStreamClient` does: + +- read the exact producing A Run's History strictly above the manifest's recorded floor; +- commit on the exact output stage token in A's marker; +- abort on the first durable Workflow Task closing boundary that proves the token absent; and +- leave the stage pending if History proves neither result. + +Applying either decision uses `make_output_stage_visible`, so any newly readable prefix creates +notifications atomically. History unavailability needed for a decision is integrity loss; a backend +or Temporal outage is a transient storage failure. + +The durable reconciliation outbox applies the same algorithm when no source or consumer Run is +cached. A dispatcher acknowledges that work only after the exact provider stage reaches the proven +terminal status. Losing the dispatcher before acknowledgement repeats an idempotent decision. + +B never observes speculative output. A Workflow Task attempt in A that stages records and then fails +can only be aborted, after which those records remain invisible and are skipped. + +## Replay annotation + +B's marker adds a foreign-output binding table. Each binding contains: + +- wait ID and binding ID; +- complete source `OUTPUT` `StreamKey`; +- provider ID and format version; +- initial cursor for the first Run of B's chain; +- source serialization context identity; +- authorization outcome and decision ID; and +- finished state when `FINISH` has been consumed. + +The existing activation-segment schedule is reused. A segment run refers to a wait ID and an exact +inclusive output range. For foreign output, each run additionally records the committed record count +needed to validate ranges that contain provider offsets belonging to aborted stages. + +On replay: + +1. Core issues the recorded replay job and segment schedule. +2. Python reconstructs the binding from the marker, not from mutable Workflow arguments, Temporal + Describe, the authorizer, or provider tail state. +3. The provider reads each exact inclusive committed range. +4. Python verifies provider binding, monotonic offsets, exact first and last offsets, record count, + absence of an unresolved barrier at or before the last offset, and presence of every referenced + record. +5. Records are decoded under A's recorded serialization context and delivered in the recorded global + segment order. + +Replay never asks what comes after the recorded last offset and never resolves a pending stage. +Immutable output bytes plus exact range validation provide the same integrity boundary as current +input replay. Missing, reordered, unexpectedly pending, or no-longer-committed records raise +`StreamIntegrityError`. + +An output stage cannot change from committed to aborted or vice versa. A barrier that was pending +during live execution cannot later insert records inside a range B already consumed, because B could +not read past that barrier. This monotonicity makes exact committed-range replay well-defined. + +## Continue-As-New + +B's internal continuation header carries, for every unfinished foreign subscription: + +- source reference and binding ID; +- provider binding; +- last durably consumed cursor; +- authorization decision ID; +- source serialization context identity; and +- logical wait identity needed to recreate the subscription. + +The successor Run updates the durable registration's current Run ID conditionally. A predecessor's +late cleanup cannot remove the successor registration. The source stream remains stable across A's +own Continue-As-New because its key uses A's first execution Run ID. + +B's finished subscriptions carry terminal state but no active provider registration. Re-publishing +to a topic after `FINISH` remains an output protocol error; a successor does not reopen it. + +## Cancellation, closure, and cleanup + +Closing or cancelling B's subscription removes the registration conditionally. B Workflow +completion and Continue-As-New finalization include the same cleanup obligation. An unconfirmed +remove is retained as owed manager state and retried; it is never converted into deletion of whatever +new value happens to occupy the key. + +Termination may prevent B from running cleanup. Stale registrations are therefore also removed when +a dispatcher receives authoritative proof that the target consumer chain is closed or reused. Until +then, a stale registration costs only coalesced notification state and harmless failed wake attempts; +it never blocks A's readable prefix. + +A's closure does not remove consumer registrations or end B's iterator. Only `FINISH` does. This +matches current output semantics and avoids making a transient inability to Describe A look like an +ordered terminal record. + +## Retention + +Once B consumes A's output, those records become replay dependencies of B's History. The provider +must retain them for the longer of: + +- A's external-client resume policy; +- A's own pending-stage reconciliation window; and +- every consuming B chain's open lifetime plus applicable Temporal History retention/replay window. + +This can be much longer than the source's original client-resume window. Provider GC must consult +replay-retention leases and terminal/retention metadata or use a configured conservative retention +that satisfies all consumers. Elapsed source age alone is not sufficient. + +An active registration is not itself the retention record. On first delivery, the consumer runtime +creates a replay-retention lease before handing the record to Workflow code, then monotonically +widens its first/last referenced range as delivery advances. Updating before Workflow Task acceptance +may over-retain after a failed attempt, which is safe; updating only afterwards creates a crash window +in which History commits a dependency the provider does not know to keep. + +Closing a subscription, consuming `FINISH`, or completing B removes the active registration but does +not delete its replay-retention lease. Authoritative B closure changes the lease to a terminal state +whose release time covers B's applicable Temporal retention, archival, and replay policy. A +Continue-As-New successor keeps the same consumer-chain lease active. The provider may trim a source +range only when no active or terminal lease can still reference it. + +Missing retained records are integrity failures, not an instruction to restart at the current tail. +The SDK chooses no default retention duration. + +## Delivery and ordering semantics + +- Each subscription is broadcast and has an independent cursor. +- Records are yielded in provider order within one source topic. +- `FINISH` is ordered and terminal. +- Ordering across topics or source Workflows is only the order B actually observes. The existing + activation-segment schedule records that order for replay. +- Direct Activity output and Workflow-originated output share the source topic's existing barrier and + committed-prefix rules. +- Replaying B re-executes its code against the same logical records. This is deterministic replay, + not an exactly-once guarantee for external side effects performed by application code. + +## Failure model + +| Failure | Required behavior | +|---|---| +| A stages output but its Workflow Task is rejected | Stage aborts; B never observes its records | +| A's task commits and A's Worker dies before provider commit | Any reconciler proves the token from A's History, commits atomically with notifications | +| Both Workers restart while A's stage is pending | Global reconciliation work proves the stage, then creates visibility notifications | +| A's stage abort exposes later direct output | Abort atomically advances the readable prefix and notifies B | +| Dispatcher dies after claiming | Lease expires; another dispatcher retries | +| Dispatcher dies after Signal commit but before outbox ack | Retry uses the same request ID; Temporal deduplicates it | +| B changes park generation while a Signal is in flight | Conditional outbox ack fails; notification is recomposed for current registration state | +| B Worker restarts while B is parked | Global dispatcher scans durable outbox and Signals B without requiring cached Run state | +| Registration races source visibility | Atomic register/recheck or visibility notification observes the transition | +| Backend is unavailable during live read | Transient `StreamStorageError`; Workflow Task retries | +| Authorization service is unavailable before binding commits | Workflow Task fails transiently and retries | +| Authorization is denied and Workflow catches it | Denied outcome is recorded and replayed without reauthorizing | +| Source chain key is wrong or reused | Binding fails; no registration is installed | +| Output bytes cannot be decoded with A's context | `StreamDecodeError` | +| A referenced replay range was trimmed or altered | `StreamIntegrityError` | +| B closes without consuming `FINISH` | Registration is removed; source remains unchanged | + +## Backpressure, quotas, and fan-out + +Payload data is not duplicated per B, but coordination is. A source with `N` consumers creates up to +`N` coalesced notification obligations per visibility transition. Providers must support: + +- a maximum active registration count per source topic and per consumer Workflow; +- notification coalescing by registration and park generation; +- bounded dispatcher batches and leased claims; +- retry backoff with jitter for transient Temporal failures; +- metrics for oldest pending notification age; and +- operator inspection of registrations and poison notifications. + +Exceeding a registration quota fails the new subscription before it becomes visible to Workflow +code. It never silently drops an existing consumer. A slow consumer does not apply backpressure to A; +retention quotas and application policy decide whether to stop that consumer. + +## Observability + +At minimum, emit: + +- active foreign output registrations; +- registration attempts, denials, and conditional-cleanup mismatches; +- notifications created, coalesced, claimed, acknowledged, retried, and oldest age; +- pending-stage reconciliation items created, resolved, retried, and oldest age; +- parked versus unparked wake attempts and Signal latency; +- stale/closed target cleanup; +- local-watch versus Signal readiness; +- pending-stage reconciliations performed on behalf of Workflow consumers; +- foreign output records and logical bytes delivered; +- replay range reads and integrity/decode failures; and +- active and terminal replay-retention leases plus oldest protected source offset. + +Logs and metrics identify source and consumer using safe hashed binding IDs by default. Raw Workflow +IDs and topic names follow the SDK's existing telemetry privacy policy. + +## Core and language protocol changes + +Core does not read provider data. It needs enough information to preserve Workflow Task lifecycle and +validate wakes: + +- a binding variant distinguishing local `INPUT` waits from foreign `OUTPUT` waits; +- binding ID and source/provider metadata in progress markers and continuation state; +- record count on foreign-output replay ranges; +- wake envelope v2 parsing and binding-ID validation; +- the existing resolve/replay jobs generalized to carry the binding kind; and +- the existing wait-set, activation budget, parking, rollover, shutdown, and WFT-admission invariants + applied uniformly to both kinds. + +Unknown binding kinds, provider formats, annotation schema versions, and wake envelope versions are +must-understand failures or harmlessly suppressed wakeups according to the existing protocol boundary; +they never fall through to user code. + +## Compatibility and rollout + +This is additive and remains experimental with the broader feature. + +1. Specify the annotation, continuation, wake envelope, registration, and outbox formats. +2. Add provider conformance tests and an optional backend capability without changing current + provider registration requirements. +3. Add Core protocol support behind a Worker feature flag. +4. Add Python runtime support with the authorizer defaulting to deny. +5. Implement Redis registration/outbox scripts with atomic visibility notification. +6. Enable the public API only when Core and the configured provider both advertise support. +7. Remove the feature flag only after crash, replay, Continue-As-New, retention, and multi-Worker + handoff tests pass. + +An older Worker that encounters a marker containing a foreign-output binding must fail clearly as an +unsupported annotation version. It must not replay the Workflow while ignoring that binding. +Likewise, a capability-aware provider prefix rejects output mutations from SDKs that predate atomic +notification creation. Mixed source publisher versions must fail closed rather than create readable +records with no durable wake obligation. + +## Required tests + +### API and binding + +- B can consume A output from `BEGINNING` and from an explicit opaque cursor. +- Missing, mismatched, reused, and cross-namespace source references are rejected before registration. +- A source and B input topic with the same name remain physically isolated. +- Source DataConverter context is used; consumer context is not substituted. +- Default deny, allowed authorization, caught denial replay, and authorization outage are distinct. + +### Visibility and races + +- B never observes output from a rejected A Workflow Task. +- Committing a stage notifies every existing B registration. +- Aborting a pending stage exposes and notifies later committed direct output. +- A pending stage is reconciled after both source and consumer Workers restart with neither Run + cached. +- Registration-before-commit and commit-before-registration both deliver without polling luck. +- A registration concurrent with `FINISH` sees the terminal exactly once. +- Several commits coalesce without losing the latest readable tail. + +### Wake durability + +- A, B, and dispatcher crashes are injected before and after provider mutation, Signal send, Signal + acknowledgement, and outbox acknowledgement. +- A parked B is woken after both Workers restart and neither Run was cached. +- A generation change during dispatch prevents stale acknowledgement and causes recomposition. +- Racing dispatchers send the same parked request ID. +- Later unparked visibility transitions use distinct request IDs. +- A closed or reused B chain is removed without deleting a successor registration. + +### Replay and lifecycle + +- B replay reads exact A output ranges and never the current tail. +- Missing first, middle, and last records; wrong counts; reordered offsets; and an unexpected pending + barrier fail as integrity errors. +- Mixed local input, own output, and foreign output in one retained Workflow Task reproduce one + activation-segment schedule. +- B Continue-As-New restores the source binding and cursor; a late predecessor cleanup preserves the + successor registration. +- A Continue-As-New preserves the source stream; A Workflow ID reuse does not. +- B cancellation and completion clean up registrations; B termination is eventually cleaned by + dispatcher target validation. +- Registration cleanup preserves replay-retention leases until B's replay window ends. +- A crash before Workflow Task acknowledgement may over-retain but cannot under-retain a delivered + range. +- `FINISH` ends iteration, while source Workflow closure without `FINISH` does not. + +### Scale and conformance + +- Many B workflows receive one source topic independently. +- One slow B neither blocks A nor another B. +- Registration quotas fail new subscribers without dropping current ones. +- Every advertised provider passes atomic register/recheck, commit-notify, abort-notify, both outbox + lease protocols, conditional revision acknowledgement, replay-retention, exact committed range, + and injective key tests. + +## Alternatives considered + +### External relay from A output to B input + +This works with today's APIs and remains the operational fallback. It duplicates every payload, +requires another service with cursor/idempotency state, doubles provider retention, and introduces a +second stream whose replay relationship to A must be managed by the application. It is simpler for +the SDK but does not provide native zero-copy Workflow-to-Workflow streaming. + +### Let B call `ExternalOutputStreamClient` + +Rejected. Backend reads, sleeps, and Temporal History requests inside Workflow code are +nondeterministic and bypass Workflow Task replay markers. + +### Alias A output as B input + +Rejected. It violates direction isolation, uses incompatible record and staging contracts, decodes +under the wrong Workflow context, and still does not solve the wake target. + +### Long-running Activity in B + +An Activity can read A and return batches, but Activity results place payloads in B's History unless +it republishes them to B's input stream, which reduces to the external relay. Activity cancellation, +heartbeat, and retry state also become part of the streaming protocol. + +### Temporal Timer polling + +A durable polling timer eventually rechecks after restarts but adds recurring History events and +fixed latency while idle. It may be offered later as an optional safety net, not as the primary +wakeup protocol. + +### Provider pub/sub without a durable outbox + +Rejected. Pub/sub is a useful local fast path, but a notification published while B's Worker is down +is lost. Correctness cannot depend on a process being continuously connected. + +## Correctness invariants + +Implementation must preserve all of the following: + +1. **Committed-only visibility:** B receives no record from an uncommitted A Workflow Task. +2. **No lost readiness:** every readable-prefix advancement is observed by atomic registration + recheck or creates a durable notification for every active registration; pending output that can + block such advancement has durable reconciliation work. +3. **Wake durability:** a notification is not acknowledged until an idempotent Signal is accepted for + the registration state it names. +4. **Exact replay:** B re-reads only ranges named by its marker and reproduces their global delivery + order. +5. **Binding isolation:** source chain, direction, topic, provider, consumer chain, and wait ID cannot + collide or be substituted. +6. **Source serialization:** live and replay decoding both use A's recorded serialization context. +7. **Monotonic visibility:** pending output is a barrier; commit and abort are irreversible and may + only advance the readable prefix. +8. **Source independence:** a slow, failed, or closed B never blocks A's output commit or external + clients. +9. **Successor safety:** stale cleanup or wakes from a predecessor Run cannot delete or satisfy a + successor's registration. +10. **Replay retention:** active-registration cleanup cannot release any record while retained B + History can still reference it. +11. **Payload-free Temporal transport:** markers and wake Signals contain identities, cursors, + counts, and proofs, never stream payload bytes. + +These invariants, rather than the illustrative class and method names, define the feature. diff --git a/arch_docs/streaming-poc-docs/required-tests/tests-m1.md b/arch_docs/streaming-poc-docs/required-tests/tests-m1.md new file mode 100644 index 000000000..deae515f5 --- /dev/null +++ b/arch_docs/streaming-poc-docs/required-tests/tests-m1.md @@ -0,0 +1,302 @@ +# Milestone 1 required tests — 102 cases + +One stream, end to end. Milestone 2's 17 cases are in `tests-m2.md`; the two partition the +119 required cases exactly. + +This list is read at test time, not by a human: `tests/contrib/external_workflow_streams/ +m1_gate.py` parses the count in the heading and every bullet below it, and maps each case +to the test that covers it. A case added here with no mapping fails that gate rather than +passing unnoticed. + +Every bullet is one case. If you add or remove one, update the count in this heading; the +gate checks that the two agree. + +## Progress, observation, and marker transport + +- A consumed record followed by a normal completion commits its marker. +- A consumed record followed by an Activity command commits its marker, ordered before the + Activity command, and replay delivers the record exactly once. +- A consumed record followed by a terminal command (complete / fail / continue-as-new) commits + its marker before the terminal command. +- A retained Workflow Task spanning several progress reports produces exactly one marker. +- Every input-direction completion path in the finalization-ownership table produces exactly one marker whose + annotation ends with a terminal — normal, command-producing, terminal command, park + confirmation, all-fenced immediate park, deadline rollover, budget rollover, and shutdown with + a Workflow Task open — while an aborted park and a shutdown with no open Workflow Task produce + none. This is the cross-path assertion; the per-path integrations are separately owned, so + only this gate can make it. +- A subscription to an empty stream that never receives a record still emits an observation + delta, and its marker carries provider identity, stream key, and explicit start cursor. +- An activation that drains and observes nothing encodes an empty segment, and the codec + round-trips an annotation with zero runs in a segment and with zero segments overall. +- Annotation exceeding the byte-budget high-water mark forces rollover instead of growing the + marker, and replay reassembles the two markers in Workflow Task order. +- A single-stream batch of many records encodes as one run. +- Encoded marker bytes are asserted directly, not inferred from run count: a single-stream batch + with sparse control records stays flat in bytes as record count grows by three orders of + magnitude. + +## Retention, rollover, and timers + +- Readiness wins immediately before idle timeout. +- Idle timeout wins immediately before readiness. +- Backend recheck aborts complete-set parking and removes every park intent. +- Append after confirmed park produces a Signal wakeup. +- Duplicate/stale readiness and Signals are harmless. +- With one active wait, its write fence permits immediate parking without closing the stream. +- WFT rollover preserves every active subscription, cursor, observation delta, and readiness + generation. +- A rollover deadline that expires with no Python activation outstanding still produces a + complete marker: Core issues `FinalizeExternalStreams`, Python returns the terminal, and only + then is the marker written. +- A continuously fed stream with inter-record gaps below the idle timeout survives at least one + rollover and never exceeds the Workflow Task timeout. +- Rollover fires on a Worker registering Workflows and **no** Activities, i.e. with + `enable_local_activities = false`. +- A Signal sent while a Workflow Task is retained is delivered no later than the rollover + deadline. + +## Wakeup after unparked completions + +- An append after a completion that carried server-bound commands wakes the subscription: the + watcher observes `NoOpenWorkflowTask` and sends the wake Signal. +- The same, after a rollover completion. +- The three non-local acknowledgements are distinct: a confirmed park returns `Parked`, a cached + Run between Workflow Tasks returns `NoOpenWorkflowTask`, and an evicted Run returns + `RunNotFound` — with the corresponding watcher teardown behavior for each. +- A wake Signal with an unknown envelope version, a stale generation, or a foreign + `first_execution_run_id` is suppressed from user handlers and does not resume the Workflow. +- Two producers retrying the same wake produce the identical Temporal request ID and the server + deduplicates them. +- An unparked wake carries `park_generation = 0` and is accepted as a recheck, while a non-zero + generation the Run does not recognize is ignored as stale; two Workers shutting down at + different times derive **different** request IDs and both wakes are delivered. + +## Shutdown, eviction, and finalization + +- Shutdown with a Workflow Task open issues `FinalizeExternalStreams{SHUTDOWN}`, writes a marker + carrying the returned terminal, and completes with `force_new_wft = true`; a second Worker + receives the replacement task and reconstructs the subscription from that marker. +- Shutdown in the `NoOpenWorkflowTask` window writes **no** marker, resolves the Run's state + through the read-only status probe rather than a false readiness notification, sends an + acknowledged unparked wake Signal, and a second Worker receives a Workflow Task and + reconstructs the subscription — without waiting for an unrelated Workflow event and without + completing a nonexistent Workflow Task. +- Teardown racing finalization never commits an annotation without its terminal: the Python + manager's Run state survives until `RemoveFromCache`, and when finalization is forced to fail — + the manager's Run entry removed underneath it — Core writes no marker, the Workflow Task fails, + and the retry replays from the previous marker with no record lost and no cursor moved. +- Eviction with no open Workflow Task writes no marker at all, and the invariant that an + unwritten annotation exists only while a Workflow Task is open holds across both eviction + states. +- An unacknowledged shutdown wake follows the documented policy: retried within the grace period + under the same request ID, then reported through the + `temporal_external_stream_shutdown_wake_failed` metric without blocking shutdown and without being reported as delivered. +- With no buffered output, `FinalizeExternalStreams` calls no input-provider method: with a provider + that raises on every input call, finalization still produces a marker with a terminal, and provider + code runs on neither the Workflow thread nor the terminal-encoding path. + +## Replay + +- Replay performs no live waiting and reproduces recorded boundaries. +- Replay reproduces activation segmentation: a `wait_condition` registered mid-stream fires on + the same delivery under replay as it did live, for a marker spanning several activations. +- A Workflow that subscribes to an empty stream, parks, is evicted, and replays reproduces the + same empty boundary and starts from the marker-recorded cursor without reading live backend + state. +- Deleted or trimmed replay records fail as `StreamIntegrityError`; a backend that is merely + unreachable during replay fails the Workflow Task with the transient error type instead. +- Deleting the first, the middle, and the last record of a recorded range each fails as integrity + loss rather than substituting a later record — the middle case is the one a start-plus-count + encoding would have passed. +- A record whose bytes are intact but whose DataConverter configuration does not match fails as + `StreamDecodeError`, with a different metric from `StreamIntegrityError`. +- An annotation that does not match the subscriptions Workflow code creates fails as + nondeterminism, not as a storage failure. +- Worker crash before marker commit re-reads the same offsets. +- The first Run of a chain records an explicit start cursor rather than resolving one from live + backend state. +- A pending timer or activity in the same activation suppresses retention; subscriptions survive + to the next Workflow Task. + +## Provider conformance and the Python runtime + +- A provider whose range read is implemented with exclusive (`XREAD`-style) semantics fails the + conformance suite; the inclusive range read returns the record at `first_offset` itself. +- A consumer parked at the current tail resumes correctly after an append whose ID could not have + been predicted — a provider requiring a nameable next ID fails the suite. +- Offsets compare numerically as `(milliseconds, sequence)`, including across a millisecond + boundary where lexical comparison would be wrong. +- Reusing a `(session_id, sequence)` pair with byte-identical content is a no-op returning the + original offset; reusing it with different bytes is rejected as an error. +- A backend read slower than the 2-second activation deadlock timeout delays readiness and does + not fail the Workflow Task. +- A replay range read and a park transaction that each take longer than the deadlock timeout both + complete, without running provider code on the Workflow thread and without a `_DeadlockError`. +- `_apply` performs no I/O: an activation drains only from the buffer, verified by a provider that + raises if called from the Workflow thread. +- A full subscription buffer stops prefetch without dropping records or blocking the Workflow + thread. +- Eviction discards speculative prefetch state: a subscription whose `prefetch_cursor` ran ahead + of `committed_cursor` restarts from `committed_cursor` and re-delivers the same records. +- Run eviction and Worker shutdown tear down every watcher, buffer, and backend connection for the + Run. +- A provider that does not declare structural immutability is rejected at Worker registration, + before any Workflow can name it, rather than failing later at replay. +- A trimmed range (Redis `XTRIM`/`MAXLEN` removing the head of a recorded range) and a deleted + write fence both fail as `StreamIntegrityError`, distinctly from an unreachable backend. + +## Additional protected boundaries + +New cases are appended here rather than filed under the sections they belong to, because the gate +maps cases by position: inserting one renumbers every case after it and silently repoints every +mapping. + +- The first live drain after a replay sees only records past the marker's boundary **with no + event-loop turn in between**: the reposition has to have happened by the time the replay returns, + not merely be scheduled onto the manager's loop. A test that yields before draining passes either + way and proves only that the loop won a race. +- A marker with *k* segments produces exactly *k* condition-checking drains **counted across the + whole activation**, driver plus the trailing `_run_once` the job set always runs, with one empty + segment among the *k*. Counted inside the replay driver alone the number was always right and the + extra drain was invisible. +- The completion carrying the segment that crosses the byte-budget high-water mark is the one that + requests rollover, and it carries a decodable terminal — the flag is read after the segment is + closed, not before. +- A frame larger than the slack a fractional high-water mark leaves ends the activation with + `BUDGET_ROLLOVER` and requests rollover, rather than raising; the annotation still closes with a + terminal. A subscription set whose header and terminal alone cannot fit an empty annotation is + refused at `subscribe()`, leaving no half-registered wait behind. +- Every coordination step after a durable append is inside `publish()`'s acknowledged-wake contract: + a provider failure in the parked-set read, the generation read, or the claim raises the + durable-but-unacknowledged error carrying the offset and an explicit recovery, and recovering the + wake alone leaves exactly one record — and, for `finish_writing()`, exactly one fence. +- A shutdown grace period that expires inside a hanging wake counts **every** subscription it + abandoned, including those the serial loop never reached, and a Run whose status probe cannot + answer is counted rather than passed over; a Run that is parked or has an open Workflow Task is + not counted, or the metric fires on every clean shutdown. +- An external payload store that cannot be reached is row one, not row three, on both delivery + paths: a storage driver whose `retrieve()` raises an ordinary provider exception reaches the + consumer as `StreamStorageError` rather than as a converter mismatch, and the classification rule + leaves it alone rather than relabelling it. +- An activation the annotation byte budget stopped is followed by one that can deliver again: the + same completion requests rollover and carries a terminal, with a frame larger than the slack the + fractional high-water mark leaves but not large enough to reach the mark, so the mark cannot stand + in for the condition. Stopping without asking wedges the Workflow permanently. +- A manager with no run-status probe wired reports no abandoned wakes at shutdown, since it has no + sweep obligation to have failed at; a metric that fires where the mechanism is unconfigured is not + alertable. +- A marker with **no segments**, replayed on a Run whose watcher has already buffered records + published while it was evicted, closes before the activation's own drain: that drain hands nothing + over, the cursor sits at the marker's boundary, and the records arrive on the activation the + watcher's re-read announces rather than twice. +- A record is priced by the largest run the **annotation** has encoded, not the largest in the open + segment: the measurement survives the segment being closed, so the first record of an activation is + not priced at the bare floor and then found unrecordable. +- A run larger than the spill margin is refused as a capacity limit naming the provider's offsets, + non-retryably, rather than surfacing as an internal byte-budget error. +- The `subscribe()` capacity floor covers everything an empty annotation carries -- header, terminal, + one segment frame, and the margin -- so a set that clears header-plus-terminal alone is still + refused rather than failing on its first completion. +- Abandoning a replay leaves replay mode without running the consumed check or the cursor move, so a + failing activation reports its own error rather than a nondeterminism error blaming a `subscribe()` + nobody touched, and commits nothing. +- An append carrying a prefetch epoch a reposition has retracted is refused rather than buffered, so + a watcher read that began before the reposition cannot undo it -- asserted as a contract, because a + single-threaded loop cannot place the reposition between a check and an append with no await + between them. +- A replay activation delivers nothing the annotation does not name, with a poison record already in + the live buffer and pending waits resolved on every drain as a coalesced readiness job resolves + them, and records no run of its own. + +- One activation hands Workflow code at most `MAX_RECORDS_PER_ACTIVATION` records **across + subscriptions consumed independently of each other**, with no `merge()` involved: two never-empty + subscriptions drained by two coroutines that yield after every value receive one budget between + them, both end blocked on a readiness future, and the segment records exactly what was handed over. + A ready list carried across an activation boundary is charged to the activation that inherits it, so + the carry-over plus what is newly delivered is still one budget. +- Two concurrent publishes take their sequence numbers in **invocation** order, not in the order their + payload codecs finish: a retry that reuses the session id, makes the same calls in the same order, + and releases the encodes the other way round is idempotent and leaves two records — and the + cross-topic version, where the swapped key cannot collide and nothing would raise, appends no + duplicate on either topic. +- Cancellation delivered after a durable append leaves `publish()` as the durable-but-unacknowledged + error carrying the offset and exactly one recovery, for each of the four post-append stages, and + performing that recovery leaves one record — one fence, for `finish_writing()`. Cancellation before + the backend is called at all — while the payload is still encoding — is still a cancellation, with + nothing appended, no Signal sent, and nothing left owed. +- A second coroutine blocking on one subscription is refused deterministically, on the single-wait path + and inside `merge()` — which refuses before registering anything — while the consumer already + blocked still receives the next record and is left resolvable by `close()`. Iterating one + subscription again after breaking out of a previous `async for` is still allowed, and resumes. +- A `Stale` readiness report whose retry discovers `RunNotFound` sends the owed wake exactly once + **and** tears the watcher down, with the retry position parameterized, and stops retrying at the + answer that cannot change. + +- An append interrupted **after the backend committed and before it answered** is an unknown outcome + carrying the exact record, not a failure and not a bare cancellation: settling it with + `resolve_append()` recovers the original offset and leaves one record — one fence, for + `finish_writing()` — and the same holds when a `ConnectionError` replaces the cancellation. While it + is unsettled the stream refuses the `publish()` that would duplicate it, and accepts one again once + it is settled. Settling an append that never landed appends it once, under the sequence the + interrupted call drew. `AppendConflictError` stays a refusal, since re-appending cannot change it. + +- The unknown-outcome recovery is bound to the operation, the stream and the producer instance. A + refusal issued while an append is unsettled reports **that** append's wake, lease and cancellation + rather than the refused call's, and recovering by the refusal's own fields sends the one wake it + owed. `resolve_append()` refuses a record whose unsettled append is on another topic, refuses + different bytes under the outstanding key without clearing it, and refuses a replacement producer + that shares the session id — whose own recovery, re-running the same calls in the same order, + deduplicates the appends and leaves its sequence and unparked-wake request IDs correct. A recovery + given no wake policy uses the interrupted call's, so a `wake=False` fence gains no Signal. + +- If `resolve_append()` itself commits and loses its response, the retained operation and the newly + raised error remain one state: deliberate wake and lease overrides become the next recovery's + defaults, while cancellation delivered on any attempt remains set. Resolving from the later error + sends the owed wake and leaves one record. + +### Workflow-originated output promotion cases + +- No Workflow-originated output record becomes visible before the producing Workflow Task's marker + commits. +- An explicitly failed or rejected Workflow Task leaves no visible output. +- A discarded speculative Workflow Task stages one token and its redelivery stages another when + event IDs are reused, for both identical and changed logical manifests; History commits only the + accepted attempt's token. +- A pending speculative token aborts at the first durable Workflow Task closing boundary strictly + above its exact History floor, without requiring speculative Scheduled or Started events. +- The producing task's History floor is the event immediately before its Scheduled event and + excludes the preceding Workflow Task close; moving the floor below that close demonstrates the + false abort the exact-floor rule prevents. +- A disconnected speculative Update with no later durable event remains pending regardless of + elapsed time, and retention loss is eventually classified as integrity failure. +- A crash after staging but before server completion resolves to abort or retry without exposing a + phantom record. +- A cold `ExternalOutputStreamClient` repairs a committed pending batch after server acceptance and + Worker loss, including after the Workflow has completed. +- Repeated output stage, commit, abort, and History reconciliation calls are idempotent. +- Reusing one output stage identity with a different logical fingerprint is rejected atomically. +- Output replay validates recorded commands and segmentation without backend writes, token minting, + live capacity splits, or wall-clock flushes. +- Replay uses the marker's output segmentation when a codec's encoded size differs from live + execution. +- A Workflow publish made during a retained Workflow Task becomes visible within the configured + output-flush quantum under healthy dependencies. +- Several publishes inside one output-flush window produce one marker, while crossing *n* windows + produces *n* markers and the corresponding Workflow Task lifecycle cost. +- A pending Workflow output batch blocks later directly committed Activity output from overtaking + it on the same topic. +- Activity output retry produces one visible record for each `(session_id, sequence)` identity. +- A randomized payload codec preserves logical stage identity and the first successfully staged + encoded bytes across a retry. +- Two logically identical Payloads whose metadata maps were inserted in opposite orders have the + same versioned canonical frame, logical byte count, and SHA-256 fingerprint. +- Concurrent message Updates scan from their own output boundaries to their own returned `turn_id` + and cannot adopt one another's `turn_started` event. +- An output-stage outage blocks Workflow Task progress and is classified as transient external + storage failure. +- A missing staged record or expired deciding History produces integrity failure. +- Output marker and activation batch budgets are hard bounds, including an oversized topic name + and a single oversized logical record. +- The public `external_stream` options default `idle_timeout` to exactly one second, and deriving + options with another timeout does not mutate that default. diff --git a/arch_docs/streaming-poc-docs/required-tests/tests-m2.md b/arch_docs/streaming-poc-docs/required-tests/tests-m2.md new file mode 100644 index 000000000..da761da1b --- /dev/null +++ b/arch_docs/streaming-poc-docs/required-tests/tests-m2.md @@ -0,0 +1,34 @@ +# Milestone 2 required tests — 17 cases + +Multiple streams, `merge`, same-stream subscriptions, and Continue-As-New. Together with +`tests-m1.md`'s 102 cases it partitions the 119 required cases exactly. + +Every bullet is one case. If you add or remove one, update the count in this heading; the +gate checks that the two agree. + +- Readiness on any one of several streams resets global quiescence. +- One idle stream cannot park the WFT while another stream is active. +- A write fence on only one stream cannot bypass the global idle timer. +- All fenced streams can request immediate complete-set parking. +- An alternating two-stream batch encodes one run per delivery and triggers budget rollover + rather than exceeding the budget. +- Simultaneously ready streams are coalesced into one activation when observable together. +- Multiple subscriptions preserve the recorded global delivery ordering. +- Two subscriptions to the same stream name each receive every record from their own cursor + (broadcast), and their cursors commit and restore independently across Continue-As-New. +- Two same-stream subscriptions park, wake, cancel, and Continue-As-New without overwriting each + other's park intent or cursor, verified against the backend by inspecting both intents. +- Subscriptions configured with different idle timeouts reduce to `min` deterministically, and + the same reduction is observed on replay. +- A wake Signal for one parked stream causes all streams to be rechecked. +- Continue-As-New restores its initial cursor from History. +- A multi-activation retained Workflow Task carrying both input and output annotations reproduces + exactly the live drain count, rather than summing two replay drivers. +- Output cursor resume has neither gaps nor duplicates across Workflow Task rollover and + Continue-As-New. +- Input and output topics with the same user-visible name remain physically isolated. +- Output-flush and input-park races in both orders produce exactly one Workflow Task terminal and + one output marker: an output winner invalidates the quiescence generation, rolls back every + installed intent, and forces a replacement task; a park winner stages output before completing. +- A finished output topic survives Continue-As-New in the reserved must-understand header and + rejects a successor publish before any backend read. diff --git a/arch_docs/streaming-poc-docs/spec/README.md b/arch_docs/streaming-poc-docs/spec/README.md new file mode 100644 index 000000000..5c48edbd0 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/README.md @@ -0,0 +1,74 @@ +--- +doc_id: EWS-SPEC-INDEX +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +--- + +# External Workflow Streams — normative reference + +This directory is the canonical implementation contract for accepted External Workflow Streams +behavior. It contains exact lifecycle rules, protocol messages, provider operations, annotation +formats, runtime ownership, and failure classification. The human-facing [`guide/`](../guide/README.md) +summarizes these documents but does not override them. + +## Structured lookup index + +| Document ID | File | Canonical for | Related ADRs | +|---|---|---|---| +| `EWS-SPEC-PUBLIC` | [`public-surface.md`](public-surface.md) | Public names, handle roles, Worker configuration, stream identity, coexistence boundary | 001, 019, 021, 037, 040, 048 | +| `EWS-SPEC-BACKEND` | [`backend-contract.md`](backend-contract.md) | Provider capabilities, cursor and key semantics, immutability, staging, barriers, park operations, producers, fences, retention | 002, 003, 012, 019, 020, 040, 044, 047 | +| `EWS-SPEC-WFT` | [`wft-lifecycle.md`](wft-lifecycle.md) | Workflow Task admission, retention, timers, parking, output/park arbitration, shutdown, eviction, wake durability | 004, 009, 016, 017, 021, 030–032, 043, 045 | +| `EWS-SPEC-ANNOTATION` | [`annotation-format.md`](annotation-format.md) | Input replay grammar, output manifests, segmentation, byte budgets, replay checks, continuation state | 005–007, 018, 022, 027, 033, 039, 046, 048 | +| `EWS-SPEC-PROTOCOL` | [`core-lang-protocol.md`](core-lang-protocol.md) | Completion commands, activation jobs, marker envelope, readiness calls, Core state | 008, 013, 041 | +| `EWS-SPEC-PYTHON` | [`python-runtime.md`](python-runtime.md) | Sandbox boundary, decoding, output staging, cursor positions, delivery budgets, merge, close, job dispatch | 010, 011, 026, 028, 029, 034, 035, 037 | +| `EWS-SPEC-WAKE` | [`wake-signal.md`](wake-signal.md) | Signal envelope, request IDs, interception, wake cycles, producer send and recovery | 023, 025, 036, 038, 042 | +| `EWS-SPEC-FAILURE` | [`failure-taxonomy.md`](failure-taxonomy.md) | Failure classes, completion behavior, error types, metrics, operator response | 014, 015 | + +The corresponding decision records are indexed in [`../decisions/README.md`](../decisions/README.md). +Required observable behavior is indexed in [`../required-tests/`](../required-tests/). + +## Retrieval procedure for implementers and coding agents + +1. Select the document that owns the behavior from the table above. Do not infer a cross-component + rule from one implementation file. +2. Search that document for the exact protocol symbol, state name, provider operation, error type, or + lifecycle boundary involved. +3. Follow the linked ADR before changing the rule. The specification states current truth; the ADR + records why competing designs were rejected. +4. Check the required-test lists for the behavioral obligation and its mapped test before changing + either implementation or documentation. +5. Consult [`../verification-hazards.md`](../verification-hazards.md) before treating a failing or + passing integration result as evidence. + +## Precision conventions + +- `status: normative-pre-production` means the document is authoritative for the implemented design, + while the feature's production-readiness gate remains open. +- `doc_id` is a stable lookup key. File headings are descriptive and may become more readable without + changing that identity. +- Protocol names, state names, field names, operation names, and error types are exact identifiers. +- Tables describing outcomes or legal combinations are exhaustive unless explicitly labelled as an + example. +- A failure consequence explains why a rule is load-bearing; it is part of the contract rather than + background narrative. +- Candidate behavior belongs under [`../proposals/`](../proposals/) and is never silently mixed into + these specifications. + +Some ADR headers retain `P…` and `C…` labels from the implementation plan that preceded document +consolidation. They are historical annotations, not active requirement identifiers or links. Use the +ADR number, stable specification `doc_id`, canonical section, and required-test mapping for current +traceability. + +## Source-of-truth rule + +Each normative fact has one home. Other specifications link to that home rather than restating it. +Guide diagrams intentionally simplify the system and must always link back here. If a summary, +proposal, ADR rationale, test description, and specification differ, resolve them in this order: + +1. accepted specification; +2. accepted ADR for design intent; +3. required test for observable coverage; +4. human guide or proposal summary. + +An inconsistency is a documentation defect; it is not permission to choose the most convenient +interpretation. diff --git a/arch_docs/streaming-poc-docs/spec/annotation-format.md b/arch_docs/streaming-poc-docs/spec/annotation-format.md new file mode 100644 index 000000000..9294d2aa0 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/annotation-format.md @@ -0,0 +1,515 @@ +--- +doc_id: EWS-SPEC-ANNOTATION +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: replay-annotation-and-continuation-format +related_adrs: [ADR-005, ADR-006, ADR-007, ADR-018, ADR-022, ADR-027, ADR-033, ADR-039, ADR-046, ADR-048] +--- + +# Replay annotation format + +The opaque bytes Python encodes, Core stores in a marker, and Python decodes on replay. Core never +parses it. + +## The invariant + +Workflow code is deterministic given three things, and only these three need to be reproduced: + +1. The identical ordered sequence of stream-future resolutions delivered to Workflow code. +2. The identical set of points at which the runtime returned control with no further stream data + available. +3. The identical logical output records, topic-local order, and activation segment in which Workflow + code published them. + +Everything the annotation records exists to reproduce exactly that. A detail that affects neither +need not be recorded; a detail that affects either must be. + +The second condition is not redundant. Under `_single_batch_activation`, each stream activation +drives exactly one `_run_once(check_conditions=True)` drain, and stream jobs land in the non-query, +non-signal job set, so `workflow.wait_condition` predicates are evaluated once per activation. +Reproducing only the record order while changing how many drains occurred would change when +conditions fire. + +## Activation segmentation + +A retained Workflow Task may span many `ResolveExternalStreamWaits` activations but produces one +marker. That marker may cover both consumed input and produced output. Replay therefore must not +collapse the activations or run one event-loop schedule per direction (ADR-018, ADR-046). + +Core stays annotation-blind, so segmentation is reproduced inside Python rather than by issuing +multiple activations. The annotation is divided into **segments**, one per original activation. On +replay, Core issues a single `ReplayExternalStreams` job, and Python's replay driver walks the +segments in order, performing one event-loop drain per segment. The live run's *k* activations +become *k* drains inside one replay activation, so coroutine scheduling and condition evaluation +match. + +The optional output manifest records one per-topic count vector for every segment, including empty +segments. Those vectors are attached to the same segment schedule as the input runs. Input and +output therefore share one replay driver and one drain count; their segment counts are never added. + +*k*, and not *k + 1*. **The replay driver is not the only thing that drains.** The job lands in the +non-query job set, so the activation runs one `_run_once` of its own for that set once every job in it +has been applied — which happens whatever the driver did. So the driver drains the first *k - 1* +segments and arms the last, leaving that drain to the activation. This also mirrors the live run, +where each activation's single trailing drain served the records that activation had just been handed. + +Closing the replay therefore moves to after that drain: the check that nothing recorded was left +undelivered, and the cursor reposition onto the marker's committed boundary, both belong after the +final segment has actually been delivered. + +**A failing activation abandons the replay rather than closing it.** Leaving replay mode is +unconditional — a Run stuck in it has every later drain return nothing at all — but the checks and the +cursor move are skipped when an error is already propagating. The consumed check raises whenever a +recorded delivery is still armed, and after a failed drain one always is, since the drain that would +have taken it is the one that failed; an exception raised while another propagates **replaces** it, so +closing from a bare `finally` reported every failing replay activation as a nondeterminism error +blaming a `subscribe()` call nobody touched. That loses the diagnosis and the classification with it: +a `FailureError` out of Workflow code fails the Workflow, while a nondeterminism error fails the +Workflow Task and is retried. The cursor move has to be skipped in any case — an activation that +failed committed nothing. + +**A marker with no segments at all defers nothing, and closes before that drain instead.** A Workflow +Task that bound a wait and blocked with the stream never delivering writes a header and a terminal and +no segment. There is then no recorded segment for the activation's drain to serve, which makes that +drain a *live* one: records that arrived while the Run was evicted are already in the buffer and it +would hand them over. Repositioning after it would retract exactly what it had just delivered — cursor +back to the marker's boundary, buffer cleared — and the watcher would re-read and re-deliver records +Workflow code already had. Closing first retracts the buffer *before* the drain, so the drain finds +nothing and the watcher re-reads from the committed boundary; the records arrive on the activation that +re-read announces. + +A segment's delivery list is the recorded **global** order across waits, not one list per wait: it +is the order the runtime handed records over, which is the order Workflow code took them in. The +segmentation reproduces how many drains an activation ran; the order within a segment reproduces +which wait each of those drains served. + +This is safe with respect to Workflow time: all segments of a marker belong to one Workflow Task, +so `workflow.now()` is constant across them in both the live run and replay. Commands produced +during a retained Workflow Task are buffered until the task completes, in both directions. + +## Schema + +The annotation is a versioned binary encoding. `schema_version` leads so an incompatible format is +rejected explicitly. This unreleased feature has one current format and no legacy decoder. + +```text +annotation := header, (bindings | segment)*, terminal + +header := schema_version, streams[] // wait_id -> binding +bindings := streams[] // the same, for waits bound after the header +binding := (stream_key, start_cursor, provider_id, provider_format_version) + +segment := run*, segment_end_reason +run := (wait_id, first_offset, last_offset, count, control_positions) +segment_end_reason := NO_DATA_AVAILABLE | BATCH_LIMIT | FENCE_REACHED | BUDGET_ROLLOVER + +terminal := blocked_snapshot[] // wait_id -> cursor boundary: BEGINNING | AFTER(offset) +``` + +### Bindings are per wait + +A binding carries the stream identity chosen by Workflow code and the identity and format version +of the backend configured on the Worker. A stream-name mismatch is row four of +`failure-taxonomy.md` — nondeterminism, fixed by versioning the Workflow. A `provider_id` or +`provider_format_version` mismatch is a deployment problem, with the Workflow unchanged and the +store not necessarily damaged. + +### A wait bound after the header + +`subscribe()` may run at any activation of a retained Workflow Task, so a wait can be created after +the header frame has already gone to Core. Core appends observation deltas and never rewrites what +it already holds, so a header cannot be extended in place — that wait's binding rides its own +**bindings frame** instead (ADR-027), emitted with the delta of the activation that registered the +wait and therefore ahead of both the segment that first records a run for it and the terminal. +Without it the wait reaches the marker as runs and a terminal entry with no stream key, no backend, +and no start cursor, and replay of *unchanged* code fails as a wait "the Workflow did not create". + +Decoding merges every bindings frame into the table the header opens, so what replay reads is one +flat `wait_id -> binding` table and it never has to ask when a wait joined. + +A wait is bound **exactly once** per annotation. A second binding for the same `wait_id` is a decode +failure rather than a value replay picks between: whichever of the two stream keys it chose could be +the one the records were not written to. + +A late wait carries **its own** start cursor, not wherever the waits already in the header have +reached. Recording another wait's position for it names a boundary this subscription never stood at +— and for a wait that receives nothing, the start cursor is the whole of what replay knows about it. + +### Runs + +- A `run` is a maximal set of **consecutive deliveries from one stream**, in observed global order. + Alternating streams produce one run per delivery; a single-stream batch of 100,000 records + produces one run. This is why the schedule is encoded as runs rather than as individual + deliveries. +- **Both endpoints are recorded, not a start plus a count** (ADR-006). Backend offsets are ordered + but not dense, so `(first_offset, count)` does not determine where a run ends. +- The sequence of runs across all segments *is* the global cross-stream schedule. Per-stream ranges + alone are insufficient, because concurrent Workflow coroutines could otherwise observe a different + order. +- Control records occupy offsets and advance the cursor but are never yielded to Workflow code. + `control_positions` is a **sparse** list of the relative indices within the run at which a control + record occurred — not one kind tag per record. Fences are rare by construction, so this keeps a + run's control encoding proportional to the number of fences. +- **No field in a run is per-record.** A run costs two offsets, a count, and a sparse control list + whether it covers ten records or a hundred thousand. + +### Segments and the empty cases + +`segment*` and `run*`, not `+`, in both cases deliberately: + +- **An empty segment is meaningful**: an activation that drained and found nothing still ran one + `_run_once`, and replay must reproduce that drain or `wait_condition` predicates fire a different + number of times. +- **An annotation with no segments at all is meaningful too**: it is the first subscription to an + empty stream, where the header and terminal carry everything replay needs. + +### Positions + +- Every position in the annotation is a **cursor boundary**, `BEGINNING` or `AFTER(offset)`. Run + endpoints are record offsets; blocked snapshots are boundaries. The two are not interchangeable + and the encoding keeps them distinct types. +- `start_cursor` in a wait's binding makes its initial position explicit rather than re-derived. +- `ParkReason` is **not** in the terminal. It lives once, in the Core-readable + `ExternalStreamMarkerData.terminal_boundary` (ADR-008). + +## Output manifest and logical identity + +Workflow-originated output is represented by the optional +`ExternalStreamMarkerData.output` manifest. It contains no record payloads or provider offsets. It +records: + +- schema and fingerprint versions; +- the opaque provider stage token, the producing Run ID, and the exact History event immediately + preceding this Workflow Task's `WorkflowTaskScheduled` event; +- provider ID and provider format version; +- for each topic, the record count, deterministic logical byte count, SHA-256 logical fingerprint, + and whether the batch includes the explicit `FINISH` record; and +- a per-topic record-count vector for each shared activation segment. + +Fingerprint version 1 operates before payload codecs and external-payload storage. A logical record +frame contains the length-prefixed UTF-8 topic, length-prefixed numeric record kind, metadata-entry +count, metadata keys sorted by their UTF-8 bytes with length-prefixed keys and values, and +length-prefixed payload data. The batch digest is SHA-256 over the ordered canonical frames, each +itself length-prefixed. `logical_byte_count` is the sum of frame lengths, excluding those batch-level +prefixes (ADR-046). + +On replay, Python recreates those logical frames using only the deterministic payload converter and +validates topic order, kind, count, logical bytes, fingerprint, finished state, and the shared +segment schedule. It does not mint a token, stage bytes, consult live capacity or latency settings, +or run a payload codec. A mismatch is nondeterminism. + +On a live staging retry, the same token and logical manifest are mandatory even when a randomized +codec would produce different encoded bytes. The provider's first successfully staged encoded bytes +remain authoritative. The exact History floor plus stage token is the positive commit proof used by +lazy reconciliation; no elapsed-time inference can replace it (ADR-044). + +## Replay validation + +Replay reads the inclusive range `[first_offset, last_offset]` for each run and verifies: + +1. both endpoints are present, +2. the range contains exactly `count` records, +3. ordering is strictly increasing under the provider's comparator, +4. `control_positions` match. + +A first, middle, or last deletion each fails a different one of those checks, and all four are +cheap. **These four checks are the whole of replay validation.** They are sufficient precisely +because every provider guarantees the bytes cannot change (ADR-003), so the only damage replay has +to detect is a record that is no longer present. + +A provider that cannot support this verification over a compact range must encode the exact offset +sequence in place of a compact run. + +Replay then follows the marker's recorded availability/blocking decisions rather than consulting +current stream timing. + +### What replay checks about the Workflow's own code + +Those four are about the bytes. Three more are about the code that is replaying, and all three +report row four of `failure-taxonomy.md` — nondeterminism — rather than integrity loss, because the +recorded ranges are exactly where they were written and it is the Workflow that moved. + +1. **Each recorded wait's binding**, when that wait is registered, and again when a marker is + replayed onto subscriptions that already exist. The Workflow-code comparison is the stream + *name*. +2. **Every recorded delivery was taken.** A replay that reaches the end of the last segment still + holding one is running code that consumed less than History says it consumed. +3. **Every bound wait was recreated**, checked once after the last segment as `bound ⊆ registered` + and reported naming the waits that are missing. + +The third does not follow from the other two, and what it catches is the ordinary shape rather than +an exotic one. A binding is written for a subscription whether or not anything was ever delivered +through it — the first observation has to carry provider identity, stream key, and start cursor even +for a stream that stayed quiet for the whole Workflow Task — so a `subscribe()` on a quiet stream +produces exactly a binding with no runs behind it, which every check that reasons from deliveries is +blind to. Two removals escape without it. Removing the **last** `subscribe()` on a quiet stream +renumbers nothing and leaves nothing undelivered, so the replay is accepted although the Workflow +now holds one subscription fewer than History says it did. Removing a **middle** `subscribe()` where +the later waits name the same stream and backend is worse: every survivor renumbers down by one, the +binding comparison therefore compares equal for all of them, and the records recorded for wait *k* +are consumed by what was subscription *k+1* — a different cursor, a different consumer, and nothing +left over for the delivery check to find. + +**The check is one-directional deliberately** (ADR-033). `bound ⊆ registered` is the invariant; +`registered ⊆ bound` is not, because replay runs the Workflow forward past the Workflow Task the +marker covers and the subscriptions it makes there belong to the *next* marker's header. For the +same reason it is checked once at the end of replay rather than before each segment: a wait bound by +a later frame legitimately does not exist yet while an earlier segment is being delivered. + +## Byte budget + +The encoder enforces a hard budget, `MAX_ANNOTATION_BYTES`, chosen well below the server's event +size limit and declared as a constant rather than a guideline. It is enforced at encode time, not +checked afterwards (ADR-007): + +1. When the accumulated annotation for the current Workflow Task passes a high-water fraction of + the budget, Python sets `request_rollover` on its next `WorkflowStreamProgress`. +2. Core rolls the task over as it does on deadline expiry, **minus the finalization round trip**: + the triggering `WorkflowStreamProgress` came from Python and already carries the terminal, so + Core writes the marker and completes with `force_new_wft = true` without issuing + `FinalizeExternalStreams`. This is the one Core-initiated completion that needs no finalization + job, and it needs none because Python decided the boundary. +3. The next Workflow Task starts a fresh annotation whose header records the current cursors, so + consumption continues uninterrupted across the split. +4. The segment that triggered it ends with `BUDGET_ROLLOVER`, which is how replay knows the batch + continues in the following marker. + +An annotation can therefore never exceed the budget: the runtime rolls the Workflow Task over +instead of growing the marker. Replay reassembles multi-marker batches in Workflow Task order, +matching each marker to its Workflow Task rather than concatenating blindly. + +### The high-water mark alone is not a bound + +A high-water mark is a *fraction* of the budget, and it becomes true only once a frame that crossed it +has been emitted. Frames are indivisible, and three of the four have no length this side chooses: a +binding carries the namespace, Workflow ID, first-execution Run ID, stream name, and provider id; a +run carries two provider-supplied offset strings. Any of them can be larger than the +slack a fractional mark leaves, and the batch a frame records cannot be moved to the next annotation — +its deliveries happened in *this* Workflow Task, and that task's marker is where replay has to find +them. So "roll over at 75%" is a policy, not a guarantee, and three further rules are what make the +guarantee hold for a frame of any size. + +**The closing frames are reserved, not checked.** The terminal, and a bindings frame for any wait +registered since the header went out, are priced in advance and only they may spend that reservation. +Segments are refused before it is gone. Neither may ever be refused, because both record something +that already happened — and an annotation Core writes with no terminal is durable and cannot be +decoded past the frame after it. + +**Delivery stops before a segment it could not record.** The annotation budget is a second delivery +budget alongside the per-activation record cap, and the smaller one wins. It is spent *before* a +record is handed to Workflow code — the only point at which stopping is still an option — priced per +record by the largest run this annotation has actually encoded, floored so that the first record of an +annotation is costed pessimistically rather than optimistically. The measurement belongs to the +**annotation, not the segment**: the open segment is emptied at the end of every activation, so a +per-segment maximum re-applies the bare floor to the first record of *every* activation, and a real +run costs two provider-chosen offset strings. Delivering on that price hands over a record the closing +segment then cannot record. The activation then ends with +`BUDGET_ROLLOVER` and the same completion asks Core to roll over. Records the annotation budget left +buffered have their readiness re-reported exactly as the record cap's do. + +**Stopping and asking are one mechanism, and either alone wedges the Workflow.** Stopping bounds the +marker; the rollover is what gives the next Workflow Task a fresh annotation to deliver into. Stop +without asking and the next activation begins against the same full annotation, is handed a delivery +budget of zero, delivers nothing — and therefore *observes* nothing. So the condition may not be +phrased in terms of what this activation observed: "the annotation can afford no more" is a property +of the annotation, and one written as a property of the activation stops being true the moment the +completion path takes the delta, which is what clears the observed flag. + +**The rollover request rides the completion that crossed the line.** It is read *after* the +activation's segment is closed, not before. Read before, the flag describes the annotation as it stood +one activation ago: the segment that crossed the high-water mark went out with +`request_rollover = false`, and the following activation was free to add another frame before Core had +ever been asked to close anything. One activation of delay is the entire margin the mark exists to +provide. + +**A subscription set whose header cannot fit is refused at `subscribe()`.** A rollover writes a +*fresh* header, so an oversized header is not something rollover can fix — every annotation would be +the same size, and the Workflow Task would fail identically on every retry with no marker ever +written. The capacity question is therefore asked where the answer is still "do not make this +subscription": inside the Workflow's own `subscribe()` call, priced against an empty annotation +(header plus terminal, no segment), deterministic and so reproduced under replay. + +A price is not a proof, though: nothing has seen the offsets of the record it is pricing. Two things +close that gap. + +**A margin absorbs a misprice and turns it into a rollover.** Behind the closing reserve sits a spill +margin that a segment frame — and only a segment frame — may overrun into. Affordability is measured +against the line *before* the margin, so a mispriced record lands in it rather than past the budget, +the same completion asks Core to end the Workflow Task, and the terminal is still reserved behind it. +The margin also fixes the floor `subscribe()` checks: a fresh annotation is guaranteed room for its +header, its terminal, one segment frame, and the margin, which is what makes at least one record +affordable in every annotation. Without that guarantee a Workflow can deliver nothing, observe +nothing, and therefore never even ask for the rollover that was supposed to save it. + +**A run larger than the margin is a capacity limit, reported as one.** A provider whose offsets are +far longer than anything measured can make a single run cost more than the whole budget has left, and +no rollover helps — the next annotation has to carry the same run. That boundary is refused where the +record has been drained but not yet handed to Workflow code, as the same non-retryable +`ExternalStreamCapacityError` that `subscribe()` raises, naming the provider's offsets. It fails the +Workflow, not the Workflow Task, so the encoding that cannot fit is not retried forever. + +`AnnotationBudgetExceeded` remains behind all of that as the encoder's own last line, and it is a +**non-retryable application failure** for the same reason: the server retries Workflow Task failures +regardless of cause, so a plain exception there is the permanent retry loop ADR-007 exists to rule +out. + +### What actually drives marker growth + +One honest worst case remains: an alternating multi-stream workload, `A,B,A,B,…`, has a schedule +transition per record and cannot be compressed by range encoding. It produces one run per delivery, +hits the high-water mark sooner, and rolls over more often. Bounded marker size is bought with +additional Workflow Tasks. + +The output manifest has its own hard 64 KiB encoded-size bound. Python measures the exact protobuf +before running a codec, external payload store, or provider operation. A single manifest that cannot +fit is rejected as `ExternalStreamCapacityError`; topic names or activation-segment metadata must be +reduced. Output record and logical-byte capacities are also computed from the canonical pre-codec +frames for the whole retained Workflow Task, never reset per activation. Reaching either requests a +replacement task; one record larger than the configured logical-byte capacity is rejected before +external I/O (ADR-046). + +For the input annotation, that is the *only* driver of growth, so its budget is not expected to fire +at all in the single-stream input scope of Milestone 1. It is still enforced at encode time rather +than assumed, and tests assert **encoded byte size** rather than run count — a run-count assertion +tells you nothing about what a future per-record field would cost. Output topic and segment metadata +is bounded independently as described above. + +## Delta accumulation + +`WorkflowStreamProgress.observation_delta` carries the frames produced since the previous progress +report for the same Workflow Task — the header on the first of them, then whichever bindings and +segments that activation produced. Core appends each delta to `ExternalWaitSet.replay_annotation` +and writes the accumulated result as the marker's `replay_annotation`. Core never parses either. + +Output uses the same marker rather than a second marker or replay job. The compact output manifest +is recorded alongside any input annotation when the output batch is staged. Its segment vectors +describe the shared activation schedule: they align with input segments when input is present and +supply that schedule themselves for an output-only marker (ADR-046). + +## Replay read path + +Replay distinguishes three kinds of waiting: + +| Kind | On replay | +|---|---| +| Waiting for *new* records — watchers, idle timer, park handshake | Never occurs. Core starts no timers and Python starts no watchers. | +| Reading *recorded* offsets | Occurs. Ordinary blocking backend I/O, bounded by the provider's read timeout. | +| Blocking on backend latency or unavailability | Not a determinism failure — a transient backend failure. | + +Python reads the full recorded offset range for each stream up front, in as few provider calls as +the range allows, then delivers from memory in recorded order. Replay I/O cost is therefore a +function of the consumed range, not of segment count. + +**A replay drain takes from the front of the segment, and only while the front belongs to the wait +that is asking.** The segment is the global order, so a drain that searched past another wait's +record would hand its asker a record that came *after* it live: a segment recorded as (wait 2, +wait 1) replays as (wait 1, wait 2) for any reader that asks in `wait_id` order, and a wait +appearing twice with another between them collapses two of the segment's drains into one. A drain +that meets another wait's record answers empty, which is exactly what the live drain answered — +the record was not in that wait's buffer yet — and it leaves the record where the drain that +recorded it will find it. + +## Cursor origin + +A cursor is never derived from mutable backend state, on any Run. + +- **First execution of a chain:** the subscription starts at the `BEGINNING` boundary, recorded as + the `start_cursor` of its binding in the subscription's **first** observation delta — which is + emitted whether or not a record was ever delivered. That is the header frame for a wait the + annotation opened with, and a bindings frame for a subscription created after the header went + out. +- **Subsequent Runs:** the committed continuation state arrives in the reserved Continue-As-New + header as an `AFTER(offset)` boundary, persisted in `WorkflowExecutionStarted` and restored before + any subscription is established (ADR-022). + +Both paths populate the same binding field, so replay reads an explicit starting boundary in every +case, including the case where the stream was empty for the subscription's entire life. + +### The continuation header carries the whole binding + +Keyed by `wait_id`, and carrying per wait what a binding above carries: the cursor, the stream name, +and the provider identity — `provider_id` and `provider_format_version`. The cursor +alone cannot say *what it is a position in*, and an offset means nothing outside the store that +produced it, so a successor that resumed on the wait number alone would hand a predecessor's offset +to a store that never held those records — which the backend accepts, and which skips everything +before that boundary silently. + +Marker replay makes the same comparison for every recorded range it reads. **A successor Run's first +live read happens before it has written a marker that could make it**, which is the whole reason the +continuation carries the binding rather than leaving it to replay (ADR-039). + +Which half disagrees decides how it is reported, and it is the same split the binding table uses: + +| Disagreement | Reported as | +|---|---| +| Stream name | Row four of `failure-taxonomy.md` — Workflow code moved | +| `provider_id`, or `provider_format_version` | Row one — the configured backend changed | + +Both are raised where the subscription is registered, before the cursor reaches the manager, so no +backend ever reads at a boundary that was not produced against it. + +**A recorded format version is compared exactly, including zero.** Nothing in the provider contract +reserves a sentinel value. + +### The continuation is snapshotted where no Workflow code can still run + +Creating the Continue-As-New command does not end the activation: the Workflow's event loop keeps +draining ready tasks after a terminal command is added, so a stream consumer scheduled or unblocked +after that point still advances the consumption cursor. Those consumptions do reach the +predecessor's final marker, which is closed on the way out, so a header serialised at command +creation names an earlier boundary than History does and the successor is handed a record a second +time. + +Every Continue-As-New command on the completion is therefore re-serialised at the point the +activation emits its stream commands — the same place the observation delta that commits this +boundary is emitted, and the first point at which the boundary has stopped moving. Every command +rather than the last, because two top-level coroutines can each raise one and only Core decides +which terminal survives. + +Neither this nor anything about the header's content needs an SDK internal flag. Core matches a +Continue-As-New command to its `WorkflowExecutionContinuedAsNew` event by command type alone and +never compares the command's headers against the recorded ones, so a replay that regenerates the +header at the later boundary cannot disagree with a History written at the earlier one. What the +successor resumes from is the copy in its own `WorkflowExecutionStarted`, which is durable and which +replaying the predecessor does not rewrite. + +### The continuation format is must-understand + +The provider binding is encoded inline in every continuation entry and decoded before Workflow +code runs. A Worker that cannot validate the binding must not start the successor Run. Because the +feature is unreleased, the encoder writes one current schema and the decoder accepts exactly that +schema; there is no legacy reader, writer-stage selector, or compatibility mode. An incompatible +version fails explicitly instead of restoring a cursor under checks it does not understand. + +### Finished output topics use a separate must-understand header + +Input cursor continuation and output terminal continuation are distinct reserved headers. The output +header stores its own schema version, provider ID, provider format version, and the sorted set of +topics whose explicit `FINISH` record committed in the predecessor chain (ADR-048). + +The successor decodes and validates that header before Workflow code runs and before any backend +read. It restores those topics as permanently finished, so a later `publish()` or `finish()` for one +fails immediately and deterministically. A missing, unsupported, or mismatched header is never +silently ignored. + +Only an explicit Workflow `finish()` or direct producer `finish_writing()` appends `FINISH`. +Workflow success, failure, cancellation, termination, and Continue-As-New do not synthesize a topic +terminal. + +## Subscription numbering is a nondeterminism hazard + +`wait_id` is allocated from a per-Run counter in `subscribe()` call order, which puts it in the same +hazard class as timers and activities: inserting, removing, or reordering a `subscribe()` call +renumbers every later wait in that Run. + +- Such a change is nondeterministic for Workflows already running against deployed code and must be + gated behind `workflow.patched()`, exactly as an inserted timer would be. +- Adding a subscription on a code path a running Workflow has not yet reached is safe. +- **Detection is by binding, and it is not complete.** A wait whose stream changed is + reported where it is registered, and a wait the marker bound that the Workflow no longer creates is + reported at the end of replay — both as row four of the failure table (`failure-taxonomy.md`) + rather than as a silently different stream result. What no check can see is a change among waits + whose bindings are *identical*: inserting a `subscribe()` in the middle of several subscriptions to + the same stream renumbers them without changing any comparison, and an + addition at the end is a supported change, so the two are indistinguishable from the annotation. + That residue is why this is stated as a rule about Workflow code rather than left to a check. diff --git a/arch_docs/streaming-poc-docs/spec/backend-contract.md b/arch_docs/streaming-poc-docs/spec/backend-contract.md new file mode 100644 index 000000000..45c2a4573 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/backend-contract.md @@ -0,0 +1,377 @@ +--- +doc_id: EWS-SPEC-BACKEND +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: backend-provider-contract +related_adrs: [ADR-002, ADR-003, ADR-012, ADR-019, ADR-020, ADR-040, ADR-044, ADR-047] +--- + +# Backend contract + +What a stream provider must implement to be registrable, and what the producer side must supply. + +## Independent provider capabilities + +Input and output are separate ABCs and conformance suites: + +- `StreamBackend` supplies append, inclusive replay reads, exclusive live reads, and input + park/wake coordination. +- `OutputStreamBackend` supplies staged output, direct committed append, barrier-aware reads, + committed tail, stage inspection, and offset comparison. + +An input-only provider remains valid. An output-only provider can serve +`ExternalOutputStreamProducer` and `ExternalOutputStreamClient`. The first-release Worker option is +validated as `StreamBackend`, so Workflow-originated output requires the configured provider to +implement **both** contracts. Capability separation prevents output staging from becoming a +breaking requirement for existing input providers. + +Every implemented capability declares structural immutability, `provider_id`, and +`provider_format_version`. + +## Input operations + +A backend implementation must provide: + +- Immutable, append-only records. +- Stable, totally ordered offsets within a stream. +- An **inclusive range read** over an explicit `[first_offset, last_offset]` pair, and a separate + **exclusive** watch for records strictly after a boundary. These are two distinct operations — + see "Cursor semantics". +- **A guarantee that a record's bytes cannot change once written.** Mandatory, checked at + registration (ADR-003). +- Atomic or otherwise race-free coordination for parking and wakeup. +- Retention sufficient to re-read every record referenced by retained Workflow History. +- Idempotent coordination operations and detection of missing records. +- **Idempotent append that is idempotent on identity, not on key alone** (ADR-020). An append + reusing an existing `(session_id, sequence)` with byte-identical content is a no-op returning + the original offset; the same pair with *different* bytes must be rejected as an error. +- **Key derivation that is injective** — see "Key derivation must be injective". + +Control records and data records share the same offset sequence. Control records are consumed by +the runtime and are not yielded to Workflow code. + +## Output staging and reads + +Workflow output is staged per topic sub-batch under an immutable `OutputStageManifest` carrying the +output `StreamKey`, provider binding, unique stage token, current Run, exact History floor, +sub-batch index, fingerprint version and digest, record count, and logical byte count (ADR-044). +The provider exposes: + +- `stage_output(manifest, records)`: atomically places an ordered `PENDING` sub-batch. Repeating the + exact logical manifest returns the first placed offsets and bytes; another manifest under the + same `(stage_token, sub_batch_id)` is a conflict. +- `commit_output(manifest)` and `abort_output(manifest)`: idempotent terminal transitions that + cannot be reversed. +- `output_stage(manifest)`: exact stage inspection for reconciliation. +- `append_output(key, record)`: an immediately committed singleton used by Activities and external + processes. +- `read_output_after(key, boundary, ...)`: the committed prefix strictly after the boundary plus, + when present, the first unresolved `PendingOutputBarrier`. +- `output_tail(key)`: the boundary after the readable committed prefix. It does not cross pending + data. + +A pending stage occupies provider order but is not readable. No read may return a record at or +beyond its first offset until ADR-044 commits or aborts it (ADR-047). Aborted records are skipped; +committed records are yielded in offset order. Commit and abort are coordination metadata and never +rewrite record bytes. + +Workers reconcile their own pending stages after reporting when possible. Every +`ExternalOutputStreamClient` also performs built-in lazy reconciliation at a pending head: it reads +the exact producing Run's complete History strictly above `history_floor_event_id`, commits on the +exact marker token, aborts on the first durable Workflow Task closing boundary or Workflow closure, +and otherwise leaves the barrier pending. A backend or Temporal outage is transient storage +failure; unavailable deciding History is integrity loss. + +## Cursor semantics + +A cursor is a **position boundary**, not the identity of a record (ADR-002): + +```text +cursor := BEGINNING + | AFTER(last_consumed_offset) +``` + +`BEGINNING` is the provider's beginning-of-stream boundary and is not required to be the offset of +any real record. `AFTER(x)` names the boundary immediately following the record at offset `x`, +whether or not a record after `x` exists yet. + +The boundary form maps directly onto the two primitives every provider must expose: + +- **Live resume and watch read strictly after the boundary.** In Redis this is `XREAD BLOCK` from + `last_consumed_offset`, or from the beginning sentinel `0-0` for `BEGINNING`. `XREAD` returns + entries *strictly after* the supplied ID, which is exactly exclusive-after semantics. +- **Replay reads an explicit recorded range, never "from the cursor".** The annotation records + each run's `first_offset` and `last_offset`, and replay issues an inclusive range read for + precisely that range — in Redis, `XRANGE `. Replay never asks the backend what + comes next; the answer is already in the marker. + +A provider may represent the boundary as `(offset, inclusive | exclusive)` instead. What it may +not do is require the cursor token to be the offset of a record that does not exist yet. + +**Offsets are compared by their provider's ordering rule, not lexically.** Redis IDs compare as +numeric `(milliseconds, sequence)` tuples; string comparison is wrong as soon as the millisecond +component changes width. + +## Structural immutability is a registration precondition + +**Every provider must guarantee that a record's bytes cannot change once written.** A backend that +cannot make the guarantee does not satisfy this contract and is rejected when it is registered on +the Worker, rather than compensated for at runtime. Redis Streams qualifies: an entry can be +deleted by `XDEL` or removed by trimming, but its fields cannot be rewritten in place. + +Given that guarantee, replay needs to detect exactly one class of damage — a record that is **no +longer there** — and offsets are sufficient for it. Deletion, trimming, and retention expiry are +the realistic failure modes in operation, and all of them are caught by the four range checks in +`annotation-format.md`. + +What the assumption costs, stated plainly: if a provider silently violates immutability — a buggy +custom backend, or out-of-band surgery on the stream — replay will deliver the altered bytes as +though they were original, and no error is raised. That risk is accepted deliberately and is +bounded by the registration requirement and the conformance suite. `schema_version` leads the +annotation encoding, so a per-record content-hash mode can be introduced later without a format +break if a provider ever needs one. See ADR-003. + +**A decode failure is not an integrity failure.** With immutability guaranteed, bytes present +within a validated range are exactly the bytes that were written, so a DataConverter or codec that +cannot decode them indicates a configuration mismatch between producer and consumer — not a +damaged stream (ADR-015). See `failure-taxonomy.md`. + +## Key derivation must be injective + +**Distinct `StreamKey` values must map to distinct physical keys**, and to distinct keys for every +structure derived from one: records, idempotency state, park intents, and claims. + +Every string component of a stream identity — namespace, Workflow ID, first execution Run ID, and +stream name — is user-chosen, and direction is an additional enum component. All five participate +in physical identity. An input and output topic with the same four strings must not share records, +idempotency state, park metadata, stage status, or claims. + +Each string may contain a provider's delimiter. Joining them raw +is therefore not injective: with `:` as the delimiter, `("ns", "wf", r1, f"{r2}:tokens")` and +`("ns", f"wf:{r1}", r2, "tokens")` render identically. Two unrelated Workflows then share one +stream, one idempotency hash, one park intent and one claim. Each reads the other's records, which +is visible; and each concludes the other's claim has taken responsibility for the wake, so neither +producer signals and both Runs wait forever on records already durable, which is not. + +Any injective encoding satisfies this — escaping the delimiter, percent-encoding each component, or +length-prefixing them. A provider that derives a pattern match from a key owes the same property +there: an identity carrying a metacharacter must not widen a scan onto another stream's intents. + +## Conformance suite requirements + +The suite is the deliverable, not the interface. It must contain, at minimum: + +- a case that parks a consumer at the current tail, appends a record whose ID could not have been + predicted, and resumes — a backend that requires a nameable next ID fails it; +- a case that fails a backend implementing the inclusive range read with **exclusive** semantics, + since that error is invisible until the first replay; +- a case that fails a backend comparing offsets lexically, crossing a millisecond-width boundary; +- a case that fails a backend accepting an idempotency-key reuse with different bytes; +- park intents keyed by stream alone failing the two-subscription case (ADR-012); +- a case that removes an installed intent and requires `current_park_generation` to report nothing + afterwards, since a provider that keeps answering is wrong only where nothing else looks at it; +- a case that removes the same intent twice, and one that removes an intent that was never + installed, both of which must succeed — a removal that failed is retried later against whatever + the key holds by then, so a provider that raises on the second call turns a completed cleanup into + a permanent one; +- a case that conditionally removes an intent by Run ID and park generation, refuses mismatches, + leaves a replacement intent intact, and reports an absent key as absent rather than as a + mismatch; and +- a claim that never expires failing the leased-claim case; +- output staging repeated with the exact manifest returning the first placed records and actual + terminal status, including a retry after commit or abort; +- reuse of `(stage_token, sub_batch_id)` with a changed manifest failing atomically without partial + records; +- committed direct output after a pending stage remaining unreadable until that stage commits or + aborts; +- repeated output commit and abort being idempotent while reversal is rejected; and +- input and output keys with otherwise identical components remaining physically isolated. + +## Parking operations + +`install_park_intent`, `remove_park_intent`, `remove_park_intent_if_matches`, `park_intent`, +`recheck`, `parked_wait_ids`, `claim_park_generation`, `current_park_generation`. + +`parked_wait_ids` is enumeration rather than a new concept — the `wait_id` half of the intent key, +for one stream — and it is what makes the other operations reachable from the producer side at all. +A producer knows the stream it appended to and nothing about the Workflow's subscriptions, because +`wait_id` comes from a per-Run counter inside `subscribe()` that no producer can see. + +Intents are keyed **`(stream key, wait_id)`**, never by stream key alone (ADR-012), carrying the +cursor boundary, the `park_generation`, and the current Run ID as the intent's *value*. + +The key does not include the current Run ID: `wait_id` is stable across a Continue-As-New chain +and the stream key already contains the first execution Run ID, so the key is unique within a +chain, and only one Run of a chain is live at a time. Carrying the Run ID as the value means a new +Run's intent deterministically replaces its predecessor's for the same key rather than +accumulating alongside it. + +The value must read back as it was written, because that is how a removal decided on earlier +identifies what it is removing. `wait_id` restarts at 1 in a Continue-As-New successor while the +stream key does not change, so a key alone does not distinguish a predecessor's abandoned intent +from a successor's live one. `remove_park_intent_if_matches` must compare the recorded +`park_generation` and Run ID and remove the intent in one atomic backend operation +(`wft-lifecycle.md`). Splitting the comparison and removal across calls lets a successor replace the +intent between them and lets the predecessor delete a live park. + +**It answers with three outcomes, not two.** *Removed*, *absent*, and *mismatch* — and the last two +may not be collapsed, even though neither changes anything. A provider can commit the delete and lose +the connection before its reply arrives; the retry that follows meets a key it cleared itself. Told +"absent", the consumer knows the intent it named is gone and that a record the intent silenced may +need announcing again. Told "mismatch", it knows an intent it must leave alone is in the way and that +the suppression has not ended. One answer for both makes the first case look like the second, which +is a record left undelivered rather than an intent left installed. + +**An intent exists only while its park is outstanding.** The consumer holds up one half of that +across three points — the resolve that ends a park the Run is sitting in, the reconciliation at +registration for an intent this Worker inherited rather than installed, and the cancellation that +closes a wait — with a removal none of them got confirmed carried as owed until one of them does +(`wft-lifecycle.md`). All of that is needed because the intent is durable backend state while the +record of which Worker installed it is per-Worker, so an eviction or a handoff otherwise leaves it +forever. The provider holds up the other half: once an intent is removed, `current_park_generation` must report nothing for that +subscription. A provider that answered from a remembered "last generation" beside the intent passes +every other requirement here and still breaks both of that call's readers, in the same direction and +invisibly. A producer would name a generation Core has already discarded, and a non-zero generation +the Run does not recognize is exactly what Core ignores as stale — so the record is appended, the +Signal is sent, and the Workflow is never woken. The consumer's own shutdown sweep would read the +same answer and send a parked wake where it owes the unparked one; a parked wake's request ID ignores +sender identity by design, so it arrives byte-identical to the wake that ended that park and the +server deduplicates it away. + +**Claims must be leased and renewable.** A claim is how a provider learns a wake is in flight, and +its expiry is what makes one abandoned by a crashed producer takeable again; a claim that never +expires says a wake is in flight for the life of the store, and nothing can take it back. What +keeps a producer crashing between claiming and signalling from stranding the generation is not the +lease — expiry permits takeover, it schedules nobody to take over — but the producer contract that +a producer which loses the claim sends the wake anyway (`wake-signal.md`). A provider that cannot +lease must expose observe-only semantics and always grant, which is the same behaviour every +provider's callers already rely on. + +## Producer binding + +A producer needs five things, none of which it can infer (ADR-019): + +- **The Workflow chain key**, including the first execution Run ID. `temporalio.activity.Info` + exposes `workflow_run_id` but *not* the first execution Run ID, so an Activity cannot derive the + key. The Workflow passes it to the producer explicitly — as an Activity argument, or through + whatever channel a non-Temporal producer already uses — and the producer verifies it by + describing the Workflow before its first append. Publishing under an unverified key is a + configuration error, not a silent no-op. +- **A backend connection.** A Worker has one configured backend; a plain process constructs a + provider directly. +- **A Temporal client**, for the wake Signal. +- **The same `DataConverter`** the consuming Workflow uses, including any codec, **bound to the + same serialization context** — which the producer derives from the chain key it was given, so it + matches what the consuming Worker converts under (`python-runtime.md`). The context is part of the + requirement rather than an implementation detail of it: two context-free sides agree and two bound + sides agree, but one of each encrypts under one key and decrypts under another (ADR-035). A + mismatch either way is detected at decode time on the consumer and surfaces as a distinct decode + failure. +- **A stable producer session ID and sequence**, which is what makes append idempotent under + Activity retry. Activities default it to a value derived from the Activity's identity so a + retried attempt reuses it; plain processes must supply one, and the API requires it rather than + defaulting to a fresh random value. + + *Stable* is a claim about the sequence too, and it holds only because the number is drawn when the + call is made rather than when its payload finishes encoding. A codec may do real I/O — an external + payload store, a KMS round trip — and its completion order is not reproducible across attempts, so + a sequence drawn afterwards belongs to that order instead of to the call. Two concurrent publishes + then exchange keys whenever the store answers the other way round: on one stream the backend sees + a stable key reused with different bytes and raises the conflict below, and across topics, where + deduplication is per stream key, the swap appends duplicates and raises nothing. Either way a valid + concurrent Activity is made non-retryable, or silently doubled, by external timing. + +The stream name appears exactly once on the producer side, in `topic()`. `connect()` takes the +Workflow *chain* key — namespace, Workflow ID, first execution Run ID — and `topic(name)` +completes it into the full stream identity, so one connection serves several topics and no two +arguments can disagree about the name. + +The direct output producer uses the same explicit `WorkflowChainKey`, chain verification, +Workflow-bound serialization context, stable session ID, and invocation-order sequence rule. It +sends no wake Signal: output clients watch the backend. An append with no reported outcome raises +`OutputAppendNotAcknowledgedError`; recovery repeats the exact carried record through +`resolve_append()` rather than drawing a new sequence. `finish_writing()` waits for every earlier +publish on that topic and then appends `FINISH`; unlike the input producer's write fence, it closes +the producer's topic handles. + +## Write-fence semantics + +`finish_writing()` means: + +> All writes in this producer session preceding the fence have been appended. If a consumer drains +> through the fence and no later record is immediately available, it may park now. + +The fence is an ordered stream record, so its relationship to concurrent data is unambiguous: + +```text +offset 100 data +offset 101 data +offset 102 WRITE_FENCE(producer-session-id) +offset 103 data from another producer +``` + +At offset 102, the runtime continues if offset 103 is already available; otherwise that +subscription becomes immediately parkable. The fence neither closes the stream nor asserts that +all producers are finished. A fence on one stream only marks that stream parkable — the Workflow +Task parks early only when every active subscription is immediately parkable; otherwise the idle +timeout remains authoritative. + +### The claim has to be enforced, not just documented + +**A fence is appended only after every `publish()` invoked earlier on that stream has reached a +durable append** — earlier by invocation order, across every handle `topic()` has returned for the +name, since the stream is one thing and the handle is not. Without that hold the claim is simply +false: a publish draws its sequence *before* awaiting the payload codec (see "Producer binding"), so +one invoked first can still be encoding when the fence is called, and a fence that overtook it parks +a consumer in front of data that has not been written. In a `wake=False` batch it is worse than +early — the fence carries the batch's only wake, so it spends it on records that do not exist yet. + +**The order holds fences behind data writes and behind nothing else.** Two concurrent fences make +independent claims about the publishes each of them was invoked after, and neither is inside the +other's claim, so they are unordered with respect to each other. Ordering them would only let a +fence that never reached the backend — cancelled while waiting, or refused — stand in for a data +write that went missing, and refuse a fence with every write it claims already durable (ADR-040). + +What an earlier publish's outcome does to the fence follows the three outcomes an append has +(`wake-signal.md`): + +| Earlier publish | The fence | +|---|---| +| Durable | Appended behind it | +| Failed — no record, and none coming | Refused with `PrecedingWriteFailedError`, nothing appended | +| Unknown — the acknowledgement window | Refused with **that** operation's `AppendNotAcknowledgedError` (ADR-038) | + +A fence appended over the second row's hole tells a consumer the batch is complete when it is short +a record; the caller chooses between publishing the value again — which draws a new sequence number +and lands ahead of a later fence — and accepting the batch without it, and either way the next +`finish_writing()` appends a fence because the failed write is no longer outstanding. + +The third row is a refusal on the operation's own terms and not on the fence's, because the stream +takes no further append at all until it is settled, and the record may well be durable and belong in +front of the fence. **`resolve_append()` is therefore what decides which of the first two rows the +fence lands in**, and a fence must read that decision rather than infer it: a durable resolution +puts the record ahead of the fence, and an `AppendConflictError` proves it absent and makes the +fence's answer the second row, chained from that conflict. The two are indistinguishable from +whether the record is still outstanding, since resolving it either way stops it being outstanding. + +Where both an unresolved and a definitely-failed write precede one fence, the unresolved one is +reported, because it is the one that blocks the other's recovery: republishing a failed value is +itself refused while an append on that stream is unsettled. + +## Retention + +Records and control metadata must remain available and immutable for as long as any retained Run +in the Workflow-ID chain may replay them. Backend retention must therefore cover at least the +namespace's Temporal retention for that chain. Garbage collection is allowed only after the chain +is terminal and its applicable Temporal retention/replay window has elapsed, or after an explicit +stronger archival policy guarantees replayability. + +This is an operational prerequisite, not a code deliverable. Violations surface as stream-integrity +failures. + +Output providers additionally retain committed records for their advertised client-resume window +and retain pending-stage resolution metadata for at least as long as the deciding Temporal History. +The SDK chooses no default retention period and exposes no operator force-commit/force-abort API. +Garbage collection is safe only after History proves abort, after the resume and History windows +both expire without a remaining reference, or as part of explicit integrity-incident repair. +Elapsed time alone never resolves a pending stage. diff --git a/arch_docs/streaming-poc-docs/spec/core-lang-protocol.md b/arch_docs/streaming-poc-docs/spec/core-lang-protocol.md new file mode 100644 index 000000000..8324f8d05 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/core-lang-protocol.md @@ -0,0 +1,458 @@ +--- +doc_id: EWS-SPEC-PROTOCOL +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: core-language-protocol +related_adrs: [ADR-008, ADR-013, ADR-041] +--- + +# Core/lang protocol + +Every message, call, and piece of Core state the feature adds. Names are internal to the Core +bridge. + +## Responsibility split + +**Python owns:** backend connection, authentication, reads, retries, and subscriptions; payload +serialization and delivery to Workflow code; stable stream offsets and write-fence interpretation; +background readiness watchers and readiness coalescing; the backend side of the race-free +park/wakeup handshake; creation and interpretation of opaque replay annotations, including +cross-stream delivery order; exact-offset reads and integrity checks during replay; deterministic +logical output framing; output staging and lazy promotion or abort of pending stages. + +**Core owns:** serializing stream readiness with all other Workflow inputs; tracking deterministic +external wait IDs and generations; holding a Workflow Task open while Workflow code is quiescent; +one global quiescence timeout and one output-visibility deadline per open Workflow Task, plus +Workflow Task rollover timing; issuing +activations; coordinating atomic parking of the complete external wait set, marker creation, and +Workflow Task completion; returning recorded annotations to Python during replay; intercepting the +reserved wake Signal and suppressing it from user Signal handlers. + +**Core treats the replay annotation as opaque bytes.** It does not understand stream offsets, +payloads, backend types, or codecs. Stream records never pass through `sdk-core`, Signals, or +Temporal History. + +## Completion commands + +Added to the `WorkflowCommand.variant` oneof in `workflow_commands.proto`. Tags 1–22 are in use; +new variants take 23–28. + +```protobuf +// Tag 23. Commits an observation delta for external streams. Emitted on every +// completion path where replay-visible stream state changed -- which includes +// an activation that observed no records at all -- independent of whether the +// WFT is retained. Ordered before any command whose value depends on consumed +// data. +message WorkflowStreamProgress { + bytes observation_delta = 1; + // Set when the encoder is approaching the annotation byte budget and the + // runtime wants the WFT rolled over rather than the marker grown further. + bool request_rollover = 2; +} + +// Tag 24. Reports the complete quiescent snapshot. Carries no annotation data. +// Registration and retention are separate: Core always records the set, and +// holds the WFT open only where the completion can be held open at all. +message WorkflowStreamQuiescent { + uint64 quiescence_generation = 1; + repeated ExternalStreamWait waits = 2; + google.protobuf.Duration idle_timeout = 3; +} + +message ExternalStreamWait { + uint32 wait_id = 1; + uint64 generation = 2; + bool immediately_parkable = 3; +} + +// Tag 25. Python's answer to PrepareExternalStreamPark. +message ExternalStreamParkResult { + uint64 quiescence_generation = 1; + oneof outcome { + ParkSetConfirmed confirmed = 2; + StreamSetBecameReady became_ready = 3; + } + // Terminal observation delta for the marker Core is about to write. Present + // on `confirmed`: only Python can encode the boundary, and Core must not + // write a marker whose annotation has no terminal. + bytes final_observation_delta = 4; +} + +// Tag 26. Python's answer to FinalizeExternalStreams, for the paths where Core +// decides the boundary and no park handshake runs. +message ExternalStreamFinalized { + uint64 quiescence_generation = 1; + bytes final_observation_delta = 2; +} + +// Tag 27. Reports that output for this Workflow Task is durably staged. +message WorkflowOutputStreamCommit { + ExternalOutputStreamManifest manifest = 1; + // Capacity alone requests an immediate replacement Workflow Task. + bool request_rollover = 2; +} + +// Tag 28. Reports unstaged output retained in lang and arms the earliest +// visibility deadline for the current Workflow Task. +message WorkflowOutputStreamBuffered { + google.protobuf.Duration max_publish_latency = 1; +} +``` + +`ExternalStreamWait` is a shared message, not a oneof variant, and takes no command tag. +`quiescence_generation` identifies the complete blocked snapshot. + +`immediately_parkable` is normally set after a write fence. A single fenced stream does not park the +Workflow Task; Core bypasses the idle delay only when **every** active external wait in the quiescent +snapshot is immediately parkable. + +New commands must be added to the `WorkflowCommand` conversion (`TryFrom for +WFCommand`) and accepted by `validate_completion`. + +## Activation jobs + +Added to the `WorkflowActivationJob.variant` oneof in `workflow_activation.proto`. New variants +start at 17. + +```protobuf +message ResolveExternalStreamWaits { + uint64 quiescence_generation = 1; + repeated ExternalStreamWait ready_hints = 2; +} + +message PrepareExternalStreamPark { + uint64 quiescence_generation = 1; + repeated ExternalStreamWait waits = 2; + ParkReason reason = 3; // IDLE, ALL_WRITE_FENCED, SHUTDOWN +} + +message ReplayExternalStreams { + uint64 quiescence_generation = 1; + repeated ExternalStreamWait waits = 2; + bytes replay_annotation = 3; + ParkReason terminal_boundary = 4; +} + +// Issued before Core writes a marker for a boundary Core itself decided and no +// park handshake will run: rollover-deadline expiry, and shutdown or eviction +// with active waits. Runs no user Workflow code. +message FinalizeExternalStreams { + uint64 quiescence_generation = 1; + repeated ExternalStreamWait waits = 2; + ParkReason reason = 3; // ROLLOVER, SHUTDOWN, OUTPUT_LATENCY +} +``` + +`ResolveExternalStreamWaits` contains no records. The listed waits are readiness **hints**, not an +exhaustive availability claim. On receipt, Python probes every active wait, drains all currently +available inputs, and then resumes Workflow futures. + +`PrepareExternalStreamPark` and `FinalizeExternalStreams` are runtime-internal and always cover the +complete active wait set. Python handles both without invoking user Workflow code — +`PrepareExternalStreamPark` may find new records on its final recheck, in which case it returns +`StreamSetBecameReady` and Core issues a normal resolve activation rather than running user code +from inside the park path. + +**Two** of these four jobs — `ReplayExternalStreams` and `PrepareExternalStreamPark` — require input +backend I/O. The `FinalizeExternalStreams` handler itself requires none (ADR-010). When its reason is +`OUTPUT_LATENCY`, however, the Worker's outer async layer stages the buffered output before it sends +the existing finalization response and output commit. All three runtime-internal jobs are dispatched +outside the synchronous Workflow thread; see `python-runtime.md`. + +## Marker envelope + +In `external_data.proto`, alongside the existing `LocalActivityMarkerData` and `PatchedMarkerData`: + +```protobuf +message ExternalStreamMarkerData { + uint32 schema_version = 1; + uint64 quiescence_generation = 2; + repeated ExternalWaitMarker waits = 3; + bytes replay_annotation = 4; + ParkReason terminal_boundary = 5; + ExternalOutputStreamManifest output = 6; +} + +message ExternalWaitMarker { + uint32 wait_id = 1; + uint64 generation = 2; +} + +message ExternalOutputStreamManifest { + uint32 schema_version = 1; + uint32 fingerprint_version = 2; + string stage_token = 3; + int64 history_floor_event_id = 4; + string run_id = 5; + repeated ExternalOutputTopicManifest topics = 6; + repeated ExternalOutputSegmentManifest segments = 7; + string provider_id = 8; + uint32 provider_format_version = 9; +} + +message ExternalOutputTopicManifest { + string topic = 1; + uint32 record_count = 2; + uint64 logical_byte_count = 3; + bytes logical_fingerprint = 4; + bool finished = 5; +} + +message ExternalOutputSegmentManifest { + repeated uint32 record_counts_by_topic = 1; +} +``` + +The topic manifests contain logical counts, bytes, fingerprints, and finished flags; segment entries +contain positional per-topic record counts. No output payload enters History. `ParkReason` lives here +and only here — see "Who finalizes the annotation" below. + +During replay, Core performs marker lookahead before resolving the matching external wait set, +analogous to local-activity marker lookahead: `Replaying --(Schedule)--> +WaitingResolveFromMarkerLookAhead`, then `--(HandleKnownResult)--> +ResolvedFromMarkerLookAheadWaitingMarkerEvent`, settled by `MarkerRecorded`. + +## Who finalizes the annotation + +The annotation ends with a terminal — the blocked cursor snapshot — and **only Python can encode +it**. Core is annotation-blind by design, so Core cannot manufacture a terminal, and Python cannot +encode a boundary it was never asked to finalize. Several completion paths are decided *inside +Core*, with no Python activation outstanding at the moment of decision: + +| Path | Who decides | How the terminal is obtained | +|---|---|---| +| Idle-timeout park | Core timer | `PrepareExternalStreamPark` → `ExternalStreamParkResult.final_observation_delta` | +| All-fenced immediate park | Core, from the quiescence snapshot | same | +| Rollover-deadline expiry | Core timer | `FinalizeExternalStreams{ROLLOVER}` → `ExternalStreamFinalized` | +| Byte-budget rollover | Python, via `request_rollover` | already carried by the triggering `WorkflowStreamProgress` | +| Output-visibility deadline | Core timer | `FinalizeExternalStreams{OUTPUT_LATENCY}` → `ExternalStreamFinalized`, with the staged `WorkflowOutputStreamCommit` on the same completion | +| Output logical-capacity rollover | Python | `WorkflowOutputStreamCommit{request_rollover=true}` already carries the staged manifest | +| Command-producing completion | Python | already carried by the completion's `WorkflowStreamProgress` | +| Terminal Workflow completion | Python | same, ordered before the terminal command | +| Query answered while stream waits are still registered | Core, by refusing retention to report the answer | `FinalizeExternalStreams` → `ExternalStreamFinalized` | +| Worker shutdown / eviction, Workflow Task open | Core | `FinalizeExternalStreams{SHUTDOWN}` → `ExternalStreamFinalized`. If Python cannot answer, **no marker is written** and the Workflow Task fails for retry | +| Worker shutdown / eviction, no open Workflow Task | — | No marker exists to write; see `wft-lifecycle.md` | + +An idle or all-fenced park with buffered output is a single terminal, not two. Python stages first +and orders `WorkflowOutputStreamCommit` before the confirmed `ExternalStreamParkResult`; Core writes +one shared marker and does not force a replacement. If the park recheck becomes ready, Python sends +`WorkflowOutputStreamBuffered` before `became_ready`, preserving the already chosen earliest output +deadline. ADR-045 defines the case where that deadline wins while preparation is outstanding. + +**The rule that makes this coherent, and that has no exceptions:** Core never writes a marker for a +boundary it decided without first receiving a terminal from Python (ADR-008). If a terminal cannot +be obtained, Core writes nothing and the Workflow Task is retried, because an abandoned Workflow +Task commits no cursor and loses no record, while a truncated annotation is durable and wrong. + +`FinalizeExternalStreams` is a runtime-only activation job in the same class as +`PrepareExternalStreamPark` — it runs no user Workflow code, it cannot resolve futures, and its only +legal responses are `ExternalStreamFinalized` or an activation failure. + +`ParkReason` therefore lives in exactly one place: the Core-readable +`ExternalStreamMarkerData.terminal_boundary`. Core knows the reason in every row above — it either +decided it or received it — so duplicating it inside the opaque annotation's terminal would add a +second copy that could disagree with the first. + +## Readiness and status calls + +The bridge exposes two acknowledged, thread-safe calls: + +```text +notify_external_stream_ready(run_id, wait_id, wait_generation) + -> Accepted | Stale | Parked | NoOpenWorkflowTask | RunNotFound + +external_stream_run_status(run_id) + -> WftOpen | Parked | NoOpenWorkflowTask | RunNotFound +``` + +The second is **read-only** and has no effect on the Run: it exists so the shutdown sweep can ask +what state a Run is in without claiming a record is buffered, which is what `notify_…_ready` means. +Both are answered on the same serialized local-input lane, so a status answer is as authoritative as +a readiness acknowledgement. Core already answers a read-only question on that lane through +`GetStateInfoMsg`; `external_stream_run_status` is the same shape, scoped to one Run's external wait +set. + +`Accepted` means Core serialized readiness while the Workflow Task was still open. It cancels the +global quiescence timer or logically aborts an in-progress park, then issues or augments a readiness +activation. Core coalesces all wait IDs known to be ready before shipping that activation; +notifications received while an activation is outstanding are accumulated for the next one. **There +is never more than one outstanding activation per Run.** + +The other four results all mean local readiness could not be delivered, and Python sends the wake +Signal for the last three (ADR-013): + +| Result | Meaning | Watcher action | Metric | +|---|---|---|---| +| `Accepted` | Readiness was serialized into an open WFT | Nothing further; Core will activate | local wakeup | +| `Stale` | The wait exists but its `wait_generation` moved on | Re-probe; do not signal | stale notification | +| `Parked` | A confirmed `park_generation` exists for this wait | Send the reserved wake Signal | signal wakeup, parked | +| `NoOpenWorkflowTask` | The Run is cached and its waits are registered, but no WFT is open | Send the wake Signal; **keep** the watcher | signal wakeup, unparked | +| `RunNotFound` | The Run is absent from this Core worker's cache | Send the wake Signal, then tear the watcher down | signal wakeup, evicted | + +`Stale` is the one answer that is re-asked rather than acted on, and **the re-ask's answer is the +one that counts**: a report can race a wait-generation change and be answered `Stale` while the Run +is evicted before the delayed retry that follows, so the retries can legitimately end on a different +row of this table than the first report did. Keeping the original `Stale` there sends the owed wake +and skips the teardown the row it actually landed on requires — leaving a watcher, a buffer, a +backend read loop and a Run-map entry alive for a Run this Worker no longer owns. `RunNotFound` also +ends the retries rather than using them up: the Run cannot come back, and each further attempt only +delays the wake the buffered record still needs. + +`NoOpenWorkflowTask` is the healthy state between Workflow Tasks after a command-producing +completion or a rollover, and is not an error. The three signal-sending results are distinguished by +what the watcher does *afterwards* and by what an operator should conclude, not by whether a Signal +is sent. + +`Accepted` is a promise, and it is the only result that leaves nobody but Core holding the record: +the watcher does nothing further, and it will not report that record again. Core therefore stops +answering `Accepted` the moment a completion is prepared for the server rather than retained -- +readiness from that instant is `NoOpenWorkflowTask` and the watcher signals for itself -- and a +completion reported while readiness is still pending requests a replacement Workflow Task, which is +where the resolve job for it is issued. Both halves are ADR-041. Without them a readiness accepted +against a task that is about to be reported is stranded: no activation is issued for it, no Signal +is owed for it, and the buffered record waits behind a Run that Core believes it has already told. + +The public entry point goes on `Worker`, following the shape of `record_activity_heartbeat`. Routing +into the workflow stream follows `notify_local_result` → `notify_of_local_result`. Unlike +`notify_local_result`, the stream call must return an acknowledgement rather than being +fire-and-forget. + +## Core state + +`ManagedRun` tracks multiple external waits as one quiescent set rather than adding stream state to +`LocalActivityManager`: + +```rust +struct ExternalWaitState { + wait_id: u32, + wait_generation: u64, + status: ExternalWaitStatus, + immediately_parkable: bool, +} + +enum ExternalWaitStatus { + BlockedWftOpen, + Ready, + Parking, + Parked, +} + +struct ExternalWaitSet { + quiescence_generation: u64, + waits: HashMap, + ready: HashMap, + idle_timeout: Option, + park_generation: Option, + wft_open: bool, + replay_annotation: Vec, +} +``` + +### Generalizing WFT retention + +The per-Run concept is *local work that may retain the Workflow Task*. It generalizes the earlier +local-activity-only state: + +```rust +struct WaitingOnLocalWork { + local_activities: Option, + external_wait_set: ExternalWaitSet, + wft_rollover_timer: Option, + idle_timer: Option, + output_flush_timer: Option, + output_buffered: bool, + output_latency_pending: bool, + park_rollback_resolve_pending: bool, + pending_output_commit: Option, +} +``` + +The retention predicate is "local activities outstanding, the external wait set retains, **or +unstaged output is buffered under a retainable wait snapshot**". The generalization does not change +local-activity behavior. + +External waits remain logically pending after parking, but only a wait set containing +`BlockedWftOpen`, `Ready`, or `Parking` states retains the current Workflow Task. Core must not +complete the task merely because one member becomes idle or parkable. + +### Local input routing + +`enum LocalInputs` gains `ExternalStreamReady`, `ExternalStreamIdleTimeout`, and +`ExternalStreamParkResult`; the retained-output path also supplies +`ExternalOutputFlushDeadline`. +Each carries a run ID and must be added to the `LocalInputs::run_id()` match. These use the same +prioritized local-input lane as local-activity completions (`LocalInputs::LocalResolution`). + +## Idle timer + +The idle timer belongs to Core because its outcome determines whether the Workflow Task is retained +or completed. It measures **global** quiescence across all input streams, rather than the emptiness +duration of each stream independently. + +1. Python reports any `WorkflowStreamProgress` delta, then `WorkflowStreamQuiescent` with the + complete active wait set. +2. Core accumulates the delta and starts one timer using a monotonic clock. +3. Readiness accepted for any current wait cancels the timer and ends that quiescent snapshot. +4. Python drains available inputs and reports a new complete snapshot if the Workflow becomes + quiescent again; only then does Core start a fresh timer. +5. Timer expiry enters `Parking` for every wait and queues one `PrepareExternalStreamPark` + activation. +6. Core completes the Workflow Task only after Python confirms the complete backend park handshake. + +The timer is operational wall-clock state, not Workflow time. On replay, Core does not run it; the +marker reproduces the recorded idle boundary. + +The idle deadline and the Workflow Task rollover deadline are **separate**. The rollover deadline +derives from the same `wft_timeout` that drives the existing local-activity heartbeat, but it cannot +reuse that mechanism — see ADR-017. + +## Output deadline and park arbitration + +`WorkflowOutputStreamBuffered` is legal only when the completion has a retainable quiescent input +snapshot, no user/server command, and no output-capacity rollover. The first buffered output arms a +monotonic deadline. The default is 100 milliseconds; if later topics join the retained batch, Core +keeps the earliest deadline derived from their minimum configured latency and clamps it below +ordinary rollover. Replay arms no output timer. + +The output deadline, readiness, parking, ordinary rollover, and server-bound completion compete on +the same serialized state: + +- confirmed park: accept the staged commit before the confirmed park result and write one marker; +- readiness wins the park recheck: retain the output and preserve its original deadline; +- readiness wins after lang already staged for a confirmation: discard the stale park terminal, + resolve the abandoned intents, finalize with `TASK_COMPLETED`, record the carried commit once, and + force a replacement; +- output deadline wins during prepare: invalidate the losing quiescence generation, queue resolve so + Python rolls back every installed park intent, then queue + `FinalizeExternalStreams{OUTPUT_LATENCY}`, carry a commit returned by the losing prepare or stage + during finalization, and force a replacement task after the shared marker; +- late park or timer results: treat them as stale and never write another marker. + +Suppressing the output timer while a park is outstanding is incorrect: a stalled provider park +operation could otherwise violate the output visibility bound indefinitely (ADR-045). + +## Implementation locations + +The surface the feature occupies: + +**Core protobufs** (`crates/protos/protos/local/temporal/sdk/core/`): +`workflow_commands/workflow_commands.proto`, `external_stream/external_stream.proto` (the +`WakeSignal` envelope), `workflow_activation/workflow_activation.proto`, +`external_data/external_data.proto`. + +`crates/sdk-core-c-bridge` carries the C-ABI surface used by non-Rust, non-Python SDKs; the +acknowledged readiness call is added there when those SDKs adopt the feature. Python does not go +through it — it uses its own pyo3 bridge. + +**Core orchestration:** `workflow_stream.rs` (LocalInputs), `managed_run.rs` (wait set, retention, +timers, finalization, shutdown transitions, the sink-independent timer facility), `workflow/mod.rs` +(command validation and conversion, readiness routing), `worker/mod.rs` (public entry points). + +**Core machines:** add `machines/external_stream_state_machine.rs` modeled on +`local_activity_state_machine.rs`; extend `machines/workflow_machines.rs` to register waits, +accumulate annotations, issue markers, and perform replay lookahead. Keep the stream provider out of +`LocalActivityManager`; only the scheduling pattern is shared. + +**Python SDK:** see `python-runtime.md`; output-specific modules are `_output_api.py`, +`_output_backend.py`, `_output_client.py`, `_output_codec.py`, `_output_continuation.py`, and +`_output_producer.py`, with Workflow staging coordinated by `_runtime.py` and `worker/_workflow.py`. diff --git a/arch_docs/streaming-poc-docs/spec/failure-taxonomy.md b/arch_docs/streaming-poc-docs/spec/failure-taxonomy.md new file mode 100644 index 000000000..691ca02ce --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/failure-taxonomy.md @@ -0,0 +1,184 @@ +--- +doc_id: EWS-SPEC-FAILURE +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: failure-classification-and-operator-response +related_adrs: [ADR-014, ADR-015] +--- + +# Failure taxonomy + +Four distinct outcomes, deliberately not collapsed — each has a different operator response. + +## One channel, four meanings + +All four outcomes are expressed through the same channel, because a Workflow activation completion +is a `oneof status { Success | Failure }` — a completion cannot both fail the Workflow Task and emit +a terminal command, and `Failure` carries only a `failure` and a `force_cause`. + +**There is no protocol-level "non-retryable Workflow Task failure": the server retries Workflow Task +failures regardless of cause.** What differs between the rows is the error surfaced to the user and +to metrics, and the operator response — not whether the task is retried. + +| Condition | Completion | Server behavior | Operator response | +|---|---|---|---| +| Backend unreachable, timing out, or erroring on replay | `Failure`, `force_cause = WORKFLOW_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE`, **`StreamStorageError`** error type, its own metric | Retried | None — clears when the backend recovers | +| Recorded offset or staged output missing, expired, reordered, or conflicting — i.e. deletion, trimming, retention expiry, or an impossible stage identity | `Failure`, same `force_cause`, **`StreamIntegrityError`** error type, distinct metric | Retried, and keeps failing | Repair or restore the backend, or terminate the Run | +| Record bytes present and intact, but the DataConverter or codec cannot decode them | `Failure`, same `force_cause`, **`StreamDecodeError`** error type, its own metric | Retried, and keeps failing | Align the consumer's converter/codec with the producer's — the stream is fine | +| Recorded annotation does not match the subscriptions Workflow code creates | `Failure` as a nondeterminism error, same class as a command mismatch | Retried, and keeps failing | Fix or version the Workflow code | + +Rows two through four are distinguished from row one by **error type and metric, never by the retry +behavior of the Workflow Task**. Alerting must key on those metrics rather than on Workflow Task +failure counts, which row one also increments. + +## The classification rule + +Mechanical and, since structural immutability is required of every provider, unconditional: + +> **If the range validated, the bytes are the bytes that were written**, so any subsequent decode +> failure is row three. Only a missing offset or a range that fails validation is row two. + +Row three is separated from row two because a converter mismatch is a configuration error on the +*consumer*, and reporting it as stream integrity loss sends an operator to restore a backend that was +never damaged (ADR-015). + +Row four covers every way a marker and the code replaying it disagree about the wait set: a wait the +marker bound that the Workflow no longer creates, a wait whose recorded stream is not the one the +code now subscribes, a recorded delivery the replay never took, and a stream name that disagrees +with what the predecessor Run committed for that `wait_id` across a Continue-As-New. +What is *not* row four is a subscription the annotation has no binding for: replay runs the Workflow +past the Workflow Task the marker covers, so that is the ordinary case rather than a disagreement +(`annotation-format.md`). Core-side, a marker found by lookahead with no matching state machine is +handled as local activities already handle that case. + +Nor is the provider half of either comparison row four. Configuring another implementation or +another format version leaves the Workflow unchanged and both stores undamaged, so it is row one — +retried, and cleared by a Worker carrying the recorded implementation. +That split is the same one whether the recorded binding came from a marker or from a Continue-As-New +header, because it is a claim about what the deployment registered rather than about what the +Workflow asked for (`annotation-format.md`). + +## What connects a row to a completion + +A row is only operator-distinct if something sets the cause and increments the counter, and the +three failing rows arrive at a completion by two different routes. A runtime-only job — a replay +read, a park — raises on the Worker's loop, where an exception becomes a failed completion. A +delivery raises on the **Workflow thread**, where `activate()` converts it into a failed +completion before the async layer ever sees it; out there no exception exists at all. + +So **reporting inspects the failed completion, never an exception.** Every failed completion is +matched by the application failure's type name against the taxonomy, walking the cause chain +because the raising frame may have wrapped it. A match sets +`force_cause = WORKFLOW_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE` and increments **that row's +counter and no other**; no match leaves both untouched, which is what keeps row four and ordinary +Workflow bugs out of these series. Keying on an exception instead would report the runtime-only +route and silently drop every delivery failure — most of rows two and three. + +The type *name* is therefore the seam, so the mapping from name to counter lives with the error +classes rather than at the reporting site: split apart, the two drift and a renamed class stops +being counted without anything failing. + +Classification itself happens at **both halves of the decode split** (`python-runtime.md`), because +that is where a record can fail: + +- A **preparation** failure is classified on the Worker's loop, carried on the record, and raised by + the delivery that would have yielded its value. +- A **conversion** failure is classified on the Workflow thread, where the record would have become + a value. + +Both resolve to row three rather than row two, and for the same reason in either delivery path: +replay validated the recorded range before preparing it, and live delivery read the record out of a +backend that guarantees a record's bytes cannot change once written (ADR-003). Either way the range +validated, so the rule above resolves to a decode failure. A cause that is *already* one of these +error types keeps its own row rather than being relabelled — a storage failure surfacing out of +external payload retrieval is row one, not the consumer's converter mismatch. + +### An unreachable external payload store is row one + +That last sentence is a rule about causes that arrive already labelled, and **nothing labels the one +case it names unless the preparation step does it.** With external storage configured, a record's +bytes are a *reference*: the stream holds a claim and the value does not exist until the payload store +hands it over. A driver that cannot reach the store raises whatever its client raises — an ordinary +`ConnectionError` — and the `DataConverter` does not wrap it. The consumer then sees an unclassified +exception on a record whose range validated, and the rule above turns it into row three: an operator +is told to align a converter that was never wrong, during a transient outage that clears itself. + +So the label is applied where the knowledge is, at the retrieval call inside the preparation step, +and only around that call — the user's `PayloadCodec`, which runs immediately after it, keeps row +three, because a codec that rejects intact bytes *is* the configuration mismatch. Fetching a +reference is as much a stream read as reading the stream is; both are transient and both clear when +the thing behind them recovers. + +## Integrity loss blocks the Workflow + +The externally visible result of integrity loss is a **blocked Workflow**: + +> The Workflow Task fails repeatedly with a distinct failure cause and a non-retryable error message, +> and the Workflow makes no further progress until an operator repairs the backend or terminates the +> Run. + +It does **not** terminate the Workflow (ADR-014). Retention loss is usually an operational error and +usually repairable; a blocked Workflow can be resumed after repair, a failed one cannot. + +"Non-retryable" therefore describes the error surfaced to the user and to metrics, not the Workflow +Task, which the server retries regardless. The distinction from a transient backend outage is in the +error type and the metric, not in whether the task is retried; an outage clears on its own and +integrity loss does not. **Operators are expected to alert on the integrity metric specifically.** + +**There is no terminal-failure opt-in.** See ADR-014 for why one is not implementable. + +**Integrity loss must never resolve to an alternate stream result.** + +## Additional failure rules + +- Backend failure during a live activation fails the Workflow Task with the external-storage failure + cause; an uncommitted cursor is retried from the preceding marker. +- Output staging failure before completion also fails the Workflow Task with the external-storage + cause. No commit command or marker is sent, and retry reuses the saved stage identity. +- A provider stage conflict, a missing staged record, a changed immutable manifest, or History + retention loss needed to resolve a pending stage is `StreamIntegrityError`. These conditions are + never guessed through. +- A pending output stage is an ordering barrier. If neither its exact token commit nor a later exact + Workflow Task boundary is yet visible, it remains pending. Backend or Temporal outages during + reconciliation are `StreamStorageError`; elapsed time is not an abort predicate. +- A committed output payload that cannot be decoded is `StreamDecodeError`, under the same + range-validated rule as input. This may surface to an external output client rather than through a + Workflow Task completion. +- Watcher failures are retried within provider policy. If readiness can no longer be monitored + safely, Python reports the failure to Core rather than allowing the WFT to time out silently. +- A worker crash discards uncommitted reads. Replay re-reads from the last committed marker. +- Stale readiness notifications are ignored by wait and quiescence generation. +- No cursor advances unless the marker commits the complete cross-stream annotation. +- An unacknowledged shutdown wake is reported through the + `temporal_external_stream_shutdown_wake_failed` metric rather than retried indefinitely or + assumed delivered. +- **Two errors sit deliberately outside the four rows**, because both describe Workflow code asking + for something the design does not offer rather than a stream or a converter behaving unexpectedly at + read time, and both are deterministic — so a Workflow Task failure would be retried into the + identical failure forever with nothing durable to show why. Each is a non-retryable + `ApplicationError` raised where the request is made: `ExternalStreamCapacityError` at `subscribe()`, + for a subscription set an annotation cannot carry (`annotation-format.md`), and + `ConcurrentStreamConsumerError` at the wait, for a second coroutine blocking on one subscription + (`python-runtime.md`). + +`OutputAppendNotAcknowledgedError` is also outside the four Workflow-Task rows. It is the direct +producer's explicit statement that an append outcome is unknown. The caller must retry +`resolve_append()` with the exact captured record; calling `publish()` again would allocate another +sequence and could duplicate the value. + +## Metrics summary + +| Metric | Fires on | +|---|---| +| local wakeup | readiness `Accepted` | +| stale notification | readiness `Stale` | +| signal wakeup, parked | readiness `Parked` | +| signal wakeup, unparked | readiness `NoOpenWorkflowTask` | +| signal wakeup, evicted | readiness `RunNotFound` | +| `temporal_external_stream_storage_failure` | row one | +| `temporal_external_stream_integrity_failure` | row two | +| `temporal_external_stream_decode_failure` | row three | +| `temporal_external_stream_shutdown_wake_failed` | a shutdown wake unacknowledged when the grace period expires | + +Row four has no counter, by construction: it is ordinary nondeterminism, and giving it one would put +a Workflow bug in a series an operator alerts on to decide whether to touch a backend. diff --git a/arch_docs/streaming-poc-docs/spec/public-surface.md b/arch_docs/streaming-poc-docs/spec/public-surface.md new file mode 100644 index 000000000..10fab3751 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/public-surface.md @@ -0,0 +1,130 @@ +--- +doc_id: EWS-SPEC-PUBLIC +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: public-names-roles-and-feature-boundary +related_adrs: [ADR-001, ADR-019, ADR-021, ADR-037, ADR-040, ADR-048] +--- + +# Public surface and feature boundary + +The names, handle roles, configuration boundary, and user-visible distinctions of External Workflow +Streams. Detailed lifecycle and provider behavior belongs to the linked subsystem specifications. + +## Feature boundary + +External Workflow Streams coexist with `temporalio.contrib.workflow_streams`; they do not replace or +extend that package's storage model. + +| Concern | `contrib.workflow_streams` | External Workflow Streams | +|---|---|---| +| Data storage | Append-only Workflow state represented in History | Pluggable external backend; payloads stay outside History | +| Workflow role | Produces for external clients | Consumes external input and/or produces external output | +| Input wake transport | Not applicable | Local readiness or a payload-free reserved Signal | +| Replay position | Dense offsets in Workflow state | Opaque provider boundaries and compact external-stream markers | + +The public module is `temporalio.contrib.external_workflow_streams`. No public or reserved name in +this feature may begin with `__temporal_workflow_stream`; that prefix belongs to the existing package +(ADR-001). + +Live Workflow interaction and marker-backed replay are one indivisible capability. The public +surface has no live-only mode that reads or publishes external records without recording the +corresponding input cursor boundaries or output commit manifest. Without that durable boundary, a +Workflow Task retry, eviction, or Worker restart could not distinguish work already observed from +work that had not yet been delivered or committed. + +## Names and roles + +| Role | Public entry point or type | Operations | +|---|---|---| +| Workflow input namespace | `external_stream` | `with_options(...)`, `topic(...)` | +| Workflow input topic | `ExternalStreamTopic` | `subscribe()` | +| Workflow input subscription | `ExternalStreamSubscription` | Async iteration, `close()` | +| Workflow input merge helper | `merge` | `merge(*subscriptions)` | +| External input producer connection | `ExternalStreamProducer` | `connect(...)`, `topic(...)` | +| External input producer topic | `ExternalStreamProducerTopic` | `publish(...)`, `finish_writing()`, append/wake recovery | +| Workflow output namespace | `external_output_stream` | `with_options(...)`, `topic(...)` | +| Workflow output topic | `ExternalOutputStreamTopic` | `publish(...)`, `finish()` | +| Direct output producer connection | `ExternalOutputStreamProducer` | `connect(...)`, `topic(...)` | +| Direct output producer topic | `ExternalOutputStreamProducerTopic` | `publish(...)`, `finish_writing()`, append recovery | +| External output reader connection | `ExternalOutputStreamClient` | `connect(...)`, `topic(...)` | +| External output reader topic | `ExternalOutputStreamClientTopic` | `subscribe(...)` | +| External output item | `ExternalOutputStreamItem` | decoded data plus opaque resume offset | + +Consumer-side and producer-side topics are distinct types. A Workflow topic is bound to the running +Workflow and the Worker's configured provider. A producer topic is bound explicitly to credentials, +Workflow chain identity, serialization context, and a retry-stable session. No topic handle is passed +from Workflow code to an Activity or external process as a backend capability (ADR-019). + +## Worker configuration + +The Worker option is `external_stream_backend`. Its instance lives outside the Workflow sandbox and +Workflow code cannot select or access its connection. + +`StreamBackend` and `OutputStreamBackend` are independent provider capabilities: + +- an input-only provider remains valid; +- an output-only provider can serve direct output producers and external output clients; and +- Workflow-originated output requires the Worker's configured provider to implement both contracts + in the first release. + +Exact validation and operations: [`backend-contract.md`](backend-contract.md). Sandbox ownership and +dispatch: [`python-runtime.md`](python-runtime.md). + +## Stream identity + +A stream is identified by namespace, Workflow ID, first execution Run ID, direction, and topic name. +Direction physically isolates input and output topics with otherwise identical components. The first +execution Run ID is stable across Continue-As-New and isolates later reuse of the same Workflow ID. + +Producer connections take the Workflow chain key; `topic(name)` adds the name exactly once. The +producer verifies the chain identity before its first append. Physical encoding must be injective for +all components; see [`backend-contract.md`](backend-contract.md). + +## Input behavior visible through the API + +- `external_stream` uses an `idle_timeout` of one second by default. + `with_options(idle_timeout=...)` configures the value carried by subscriptions created from those + options. Values must be positive; when a Workflow blocks on several subscriptions, their values + reduce deterministically by `min` before Core starts the one timer for the complete wait set + (ADR-016). +- `subscribe()` creates a new subscription with its own wait identity and cursor. +- Multiple subscriptions to one topic receive broadcast delivery; they do not form a work-sharing + group (ADR-021). +- One subscription permits one active consumer. A second waiter is rejected at the waiter rather + than corrupting cursor state (ADR-037). +- `finish_writing()` appends an ordered write fence for one producer session. It does not close the + input stream and does not prevent later producers or later records (ADR-040). +- Cancelling or closing a subscription does not delete already recorded state or end the underlying + topic. + +Exact retention, parking, and cancellation behavior: [`wft-lifecycle.md`](wft-lifecycle.md) and +[`python-runtime.md`](python-runtime.md). + +## Output behavior visible through the API + +- Workflow `publish()` buffers a logical record. It never returns or exposes the eventual provider + offset to deterministic Workflow code. +- Workflow `finish()` and direct-producer `finish_writing()` append the ordered `FINISH` terminal. +- Direct `finish_writing()` closes that producer's topic handles but does not write the Workflow's + Continue-As-New state. Mixed publishers must designate one terminal owner. +- Workflow closure, failure, cancellation, termination, and Continue-As-New do not synthesize + `FINISH` (ADR-048). +- External readers receive committed records only and resume strictly after an opaque boundary. +- No global order is promised across topics or between concurrently produced Workflow and direct + output. + +Exact staging and visibility behavior: [`backend-contract.md`](backend-contract.md), +[`wft-lifecycle.md`](wft-lifecycle.md), and [`annotation-format.md`](annotation-format.md). + +## Reserved wake name + +The input wake Signal is `__temporal_external_stream_wake`. It is protocol metadata rather than a +user payload and is intercepted before user Signal dispatch. Its exact envelope and request-ID rules +are defined in [`wake-signal.md`](wake-signal.md). + +## Compatibility status + +The feature is pre-production. Annotation, continuation, and wake encodings have one current format +and no legacy compatibility mode. Unsupported must-understand versions fail explicitly rather than +silently changing replay or continuation behavior. diff --git a/arch_docs/streaming-poc-docs/spec/python-runtime.md b/arch_docs/streaming-poc-docs/spec/python-runtime.md new file mode 100644 index 000000000..39817b194 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/python-runtime.md @@ -0,0 +1,474 @@ +--- +doc_id: EWS-SPEC-PYTHON +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: python-worker-runtime +related_adrs: [ADR-010, ADR-011, ADR-026, ADR-028, ADR-029, ADR-034, ADR-035, ADR-037] +--- + +# Python runtime: the out-of-sandbox manager + +How stream I/O happens without ever touching the Workflow thread, and which component owns which +position in a stream. + +## The constraint that shapes everything else + +Workflow activations cannot perform backend I/O. This is a property of the existing Python Worker, +not a stylistic preference: `_WorkflowInstanceImpl.activate()` is synchronous, runs on a thread-pool +executor under a **2-second deadlock timeout**, and drives a custom deterministic event loop in +which real network I/O cannot be awaited at all. **Any design in which `_apply` reads the backend is +wrong by construction, not merely slow.** + +Everything below follows from it. The subscription manager lives outside the sandbox on the Worker's +own asyncio loop and owns the backend connection and every watcher task; the Workflow thread only pops +from a bounded per-subscription buffer; readiness is reported to Core **only after** a record is +buffered, which is what makes the activation it produces guaranteed non-blocking; and the activation +jobs that need backend work are answered before `activate()` is ever called. A backend slower than +the deadlock timeout therefore delays readiness rather than failing an activation. + +Only an opaque handle crosses the sandbox boundary — the manager module is registered for sandbox +passthrough, and no provider instance is reachable from Workflow code. Every topic uses the one +backend configured on the Worker. + +## Decoding is split across the boundary + +"Pops from a bounded buffer" is only true if popping is the whole of getting a value, and +`DataConverter.decode` is not: it is external-payload retrieval, then the user's `PayloadCodec`, +then `from_payloads`. The first two are arbitrary asynchronous work — a network fetch, a KMS round +trip — and neither depends on the value's type. Only the third needs the topic's declared type, +and it is synchronous. The boundary falls between them: + +| Half | Runs on | Steps | Needs the type | +|---|---|---|---| +| preparation | the Worker's asyncio loop | external retrieval, then the user's `PayloadCodec` | no | +| conversion | the Workflow thread | `from_payloads` with the topic's type | yes | + +This is the split the Worker already applies to every other payload an activation carries: +`decode_activation` is awaited in `_handle_activation` before `activate()` reaches the executor. +Running the asynchronous half on the Workflow thread does not merely block — it **corrupts +History**, because the deterministic event loop turns a codec's `await` into a real Temporal timer +command, so History depends on the codec's internals and replays only while that codec behaves +identically. A codec that blocks rather than awaits fails against the sandbox's restrictions +instead, and either way arbitrary user I/O sits under the deadlock timeout (ADR-028). + +Preparation happens at exactly two points, which are the only two ways a record reaches Workflow +code: + +- **Live delivery** — in the watcher, before the record enters the subscription's buffer, and + *before* the epoch check a reposition invalidates. Preparing awaits, so a reposition landing while + a user's codec runs would otherwise slip past a check that had already passed and the retracted + records would be buffered anyway. +- **Replay** — over every segment of the replay plan, while the replay job is prepared out of + `activate()`. Replay's records never pass through a watcher, and a marker's segments are all + delivered inside one activation, so this is both the only point at which they can be prepared and + the last point before any of them is handed over. + +**A preparation failure is carried on the record rather than raised.** The watcher has no Workflow +Task to fail and no Workflow to tell, and raising there would end the watcher for the whole Run, +taking every later record with it. The error travels with the record and is raised by the delivery +that would have yielded its value — the only point at which a Workflow exists to be told, and the +only point at which the record is known to have been wanted. + +**A record that reaches the Workflow thread unprepared is refused, not converted**, whenever the +converter has a payload codec or external storage. The two cases are indistinguishable from the +bytes, because a codec's output is just another payload, so converting anyway yields a plausible +wrong value that nothing reports. A converter with neither has an empty asynchronous half and +converts correctly with no preparation at all, which is why the refusal keys on the converter rather +than on the delivery path. What it reports is a routing defect — some path delivered without +preparing — and not a user's converter mismatch. + +Nothing Worker-side crosses the boundary to make this work. The Workflow-facing API asks its opaque +handle for a codec bound to the topic's type and gets back the synchronous half only; the manager +runs the asynchronous half with no type at all, because the type belongs to the topic and the topic +is Workflow code. + +### Both halves carry the Workflow's serialization context + +A `DataConverter`'s components may implement `WithSerializationContext`, which is how a codec derives +an encryption key from the Workflow it serves. The Worker already binds a +`WorkflowSerializationContext(namespace, workflow_id)` around every payload an activation carries, on +both sides of this same split — `decode_activation` on its own loop, the Workflow instance's +converters on the Workflow thread. A stream record is a payload like any other and must arrive the +same way. Unbound, a record delivered in the **same activation** as the Workflow's own argument would +be converted under no context while the argument was converted under one, and a per-Workflow key +would decrypt nothing. The failure is not loud either: it is carried on the record, raised at +delivery, and classified as row three of `failure-taxonomy.md`, whose operator instruction is to +align the consumer's converter with the producer's — against a deployment where the two already +agree and the SDK is what dropped the context. + +Where the binding happens differs by holder, and follows from what each holder is: + +- **The runtime is per Run**, so it is built with a converter that is already bound and hands + Workflow code a codec carrying it. Bound by the Worker rather than inside the runtime, because the + runtime crosses into the sandbox while `with_context` runs user code to clone the component + converters — work that belongs on the Worker's side of the boundary, and that a per-Run handle + need do only once. + + **Except for a wait a marker bound**, which is handed a converter built from that wait's *recorded* + binding instead. The two halves would otherwise disagree about which Workflow the record belongs + to: the manager prepares a replayed record under the annotation's stream key, for the reason the + next bullet gives, while the runtime's own converter names the Run this Worker is executing. Those + are the same identity on a live Worker and are **not** the same under an offline `Replayer`, which + runs under its own placeholder namespace by default — so a converter keyed on the namespace decodes + a valid history to a different value, or refuses it outright. This is the accommodation + `_verify_binding` already makes for a replay harness's namespace, applied to conversion. + + Built on the Worker's loop before the replay job crosses the boundary, for the same reason as + above, and **not** discarded when the replay ends: delivery is not consumption, so a batch drained + during replay can be consumed over several later activations, and a record's context is a property + of where it was recorded rather than of the marker that happened to still be open. +- **The manager is per Worker** and prepares records for every Run at once, so a converter bound at + construction would be bound to nothing in particular. It derives the context per record from that + record's **own stream key**: the live path from the subscription's, replay from the annotation + header's, which is the only place a replayed record's stream is written down while the Workflow + has not yet run far enough to re-subscribe. The bound clones are memoized for the one preparation + call; kept on the manager they would accumulate an entry per Workflow ID the Worker ever served. +- **The producer is per chain** and binds once, at construction (`backend-contract.md`). + +All three key on `workflow_id`, never on any Run ID. A stream spans a Continue-As-New chain, so a +successor Run reads records its predecessor wrote; a Run-scoped key would make a chain's own records +unreadable from its first continuation onward. + +The binding is `with_context`, not `_with_contexts`. The latter additionally names the Workflow as +the payload's **storer**, and a stream record is stored by its producer rather than by the consuming +Run — nothing on the consumer's decode path reads that context anyway, since retrieval resolves the +driver from the claim embedded in the payload. `with_context` returns the converter unchanged unless +some component implements `WithSerializationContext`, so a default converter is untouched and the +ordinary deployment sees no change at all. + +**One store-context gap is open and stated rather than closed.** A producer that stores an +externally-stored payload does so with a store context naming no target, so a driver selector that +routes by target has nothing to route on. Retrieval is unaffected, for the reason above. Closing it +is a store-context question rather than a serialization-context one, and it needs a decision about +what target an Activity-hosted producer names — itself, or the Workflow the record is for — which +would change where blobs land. + +## Workflow output batching and staging + +`external_output_stream.topic(name, type=...)` gives Workflow code a publisher, never a backend +handle. `publish()` and explicit `finish()` run only the deterministic payload-converter half on the +Workflow thread, frame the logical record, and append it to per-Workflow-Task runtime state. They +perform no codec, external-payload-store, provider, Temporal-client, clock, or random operation. +Read-only Workflow contexts reject both operations. + +The runtime tracks topic order, finished topics, per-activation record-count vectors, and the minimum +record, logical-byte, and latency policy of every non-empty topic in the retained batch. Record and +logical-byte capacity is charged against canonical pre-codec frames for the whole Workflow Task. +When another publish would exceed it, that publish blocks, the current batch requests a capacity +rollover, and the replacement task releases the waiter. Replay ignores current live policies and +validates against History's manifest instead (ADR-046). + +After the synchronous activation returns, `worker/_workflow.py` owns the asynchronous half. It binds +the Workflow serialization context, applies the payload codec and external store, constructs one +immutable stage per topic under a shared opaque token, and calls the output provider. It saves the +logical attempt before the first await. A retry therefore repeats the same token and manifest even +when newly encoded bytes differ; the provider returns the first staged bytes as authoritative. +Provider conflict is integrity loss, while a transport or staging outage fails the Workflow Task as +external storage failure. + +The Worker then chooses among three completion shapes: + +- stage immediately and emit `WorkflowOutputStreamCommit` when no retainable quiescent input + snapshot exists, a user/server command is present, or capacity requests rollover; +- emit `WorkflowOutputStreamBuffered` without provider staging when input quiescence can retain the + task; or +- on `FinalizeExternalStreams{OUTPUT_LATENCY}` or a confirmed park, stage before answering and put + the finalization/park result and commit on the same completion. + +If a park recheck becomes ready, output remains logical and Python reports `Buffered` before +`became_ready`. If Core has already invalidated a park after output was staged, it carries that one +commit through resolve/finalization rather than staging again. `output_stage_recorded()` advances to +the next batch only after the compact commit command has been emitted. + +The reserved output Continue-As-New header is decoded before Workflow code runs and restores the +finished-topic set without a backend read. Its provider binding must match. Workflow completion does +not synthesize `FINISH`; only Workflow `finish()` or direct producer `finish_writing()` does +(ADR-048). + +Direct external processes and Activities use `ExternalOutputStreamProducer`, which appends committed +records under an idempotency session/sequence and requires retrying the exact record after an +ambiguous acknowledgement. `ExternalOutputStreamClient` reads the committed prefix from the backend; +only a pending head barrier makes it consult Temporal History for lazy exact-token reconciliation. +Normal reads and `tail()` are backend-only. + +## Four positions, one commit + +"Cursor" covers four different positions, owned by four different components, and conflating any two +of them is how a speculative read becomes a durable claim: + +| Position | Owner | Advances on | Survives eviction | What reads it | +|---|---|---|---|---| +| committed | the marker | a marker commit, and nothing else | yes — reconstructed from History | where replay starts | +| consumption | the runtime | a record handed to Workflow code | no | the Continue-As-New continuation | +| delivery | the runtime | a record drained out of the buffer | no | the observation delta, hence replay's recorded ranges | +| prefetch | the manager | a record read into the buffer | no — discarded outright | the watcher's next read | + +The three uncommitted positions run ahead of `committed` in that order and claim nothing durable: +reading a record is not consuming it, and consuming it is not committing it. On eviction, Workflow +Task failure, or Worker restart, every buffer and all three are discarded and the subscription +restarts from `committed` — which is exactly what makes "no cursor advances unless the marker +commits" safe to state. The manager may reposition `prefetch` *backwards* to `committed`, never +forwards past what a marker has committed, so a speculative read can never be mistaken for progress. + +### The reposition after a replay is synchronous + +While replay hands Workflow code the records a marker recorded, the watcher has been independently +reading those same records from the subscription's start cursor into the live buffer — nothing out +there knows the marker exists. So the end of a replay moves `committed` to the marker's terminal and +retracts the buffer to it, or the first live drain hands every replayed record over a second time. +Observed end-to-end as a Workflow that received `['alpha', 'alpha', 'beta']`. + +**That retraction has to have happened by the time the replay returns, not merely be scheduled.** The +drain that follows a replay is on the Workflow thread, and so is the replay itself; hopping the +retraction onto the manager's loop — as readiness re-arming legitimately does, because that only has +to happen eventually — orders it against nothing at all. One ordinary scheduler pass is enough for +Workflow code to drain first, and the fix then depends on the manager loop winning a race. + +Two things make the synchronous path safe from the Workflow thread: + +- **Only the watcher's wakeup is hopped.** Retracting touches the buffer, the three uncommitted + cursors, and the prefetch epoch, all under the subscription's lock and none of them an `asyncio` + primitive. The one asynchronous consequence — a watcher parked on backpressure now having room — is + an `asyncio.Event`, which may not be set from another thread, so that alone is posted to the loop. +- **The epoch check moved inside the append**, under the same lock the retraction takes. A watcher + compares the epoch it captured before its read against the one at append time, and with the + retraction now landing on another thread a check made outside the lock could pass and *then* have + the retraction land — putting the retracted records straight back into the buffer and re-advancing + `prefetch` past them. Under the lock, an append is either cleared by the retraction or rejected by + it, and there is no third interleaving. + +`delivery` and `consumption` differ by whatever a drained batch left unread: a Workflow that stops +iterating part-way through a batch was delivered the whole batch and consumed only its prefix. The +annotation records `delivery`, because that is the schedule replay must reproduce; the +Continue-As-New continuation carries `consumption`, because the buffer holding the difference dies +with the Run and a successor resuming from `delivery` would step silently over records Workflow code +never saw. + +## Delivery within one activation is bounded by a record count, and by annotation bytes + +One activation hands Workflow code at most `MAX_RECORDS_PER_ACTIVATION` records, and the +subscription then blocks **even though records are still buffered** — that block is what ends the +activation, because the watcher refills from the Worker's loop while the Workflow thread drains +(ADR-026). Without it the drain never finishes and the Workflow Task fails on the deadlock timeout +on every attempt, so the Workflow is stuck permanently rather than merely slowed. + +**The budget is charged where records move, and an activation begins already charged for what it +holds.** Both halves are what make the count a bound rather than a check, and each answers a +schedule that got past the other: + +- *Charged at delivery*, the drain that moves a batch out of the manager's buffer and into a + subscription's ready list — not at the consumption of each record afterwards. A drain that read the + budget and charged nothing is a reservation nobody made: the next subscription's drain saw the same + room and took it too, so two subscriptions consumed by two independent coroutines took a whole + budget each and `n` of them took `n`. `merge()` hid it by filling one record at a time, which + charges and checks in the same place. Delivery is also what the annotation records, so a count + charged at consumption bounded a different quantity than the segment it exists to bound. +- *Pre-charged with the carry-over*, because a batch is delivered whole and consumed one record at a + time. What a Workflow that stopped iterating part-way through leaves in its ready list is consumed + by the *next* activation with no drain, and so with no check of any kind — free, and it accumulates: + one subscription per activation can leave a nearly full list behind. Starting the count at what the + ready lists already hold makes what an activation may hand over — carried-over plus newly + delivered — one budget, whatever the schedule. + +Three consequences reach outside the runtime: + +- The segment that reaches the cap ends with `BATCH_LIMIT` in the annotation, so replay divides the + same records into the same activations (`annotation-format.md`). A time-based bound would cut the + segment at a nondeterministic point and diverge from the live run. +- A completion the budget stopped must **re-arm readiness** for the records left buffered. The + watcher moved `prefetch` past them when it buffered them, so no further notification is coming and + the Workflow would otherwise wait forever on data already in front of it. +- Those waits are blocked but **not immediately parkable**. They belong in the quiescent snapshot, + because that is what lets Core resolve them; declaring them parkable would ask Core to park a + Workflow Task whose records have already reached the Worker. + +The budget does not apply during replay, where delivery comes from the recorded segments, which +already fix how many records each activation received. + +A **second** budget sits beside it and the smaller of the two wins: how many more records this +Workflow Task's annotation can afford to record (`annotation-format.md`). It belongs here for the same +reason the record cap does — a record handed to Workflow code has to be recorded, a segment frame that +no longer fits cannot be moved to the next annotation, and there is no third option — so it is spent +before the record is delivered rather than discovered after the segment is built. Its three +consequences are the record cap's, with one difference each: the segment ends with `BUDGET_ROLLOVER` +rather than `BATCH_LIMIT`, and the same completion asks Core to end the Workflow Task rather than +merely the activation. Re-arming readiness and withholding immediate parkability are identical. + +## Merging several subscriptions + +`merge()` waits on several subscriptions at once and yields `(subscription, value)`. Each pass takes +**at most one record from each subscription, in `wait_id` order, resuming after the subscription +that last took one**, and each of those three is answering a different failure. + +*In `wait_id` order*, because that is what reproduces. Records that arrived in one batch across two +streams have no inherent order between them, so an interleaving that depended on dict iteration, on +arrival time, or on which watcher ran first would replay differently than it ran. `wait_id` comes +from Workflow code's own `subscribe()` order, so replay reconstructs the same total order from the +code, and the pass then depends only on which waits have a record ready — which replay reconstructs +from the recorded segments. + +*At most one record*, because the budget above covers the merged set as a whole. Draining one +subscription's ready list to the end lets the lowest `wait_id` spend the entire budget by itself, +which is a priority order rather than a merge. + +*Resuming after the last take*, because the budget covering the set means a pass can be cut anywhere +inside it, and a pass that always restarted at the lowest `wait_id` is cut in the same place on +every activation. That is not a subtle bias. With more ready subscriptions than the budget has +records — 257 against 256 — the last one is never reached at all, and it is not merely served +nothing: filling stops on the spent budget before it consults the manager, so the Worker is never +asked whether that stream has anything, for the life of the Run. The general case needs only a count +that does not divide the budget: 100 always-ready waits against 256 records leaves the pass cut at +the 56th, the first 56 taking one record per activation more than the other 44, forever, with the +gap growing by one per activation and nothing bounding it. Rotating the start position is what makes +the skew between any two continuously ready streams what it is claimed to be — a single record +(ADR-034). A control record spends its subscription's turn like any other: it is consumed, because +it occupies an offset inside a run, and it advances the rotation. + +**The rotation is not replay state.** It lives in the generator, is recorded nowhere, and is not +reconstructed — and under replay the budget does not apply, so passes are not cut where they were +cut live and the rotation reaches positions the live run never started from. What makes that safe is +that a replay drain serves from the front of the recorded segment and only while the front belongs +to the asking wait (`annotation-format.md`): a wait asked out of turn is told nothing and the record +stays for whoever asks next. Every active wait is still asked exactly once per pass, so the front's +owner is reached in every pass and the yielded sequence is the recorded global order whatever +position the pass started at. No change to *ask* order can reorder a replayed segment. That premise +is load-bearing rather than incidental: a variant that let a pass skip a wait — a per-subscription +budget, an early return once the budget is spent — would let a replayed segment come out in an order +the live run never delivered — fairness bought with nondeterminism, which is the one currency it +may not be paid for in. + +**A merged wait is one scoped operation, and the blocked set follows it in both directions.** Every +member enters the blocked set when the group blocks, because the snapshot has to name the complete +set; every member leaves it when the group's wait ends, whichever way it ended — a resolved future, +the post-registration double-check finding a record, or cancellation. The asymmetry is the trap: a +readiness activation resolves *every* pending future rather than only the waits it named, so the +group always ends together, while at most one member comes out of it holding a record. Only that one +is unblocked by taking it, so anything that unblocked members individually would leave the rest +marked blocked with their futures already discarded — waits the snapshot still names, and Core +retains and eventually parks a Workflow Task for. A caller that takes one value out of a merge and +goes on to wait for an ordinary Signal or condition has that event delayed behind the phantom wait +until the idle timeout, which is why this is not answerable by telling callers to keep consuming: a +merge is closed by `aclose()` and that does not close the subscriptions passed to it, each of which +may be consumed again sequentially. + +The next group wait re-enters the blocked state for every member and so advances every member's +`wait_generation`, including members that were never handed a record. That is a real cost and it is +paid for rather than avoided: a readiness report already in flight under the previous generation is +answered `Stale`, and what covers it is `rearm_readiness()` running on **every** completion, which +re-reports every non-empty buffer under the generation the runtime has by then already published to +the manager. Suppressing the bump instead — leaving a member blocked so its generation stands — buys +that coverage back by keeping the phantom wait. + +## Which side answers which activation job + +Runtime-internal stream jobs split in +`_handle_activation` — already `async`, and already doing pre-activation await work — rather than +running through the synchronous `_apply` chain (ADR-011): + +| Job | Answered | Why there | +|---|---|---| +| `ResolveExternalStreamWaits` | `_apply` | Readiness means "buffered", so the drain is a bounded buffer pop | +| `ReplayExternalStreams` | *prepared* in `_handle_activation`, delivered in `_apply` | Inclusive range reads plus integrity validation over the whole recorded range | +| `PrepareExternalStreamPark` | `_handle_activation` only | Installs intents and awaits the input backend; if its result is terminal and output exists, the outer Worker stages before completion | +| `FinalizeExternalStreams` | `_handle_activation` only | The finalization handler itself needs no input backend work (ADR-010); `OUTPUT_LATENCY` additionally makes the outer Worker stage output before completion | + +Routing the last three through `_apply` would put multi-second backend transactions inside a +synchronous `activate()` under a 2-second timeout, failing the Workflow Task for a healthy backend +and getting worse the more records replay must validate. Answered where they are, a transient +backend error becomes `WORKFLOW_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE` and an integrity +violation becomes `StreamIntegrityError`, rather than surfacing as a deadlock timeout that +misattributes a storage problem to the Workflow's own code. + +Watchers are torn down only on subscription cancellation, Run eviction, or Worker shutdown — never +by a Workflow Task completing. A watcher outliving the Workflow Task is what the first wakeup +mechanism in `wft-lifecycle.md` is made of. + +## Wait ID assignment + +`wait_id` comes from a per-Run counter starting at 1, incremented in `subscribe()` call order. +Workflow code is deterministic, so subscription creation order reproduces on replay and the same +subscription receives the same `wait_id`. It is stable for the life of its subscription and across a +Continue-As-New chain; its `wait_generation` increments each time that wait re-enters the blocked +state. + +Everything else is keyed off it — the annotation header, the park intent, the continuation, and +Core's wait set — so inserting, removing, or reordering a `subscribe()` call renumbers every later +wait and is a nondeterminism hazard on the same footing as inserting a timer +(`annotation-format.md`). + +## A subscription has one consumer + +`__aiter__` hands out a new generator on every call, and they are not independent views: the cursor, +the readiness future, and the blocked flag belong to the subscription. The readiness future in +particular is a single slot held in two places at once — on the subscription, where closing finds it, +and in the runtime's pending map keyed by `wait_id`, where the readiness activation finds it. + +So a second coroutine blocking on the same subscription is **refused**, on the single-subscription +path and inside `merge()` alike, with a non-retryable error raised on the Workflow thread. Without the +refusal the second waiter replaces the first in both places: readiness resolves only the newer future, +its cleanup removes the map entry, and the older one is left unreachable by the readiness activation +and by closing — a coroutine stranded for the life of the Run, while the shared blocked flag can +report that wait as not blocked at all, so Core is not even retaining a Workflow Task on its behalf. + +The refusal is on the *waiter* rather than on the iterator, which is what keeps ordinary sequential +re-iteration working: breaking out of an `async for` leaves the generator suspended at its `yield` +rather than closed, so an iterator-level claim could not tell that shape apart from two live consumers +(ADR-037). Two consumers of one *stream* is a supported shape and the way to ask for it is a second +`subscribe()` — delivery is a broadcast, so each wait gets every record and keeps its own cursor. + +## Closing a subscription + +Closing runs on the Workflow thread, so it divides the same way everything else here does: the part +that is a synchronous change to in-memory state happens immediately, and the part that touches the +backend is handed to the manager. + +The Workflow-side half ends the wait. It leaves the quiescent snapshot, iteration stops, and whatever +the buffer still holds is dropped **without being consumed**, so the consumption cursor stops short +of those records and a Continue-As-New successor receives them rather than stepping over them. +One property of the wait changes permanently: it can never block again. A closed wait that re-entered +the blocked set would be reported to Core, registered, and eventually retained or parked for on +behalf of a coroutine that no longer exists — a Workflow Task held open for nobody, which only a wake +would end. + +The Worker-side half — stopping the watcher and taking back the wait's park intent +(`wft-lifecycle.md`) — is requested through the runtime handle, which hops it onto the manager's +loop, rather than started from the subscription itself. Scheduling Worker work from the Workflow +thread is not merely misplaced in the way ADR-011 describes: nothing runs it, and nothing reports that +nothing ran it, so the watcher goes on prefetching into a buffer no one will drain for the rest of the +Run, with every symptom of that remote from its cause. The hop is what makes the request take effect +at all. + +**Closing does not forget the subscription** (ADR-029). Two things are built from the registered set +after an individual wait has ended, and both are wanted precisely for a stream the Workflow finished +reading early: the annotation binding for that wait, without which replay reads a `wait_id` no +binding covers and reports it as a wait the Workflow did not create, against code that never changed; +and the continuation cursor, without which a successor Run restarts that stream at `BEGINNING` and +re-delivers everything the closed subscription consumed. Keeping the entry is also what lets a +replayed Run satisfy the check that every wait the marker bound was recreated +(`annotation-format.md`): a closed wait was created, and a set the close had emptied would fail that +check against a Workflow that did nothing wrong. The manager's entry for the wait holds +resources and is dropped; the runtime's entry holds the record of what the wait was, and is marked +closed instead. + +## The runtime's half of shutdown + +`wft-lifecycle.md` owns the shutdown sweep itself: what is probed, when, and which answers owe a +wake. Three obligations are the runtime's own, and they constrain this code rather than Core's: + +- **Per-Run teardown is driven by `RemoveFromCache`, never by the shutdown hook.** The manager must + keep a Run's subscriptions, buffers, and cursor state alive until that job arrives, or a + `FinalizeExternalStreams` in flight finds nothing left to finalize from. +- **The sweep's two halves sit at two points in the Worker's own shutdown sequence** — the probe + immediately before Core's shutdown is initiated, the wakes and teardown after every activation has + been answered. Both live on the Worker's shutdown path rather than in the workflow worker's poll + loop, because a clean shutdown never raises out of that loop and would otherwise leave every Run + registered, its watchers running, and its owed wakes unsent. +- **An idle cached Run receives no eviction activation at shutdown**, so the sweep cannot be folded + into the eviction path — and that Run is exactly the one that most needs it, with records buffered + in a process about to exit and nothing else to tell the Workflow they arrived. + +Both halves are bounded: the probe by a grace short enough that it cannot hold up stopping the +pollers, the wakes by the Worker's graceful-shutdown grace. A wake still unacknowledged when the +grace expires is counted on `temporal_external_stream_shutdown_wake_failed` and shutdown proceeds. +It is counted rather than only logged because a dropped wake is silent by nature: the Workflow +simply waits, and nothing distinguishes that from a producer having nothing to say. diff --git a/arch_docs/streaming-poc-docs/spec/wake-signal.md b/arch_docs/streaming-poc-docs/spec/wake-signal.md new file mode 100644 index 000000000..c8eead1b3 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/wake-signal.md @@ -0,0 +1,231 @@ +--- +doc_id: EWS-SPEC-WAKE +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: reserved-wake-signal +related_adrs: [ADR-023, ADR-025, ADR-036, ADR-038, ADR-042] +--- + +# Reserved wake Signal + +The server-visible wakeup used whenever no open Workflow Task can accept local readiness. It never +carries stream payloads. + +## Where it is used + +```text +new record + ├─ open WFT → Python readiness call → Core activation + └─ otherwise → Temporal Signal → server creates WFT +``` + +"Otherwise" is three states, not one — parked, cached with no open Workflow Task, and evicted. All +three send this Signal; they differ in what the watcher does afterwards. See +`core-lang-protocol.md`. + +## Signal name + +`__temporal_external_stream_wake`. Fixed, versioned by the envelope rather than by the name, and +distinct from every `__temporal_workflow_stream_*` name already reserved by +`temporalio.contrib.workflow_streams` (ADR-001). + +## Envelope + +Core must read this Signal's fields, and Core has no access to the user's `DataConverter` or codec. +The envelope is therefore defined at the protocol level and is deliberately **not** a user payload. + +A single argument whose `Payload` uses metadata `encoding = "binary/protobuf"` and +`messageType = "coresdk.external_stream.WakeSignal"`, carrying: + +```protobuf +message WakeSignal { + uint32 envelope_version = 1; // starts at 1; Core rejects unknown versions harmlessly + string stream_name = 2; + uint32 wait_id = 3; + // 0 is reserved and means "no park generation" -- an unparked wake. Park + // generations are quiescence generations, which start at 1. + uint64 park_generation = 4; + string first_execution_run_id = 5; // chain identity, not the current run + string producer_session_id = 6; // diagnostics only +} +``` + +## Parked and unparked wakes use the same envelope + +`park_generation = 0` means the sender knows of no confirmed park for this wait and is asking for a +Workflow Task anyway (ADR-023). Core validates chain identity for an unparked wake and otherwise +accepts it as a recheck request: the runtime rechecks every active subscription on wakeup +regardless, so an unnecessary unparked wake costs at most one empty Workflow Task. + +A **non-zero** generation that the current Run does not recognize is still ignored as stale, because +there the sender is making a claim that turned out to be wrong. + +## Three properties that make this work + +- **Codec bypass.** The producer sends this Signal through a raw Workflow Service + `SignalWorkflowExecution` request built with the protocol's own serialization, *not* through the + user's `DataConverter` (ADR-025). A user codec that encrypts payloads would otherwise make the + envelope unreadable to Core, which is the component that must read it. The Signal carries no user + data, so bypassing the codec leaks nothing. +- **Stable request ID.** The Temporal `request_id` on the signal request is derived deterministically + from `(namespace, workflow_id, first_execution_run_id, stream_name, wait_id, park_generation)`, so + a producer retrying after an ambiguous failure produces the identical request and the server + deduplicates it. Generating a fresh UUID per attempt — which the public Python Signal path does — + would defeat this, which is another reason the wake path does not reuse it. + + For an **unparked** wake the generation is 0 and therefore carries no attempt identity of its own, + so the derivation additionally includes the sender's identity and a per-sender monotonic wake + counter, both held fixed across retries of that one attempt. Without that, two Workers shutting + down at different times would derive the same request ID and the server would deduplicate the + second wake away — turning a correct retry mechanism into silent loss. + + The counter is the **sender's** sequence and not the subscription's, which matters because a + subscription does not outlive its Run: an evicted Run that comes back is rebuilt with a fresh one, + and a counter held there restarts at 1 and re-derives the request ID that Run's previous + incarnation already used. The server deduplicates it, no Workflow Task is created, and a Run + holding buffered records waits for a wake that was discarded as a duplicate. Drawn from the + manager, the value is never reused for as long as the sender identity it is paired with lasts. + + That sender identity is unique per sender **instance**, drawn once when the sender is constructed + and held for its lifetime. The client identity is not enough: two Workers in one process share a + `Client` and each one's counter restarts at 1, so their first unparked wakes would derive + byte-identical request IDs and the collision above is exactly what happens. Drawn once rather + than per attempt, because a value that changed between attempts would make the retry a second wake. The value is + random, which is safe here because the wake Signal is sent from the Worker's own event loop and + never from Workflow code, so it is not replay-visible; the client identity is carried alongside it + so a request ID stays traceable to a client in server-side logs. +- **Chain identity.** `first_execution_run_id` lets Core classify a Signal that arrives after + Continue-As-New: same chain and a live wait means wake, same chain and an unknown generation means + ignore harmlessly, different chain means reject. The Signal is addressed to the Workflow ID without + a Run ID, so it always lands on the current Run of the chain. + +## Interception + +Core intercepts the Signal before user Signal dispatch, decodes the envelope without a +`DataConverter`, validates chain identity and generation, and issues `ResolveExternalStreamWaits`. +The first valid Signal creates or accompanies the new Workflow Task; duplicate wake Signals are +harmless. Python rechecks **every** active stream rather than only the stream named by the Signal. + +**Core suppresses the Signal from user handlers whether or not it validates**, so an unknown envelope +version or a stale generation can never reach Workflow code as an unhandled Signal. + +## One outstanding wake cycle per Run + +When the Worker receives `Parked`, `NoOpenWorkflowTask`, or `RunNotFound`, it owes a Signal because +Core could not accept the buffered readiness locally. That obligation is **Run-wide**, not per wait: +one Workflow Task can resolve the complete wait set, so concurrent waits and a replayed copy of the +same wait coalesce behind one outstanding cycle (ADR-042). + +Signal acknowledgement is not the end of the cycle. It proves the wake entered History, but the +Workflow Task it creates can still fail, be rejected, or replay. The cycle closes only after Core +accepts a successful completion for an activation that began during the acknowledged Signal attempt. +Each retry moves the activation boundary to its own start, so a completion observed during a failed +attempt cannot be inherited by the later successful retry. + +While the cycle is open, reconstructed readiness sends no distinct follow-up wake. After a +nonterminal completion, buffered readiness is reported again and may open a new cycle with a new wake +counter. A terminal completion never rearms delivery. Retiring a stale park intent first invalidates +the cycle whose generation it silenced, then performs the unparked reannouncement that makes the +buffer visible again. + +`BUSY_WORKFLOW` remains retryable. Treating it as terminal, or imposing a blind retry cap, can leave +a durable record buffered after its watcher advanced past the only report that announced it. + +## Producer send sequence + +1. Append the record. +2. Observe or lease-claim the current park generation. +3. Send the Signal idempotently. + +Only successfully appended records may trigger wakeup: a wake for a record that did not land produces +a Workflow Task that finds nothing. + +**The Signal is sent whether or not the claim was granted.** Losing the claim means another producer +*intends* to send; it is not evidence that one did. A lease permits takeover once it expires, but it +schedules nobody to take over, so a producer that crashed between claiming and signalling strands +the generation until some later append happens along — and the producer that stayed silent has +already reported an acknowledged wake. Sending anyway costs almost nothing, because a **parked** +wake's request ID is derived from the generation and ignores sender identity: racing producers issue +byte-identical requests and the server collapses them into one wake. A granted claim therefore saves +a round trip, not a wakeup. What the claim is for, and what its lease buys, is in +`backend-contract.md`. + +Wake signaling is independently retryable and idempotent. Retrying the wake step is a producer +obligation, not an automatic property; `publish()`'s acknowledged-wake contract is what turns it +into one. + +### The append itself has an acknowledgement window + +A backend commits on its own side and only then answers, so an `append()` that did not answer is not +an `append()` that did not happen. The Redis provider runs its atomic script server-side and receives +the result in a separate client-side step; a cancellation or a dropped connection between the two +leaves a durable record whose offset nobody holds. + +That window is therefore a **third outcome**, `AppendNotAcknowledgedError`, and not a failure. It +carries the interrupted *operation* — the exact record, byte-identical and still holding its +`(session_id, sequence)`, together with the stream it was for, the wake it owed and the lease it +chose — and `resolve_append()` re-appends *that* record on *that* topic, defaulting to the wake the +interrupted call was going to send. One call is right for both histories, because a +repeat append of byte-identical content under a used key writes nothing and returns the original +offset (ADR-020), while a key the backend never saw is appended now. Until it is settled the stream +refuses further appends from that producer, because the caller's obvious next move — `publish()` again +— draws a fresh sequence number and puts the value in the stream twice if the first attempt landed. +For `finish_writing()` the duplicate is a second write fence, which reads back as a producer session +that ended twice. + +The recovery is refused unless it is the producer instance, topic and exact bytes the outstanding +append belongs to. Each binding stops a different duplicate: a record does not name its own stream and +idempotency is scoped per stream, so another topic would append a second copy rather than deduplicate; +and a replacement producer with the same session id has its sequence and wake counters back at zero, +so settling there leaves its next publish reusing a sequence number and its next unparked wake reusing +a request ID. A producer that is gone recovers the way an Activity retry already does — same session +id, same calls, same order — which re-derives the keys and leaves the counters right (ADR-038). + +If `resolve_append()` itself is interrupted after calling the backend, its effective wake and lease +replace the previous attempt's in the one retained operation before the new +`AppendNotAcknowledgedError` is raised. Those values are therefore the next recovery's defaults. +Cancellation is sticky instead: once any attempt receives it, later transport failures do not erase +the caller's obligation to honour it after settling the append. A later refusal reports this same +canonical state, so its error cannot direct recovery with older wake or lease instructions. + +`AppendConflictError` is the one exception and the only one the contract can support: it says the key +was used with *different* bytes, so the record did not land and re-appending it would raise the +identical error. Cancellation delivered before the backend is called at all — while the sequence is +being drawn or the payload encoded — still propagates as cancellation, with nothing appended and +nothing owed (ADR-038). + +### All three steps after the append are inside the guarantee + +Once the append is acknowledged, `publish()` distinguishes exactly two further outcomes: the append +succeeded and the wake did not, or both did. **Steps 2 and 3 both belong to the first**, not only +step 3. A coordination call that +raises whatever the provider raises — a bare `ConnectionError` from observing the parked set or +claiming a generation — passes straight through a caller watching for the durable-but-unacknowledged +error, and takes the offset with it. The caller then has no statement about the record that already +landed, and its obvious move, retrying `publish()`, appends a **second** record: the sequence number +has advanced and the idempotency key with it. + +The two failures differ in what recovers them, so the error says which: + +| Failed at | `pending` | Recovery | +|---|---|---| +| Step 3, the Signal | the wakes still owed | re-send them verbatim (`retry_wake`) | +| Step 2, observe or claim | empty, `restart` set | call `wake()` again | + +**Cancellation after the append is one of these failures, not an exception to them.** It leaves the +identical state — durable record, unsent wake — and gets the identical recovery, with a flag saying +cancellation is what ended the attempt. `CancelledError` derives from `BaseException`, so a bare one +escaping here carries no offset, no `pending` and no `restart`, and cannot be told apart from +cancellation *before* the append: the caller can then neither wake the record nor safely re-publish +the value, since a second `publish()` draws a new sequence number and appends it twice. Cancellation +delivered before the backend is called still propagates as cancellation, because nothing was sent and +there is nothing to recover; cancellation delivered *inside* the call is the unknown outcome above, +not this one (ADR-036, ADR-038). + +`retry_wake` refuses an empty list rather than returning quietly, because a no-op there looks like +recovery while the record stays durable and unannounced. Calling `wake()` again is safe: it +re-observes the parked set, and a parked wake's request ID is derived from the generation rather than +the sender, so a wake another producer already sent deduplicates against it. + +`finish_writing()` carries the identical contract. The duplicate it prevents is a second write fence, +which reads back as a producer session that ended twice. diff --git a/arch_docs/streaming-poc-docs/spec/wft-lifecycle.md b/arch_docs/streaming-poc-docs/spec/wft-lifecycle.md new file mode 100644 index 000000000..725895ca8 --- /dev/null +++ b/arch_docs/streaming-poc-docs/spec/wft-lifecycle.md @@ -0,0 +1,547 @@ +--- +doc_id: EWS-SPEC-WFT +status: normative-pre-production +audience: [implementers, coding-agents, reviewers] +canonical_for: workflow-task-lifecycle +related_adrs: [ADR-004, ADR-009, ADR-016, ADR-017, ADR-021, ADR-030, ADR-031, ADR-032, ADR-043, ADR-045] +--- + +# Workflow Task lifecycle + +When a Workflow Task is retained, how it ends, and how a subscription is woken in each state it can +be left in. + +## Live consumption + +While a Workflow Task is open, the runtime reads from the last committed offset and resumes the +Workflow for each data record, across as many activations as draining takes — one activation +delivers a bounded number of records and then blocks so the activation can end, even where more +records are already buffered (`python-runtime.md`). + +The Workflow Task ends when either every active subscription has reached a write fence with no +subsequent record immediately available, or no record arrives on any of them before the idle timeout. +The timeout covers producers that crash, omit `finish_writing()`, or do not use the SDK. + +## The idle timeout is a Workflow-Task policy + +It applies to the **complete set** of subscriptions the Workflow is blocked on, not to each +subscription independently: one idle stream cannot park a Workflow Task another stream is still +driving, and a fence on one stream cannot bypass the timer. So a set whose members are configured +differently must reduce to one value — by `min`, in `wait_id` order, over the quiescent set and +nothing else, which is what makes the result reproduce on replay (ADR-016). Core clamps the result +below the rollover deadline so rollover stays authoritative, and rejects a non-positive timeout as a +malformed completion rather than coercing it. + +## Workflow Task rollover is mandatory + +A retained Workflow Task is bounded by the server's Workflow Task timeout, and a continuously fed +stream never reaches the idle parking path at all. The runtime therefore always runs a rollover +deadline derived from the Workflow Task timeout and independent of the idle timeout, and completes +the current task early when it wins; every active subscription, cursor, annotation delta, and +readiness generation survives. It needs a timer facility that does not depend on the local-activity +request sink (ADR-017). + +**Rollover bounds the Workflow Task, not an activation.** Core decides it between activations and +cannot interrupt one in progress, so it is no protection against a single activation that never +returns. What bounds an activation is the per-activation record budget in `python-runtime.md`. + +Rollover also bounds a second effect of holding a task open: while it is retained the server cannot +start another one, so Signals, Updates, and non-legacy Queries queue until it completes. Outstanding +local activities already have this property, but a stream can hold a task open far longer. Retention +latency for those inputs is bounded by the rollover deadline and nothing else, so callers who need +lower latency must lower the Workflow Task timeout rather than the idle timeout. + +## Output visibility has its own deadline + +Workflow-originated output cannot wait for an arbitrarily long input idle timeout. Each output topic +has `max_publish_latency`, defaulting to 100 milliseconds in the first release. A retained batch uses +the minimum policy of every non-empty topic in that batch; Core preserves the earliest deadline and +clamps it below ordinary Workflow Task rollover (ADR-045). + +Python sends `WorkflowOutputStreamBuffered` only when the completion has a quiescent input snapshot, +no user/server command, and no output-capacity rollover. Core retains the task and arms the output +deadline. When it expires, Core issues `FinalizeExternalStreams{OUTPUT_LATENCY}`; Python stages the +batch and returns both the existing finalization command and `WorkflowOutputStreamCommit`. Core then +writes the shared marker and forces a replacement task. + +Without a quiescent input snapshot — including an output-only Workflow — or when an ordinary command +already requires reporting the task, output is staged immediately. Capacity also bypasses buffering: +the commit requests a replacement so a blocked `publish()` can continue in a fresh batch. The +latency bound is a healthy-path visibility guarantee, not an availability SLA while the provider or +Temporal service is unavailable. + +## Workflow Task admission is lossless + +A Run owns the task token of its current Workflow Task until that task is reported. A polled +replacement can arrive before the local completion message clears the original even when no +activation jobs remain, because poll results and local completions use different input lanes. + +Admission therefore buffers whenever `ManagedRun` already owns a Workflow Task; pending jobs are +not the only evidence of outstanding work. The lower-level `_incoming_wft` invariant remains a debug +panic, while its release path defensively queues the replacement and returns without overwriting the +original slot. The original token is reported first, and only then does the replacement drain +(ADR-043). Substitution is never recovery: it loses the token for a task the Worker still owns. + +## Three independent questions at every activation return + +1. **Did replay-visible stream state change?** If yes, a `WorkflowStreamProgress` command carries the + observation delta, on *every* completion path. This is what commits the cursor boundary, and it is + not conditional on why the Workflow Task ended: if a consumed record influences a command that + lands in History while the consumption itself is never marked, replay re-delivers that record + while the command it produced is already durable, and the divergence surfaces as an unrelated + nondeterminism error much later (ADR-005). +2. **Is the Workflow blocked on streams?** If yes, a `WorkflowStreamQuiescent` command reports the + complete quiescent snapshot. This registers the wait set with Core and — on the completions that + can be held open — starts the idle timer. +3. **Does this Workflow Task hold logical output?** If yes, Python either stages it and sends + `WorkflowOutputStreamCommit`, or sends `WorkflowOutputStreamBuffered` when and only when the + quiescent completion is retainable. Output and input use the same marker and activation-segment + schedule. + +A completion may carry any compatible combination of those commands. `WorkflowStreamProgress` never +implies retention, and `WorkflowStreamQuiescent` carries no annotation data (ADR-004). + +| Condition at return | Python sends | Core does | +|---|---|---| +| No subscriptions, nothing observed | Normal completion | Reports the WFT to the server | +| Records consumed, no pending stream waits | `WorkflowStreamProgress` | Records the marker, reports the WFT | +| Records consumed **and** a terminal command (complete, fail, cancel, continue-as-new) | `WorkflowStreamProgress` **ordered before** the terminal command | Records the marker, then applies the terminal command | +| Stream waits pending **and** server-bound commands (timer, activity, child workflow, signal) | `WorkflowStreamProgress` ordered before those commands, then `WorkflowStreamQuiescent` with the same complete snapshot | Records the marker, **registers the waits, arms no timer**, reports the WFT | +| Stream waits pending, no other command, **nothing consumed** | `WorkflowStreamProgress` carrying the header on first observation plus an **empty segment**, then `WorkflowStreamQuiescent` | Accumulates the delta, registers the waits, retains the WFT, starts the idle timer | +| Stream waits pending, no other command, records consumed | `WorkflowStreamProgress`, then `WorkflowStreamQuiescent` with the **complete** set | Accumulates the delta, `ActivationCompleteOutcome::DoNothing`, retains the WFT, starts **one** idle timer for the set | +| Logical output, no retainable quiescent snapshot or a user/server command | `WorkflowOutputStreamCommit` after provider staging | Records the output manifest in the shared marker and reports the WFT | +| Logical output with a retainable quiescent snapshot and no command or capacity rollover | `WorkflowOutputStreamBuffered`, plus the ordinary input progress/quiescence commands | Retains the WFT and arms the earliest output deadline without provider staging | +| Logical output reaches record or logical-byte capacity | `WorkflowOutputStreamCommit{request_rollover=true}` after staging | Records the shared marker, reports the WFT, and requests a replacement | + +**Command ordering is normative**: `WorkflowStreamProgress` precedes every command whose value could +depend on consumed data, and so does the marker recording it. On replay that is what guarantees a +record is validated before the command derived from it is matched. + +Successive deltas accumulate while the task is retained and are written as **one** marker when it +finally completes, whether from parking, rollover, or a later activation that produces commands. + +### Registering a wait set is not retaining the Workflow Task + +The two are decided separately, and only the second is refusable. A completion carrying server-bound +commands must be reported so the server can act on them, so it cannot ask for retention — but the +Workflow is no less blocked on those subscriptions for it, and **Core is the only place that set is +recorded**. Core therefore registers the waits and arms nothing: no idle timer, no rollover deadline, +no all-fenced immediate park, because each of those exists to end a *retained* task. + +Welding the two together — withholding the snapshot on any completion that cannot be retained for — +leaves a Workflow that starts a timer and first blocks on a stream in the same activation registered +nowhere. Readiness then has no wait to resolve against, a wake Signal marks nothing ready, and every +Workflow Task a wake produces completes empty: the Run is unresumable by any wake, with its records +sitting in the stream. That is a permanent deadlock reachable from ordinary user code. + +Two more completions register without retaining. A **replayed** completion must, because the wait set +is per-Worker runtime state rather than History and nothing else rebuilds it, while retaining a +replayed task would arm wall-clock deadlines against a boundary the marker has already fixed. A +**query answered on its own activation** refuses retention in order to report the answer, which makes +it a boundary Core decided and so one owed a finalization round trip before any marker is written. + +The mirror-image rule: a boundary that tears the Run down, or that has already asked lang to +finalize, registers **nothing** — re-recording a snapshot there would bump the quiescence generation +underneath a job already in flight against it. + +## Race-free parking + +Parking is all-or-nothing across the complete set of subscriptions the Workflow is blocked on, and +uses a generation-based handshake coordinated through the backend: + +1. Core marks every wait `Parking` and sends `PrepareExternalStreamPark`. This is runtime-internal; + no user Workflow code runs. +2. Python installs a park intent per **subscription**, keyed `(stream key, wait_id)`, carrying its + cursor boundary and the park generation. **The set is the one the job names**, not every + subscription the Worker holds: a registered subscription nobody is blocked on could otherwise + abort a legitimate park on a record that belongs to no wait Core is parking, and it would leave + an intent behind for a park that never covered it. +3. After all intents are installed, it rechecks every stream in that set. +4. All still empty → `ParkSetConfirmed`, carrying the terminal `final_observation_delta`. Core + appends it, records the marker, and completes the WFT. +5. Any stream ready → Python removes every installed intent and returns `StreamSetBecameReady`. Core + aborts that parking generation and issues `ResolveExternalStreamWaits`. + +### Output and parking use the same terminal race + +If the runtime holds logical output while it answers `PrepareExternalStreamPark`, the answer depends +on the backend recheck (ADR-045): + +- Park confirmed: Python stages output and orders `WorkflowOutputStreamCommit` before + `ExternalStreamParkResult(confirmed)`. Core records one shared park marker and does not force a + replacement. +- Stream became ready: Python sends `WorkflowOutputStreamBuffered` before + `ExternalStreamParkResult(became_ready)`. No output is staged and Core retains the original + earliest deadline. +- Readiness wins after Python has already staged for a confirmation: Core rejects the stale park + terminal, resolves the abandoned park to remove every installed intent, then finalizes with + `TASK_COMPLETED`, records the already staged commit once, and forces a replacement. +- The output deadline wins while preparation is outstanding: Core invalidates the park generation, + resolves it for the same cleanup, then finalizes with `OUTPUT_LATENCY`. A commit already carried by + the losing result is retained; otherwise Python stages during finalization. Core records it once + and forces a replacement. + +The output timer remains armed during park preparation. Suppressing it would let a stalled park +provider violate the output visibility bound indefinitely. Late timer and park results are stale and +cannot produce another terminal or marker. + +An attempt that ends before Core confirms it withdraws every intent it installed — whether an +install raised, a recheck raised, or **the attempt was cancelled**. Cancellation is not the exotic +member of that list but the likeliest one: an activation withdrawn by Core and a Worker shutting +down both abandon a half-installed park that way, and in Python cancellation does not arrive as an +ordinary exception, so a rollback written for ordinary exceptions is exactly the rollback that +misses it (ADR-032). A park visible to producers for a generation Core never confirmed is one they +send wakes against and Core discards as stale, and the eviction that follows takes with it the only +record that those intents were installed at all. + +A producer appends its record *before* it observes or claims the park generation, and only then sends +the wake Signal. That ordering plus the recheck at step 3 is what closes the empty-check/completion +race: an append is either seen by the recheck or paired with a Signal. Readiness accepted before +`ParkSetConfirmed` wins, and the confirmation for that generation is then stale. + +**A park intent exists only while its park is outstanding.** Holding that true takes more than the +moments at which a removal is decided, because the intent is durable backend state while nothing that +knows about it is: the record of which Worker installed it dies with that Worker, and the +subscription that record hangs off can be closed while the Run runs on for another week. + +Three moments decide a removal: + +- **A resolve ends a park this Run is sitting in.** `ResolveExternalStreamWaits` is the notice, + because it covers both ways a confirmed park ends and Core's own state moving on is not something + the backend can observe. Python removes the intents it installed for that park. +- **Registration reconciles an intent this Worker inherited.** A Worker registering a wait that + already has an intent in the backend, having installed none itself, removes it. Registration is + where such an intent becomes visible and where its status is unambiguous: a subscription is + registered by user Workflow code running, and no user code runs inside a park, so an intent found + there belongs to a park that is over. *Reading* it is what records it. An inherited intent is + mirrored nowhere on this Worker, so until the read writes it down there is nothing for any removal + path to work from, and the resolve and the cancellation both look at the same absence and do + nothing. +- **A cancellation takes back the intent of the wait it drops.** Closing a subscription removes it + from the manager, so its intent is removed there — under the same serialization as this Run's + parking and resolving, and before the watcher stops — rather than left to either point above. A + closed wait is never registered again for the life of the Run, and the resolve iterates the + subscriptions the manager still holds, so neither of them ever reaches it (ADR-030). + +A removal that one of those decided on and the backend did not confirm is **owed**: recorded per +Run against `(stream key, wait_id)` and retried autonomously with bounded exponential backoff. +Parks, resolves, registrations and evictions still drain it eagerly under the Run's park lock, but +backend recovery alone is sufficient for progress. A ledger entry says *a removal was decided on +and has not been confirmed*, not *an intent exists* — which is what makes draining it safe from any +of those paths rather than only from the one that recorded it. The entry holds nothing that a +teardown could invalidate: no watcher, no buffer, no connection beyond the backend the removal has +to go through. It is made **before** the backend call, because a call that never comes back — the +backend raised, the task was cancelled mid-await — owes the removal exactly as an error does, and +the subscription it was reached through is usually dropped in the same breath. A fresh install at +that key supersedes it. Why the ledger is manager state rather than a field on the subscription is +ADR-031. + +The retry task is strongly held per Run by the manager. Eviction drops the cached Run but not an +owed-removal ledger or its task, so cleanup can finish while that Run remains absent. Worker +shutdown cancels and awaits the manager-owned tasks within its shutdown grace period, and then makes +one last bounded pass over whatever is still owed: cancelling a loop that is sleeping out a backoff +throws away an attempt the recovered backend would have answered, and eviction cannot make that +attempt while shutting down without waiting on the very lock a stuck loop is holding. What survives +that pass is logged, because an intent still installed as the process exits suppresses the unparked +wake for whoever picks the Run up next. + +**Every hold of the park lock that is not inside an activation is time-bounded.** The lock serializes +one Run's park work, and the paths that take it off the activation path -- the retry loop, the +registration-time reconciliation, the close, the last pass at shutdown -- all reach a backend that +can hang rather than raise. Counting attempts does not bound an attempt that never returns: the task +stops retrying, and the lock the next park, resolve or eviction needs stops being available. A hold +that runs out of time is released and treated as a failed attempt, which leaves the entry owed +exactly as an error does. Holds taken *by* an activation are not bounded, because the wait they own +is the backend exposure the park handshake already has. + +**Retiring an intent is not the same as delivering what it silenced.** Every wake sent while a stale +intent was installed named the generation behind it, and Core discards a non-zero generation that is +not the park it holds -- so a record that arrived in that window is buffered on the Worker with its +wake already counted as sent, and the watcher will not report it again without a new append. Every +path that retires an intent therefore hands that record off, one of two ways. + +**With the wait still registered here, through local readiness.** That is the drain of a cached Run, +and the registration-time reconciliation whose removal succeeds first time -- the ordinary case for an +inherited intent, and the one that never goes near the ledger. The reconciliation runs concurrently +with the watcher it races, which is how the record gets silenced in the first place. Narrow, because +the subscription is there to be read: only a wait still holding records, so a healthy park costs no +Workflow Task, and never during the sweep, which owns every wake from that point and accounts for +each one. + +**With the wait gone, through one unparked wake.** A close drops the subscription and an eviction +drops the Run's map, both deliberately leaving the ledger and its retry alive -- so the removal that a +recovered backend finally lets through is frequently the one with no buffer left to consult and no +readiness channel to use. Conditioning the announcement on a subscription being there is what keeps +the leak fixed while putting the record loss straight back. The wake is unconditional: whether a +remote producer appended during the suppression window was never visible from this Worker, so "a +record arrived" and "none did" are the same observation from here, and only one of them is silent if +the guess goes wrong. It costs one empty Workflow Task, which this design permits. Sent as work of +its own rather than under the park lock, whose budget bounds a single-key backend call and not a +Signal with retries behind it, and awaited by shutdown -- the last of these handoffs is created by the +final owed-removal pass, and a wake merely scheduled as the process exits was never sent. + +Both provider outcomes that clear the key hand off -- *removed* and *absent*. An absent key is +cleanup that happened, including a delete this Worker committed and never got the reply to +(`backend-contract.md`), and reading it as a mismatch would strand the record for good. A *mismatch* +hands off nothing: an intent is still installed, so the suppression has not ended, and it belongs to +a park the entry knows nothing about. + +**No wake this Worker sends names an intent it has already decided to remove.** An owed removal says +the generation installed at that key is one Core has discarded, so composing a wake from it produces +a Signal the service accepts and Core ignores -- which is worse than a failure, because the send +reports success. The manager holds the ledger, so the manager decides what a wake names rather than +the sender reading `current_park_generation` for itself: an entry in the ledger, or a handoff for an +intent just retired, both compose the unparked wake. That is what makes the shutdown sweep's +accounting true, since the sweep wakes the subscriptions of a Run whose stale intents the final +removal pass has not reached yet. Its wake and that removal's handoff are then one obligation, and it +is recorded as discharged on the entry so the pass does not send it twice. + +**Discovery gets the same autonomous retry the removal does.** An owed removal is recorded from an +intent's identity, so a read that never succeeds records nothing -- no ledger entry, no retry task, no +owner. A reconciliation whose reads all fail therefore keeps retrying, with capped backoff, for as +long as this Worker holds the subscription; once a read succeeds and only the removal fails, it hands +the entry to the ledger rather than draining it twice. The subscription is what bounds it, and not +only for tidiness: removing whatever is installed at that key is justified by this Worker holding the +Run with this wait registered on it, and once the subscription is gone the intent found there could +be a park another Worker is sitting in. + +**A drain removes only the intent it recorded.** The provider atomically compares the +`park_generation` and Run ID and deletes only on a match; anything else is forgotten rather than +removed. The stream key is stable across a Continue-As-New chain while `wait_id` restarts at 1 in +the successor, so an entry a predecessor Run left behind can name the key of a park a **successor** +is sitting in, and taking that out is strictly worse than the leak the ledger exists to close: it +unparks a Run whose park is real and whose producers have no reason to send anything. A process-local +park lock cannot close this cross-Run, cross-Worker race; the conditional backend operation does +(`backend-contract.md`). + +What a left-behind intent costs is the invariant's whole point: `current_park_generation` keeps +answering a generation Core has discarded, each producer wake names it and Core discards it as +stale, and because a parked wake's request ID ignores sender identity the second such wake is +byte-identical to the first and the server deduplicates it away. The Workflow waits forever on a +record that is durably present. Nor is the damage confined to the wait that owns the intent: a +stale intent keeps `parked_wait_ids` non-empty for the whole stream, which suppresses the unparked +fallback for every **other** wait on it, so a Workflow that closed one subscription can lose the +wakes of the ones it is still reading. + +Reconciliation reads before it writes, and is serialized against this Run's own parking and +resolving: a Run that never parked owes the backend nothing, and a reconciliation overlapping a +confirming park must not remove the intent that park has just installed — an unwakeable Run produced +by the mechanism that exists to prevent one. It retries a bounded number of times before leaving the +rest to the ledger, and takes that lock **per attempt** rather than holding it across the backoff: +parking is answered inside an activation under Core's deadlock timeout, so a reconciliation asleep +with the Run's park lock would spend an activation's budget on cleanup for a park that is already +over. + +The provider holds up the rest: it must stop reporting a generation once its intent is removed +(`backend-contract.md`), or every reader of that call acts on a park that is over. + +## Three generations, named separately + +| Name | Scope | Increments when | Where it appears | +|---|---|---|---| +| `wait_generation` | one subscription | that subscription re-enters the blocked state | the wait entry in the quiescence report and in readiness notifications | +| `quiescence_generation` | one complete blocked snapshot | the Workflow becomes quiescent again after any resumption | the quiescence report, the park preparation job, and the park result | +| `park_generation` | one confirmed park of a set | a park set is confirmed | the backend park intent and the wake Signal | + +`park_generation` takes the value of the `quiescence_generation` that was parked rather than being a +fourth counter. A wake Signal carries `(wait_id, park_generation)`, and those two values are the only +generation state that reaches the backend or the wire; `wait_generation` never leaves the Core/lang +boundary. + +A Signal naming a generation the current Run does not recognize is harmless: the runtime rechecks +every active subscription on wakeup regardless — the stream a wake names is only a hint — so a stale +Signal costs at most one empty Workflow Task, which this design permits. A wake sent when **no** park +generation exists carries the reserved `park_generation = 0` and is a recheck request rather than a +stale Signal (ADR-023). + +## Completions that leave subscriptions active but unparked + +A completion carrying server-bound commands, and a rollover, both end the Workflow Task without a +park handshake. In that state there is no retained task to notify locally and no park generation for +a producer to observe, so a later append has nothing to wake. Three mechanisms cover it, and one of +them always applies: + +1. **The consumer's own watchers.** Watchers stay live while the Run is cached, and one that observes + an append with no Workflow Task open sends the wake Signal itself. This is the normal path, and it + also covers a producer that crashed after appending. + + That Signal is retried in place until the server acknowledges it, because nothing behind it will + try again: the watcher moved its prefetch position past the record when it buffered it and goes + back to waiting for a *new* append, re-arming readiness needs the activation the lost wake was + supposed to cause, and the idle timer runs only while a Workflow Task is retained, which is + exactly what this state does not have. A single unacknowledged attempt is a lost record on a + Worker that is otherwise healthy and may stay running for hours; the shutdown sweep is a backstop + for the end of that Worker's life, not a retry policy. The retries are the **same** wake — every + attempt derives the identical request ID (`wake-signal.md`) — so one that did in fact arrive is + deduplicated server-side rather than producing a second empty Workflow Task. +2. **A forced replacement Workflow Task — only while a Workflow Task is open.** A Run being shut down + that still holds a task finalizes its marker and completes the task requesting a replacement. +3. **An unparked wake Signal — when no Workflow Task is open.** A replacement is requested *on the + completion of an existing task*, and there is none, so the runtime sends the reserved wake Signal + itself and waits for the server to acknowledge it before the watchers go away. + +Mechanisms (2) and (3) are runtime-owned state transitions, not something the consuming Workflow +requests (ADR-009), and both are offers to the task queue rather than promises by the shutting-down +Worker: any eligible Worker may pick the task up and reconstruct the subscription from the marker, +and if none is available the task times out and the server retries it. + +**A marker is never written without its terminal.** If the boundary cannot be finalized the Workflow +Task is abandoned and retried rather than committing a partial annotation as a best effort: an +abandoned task commits no cursor and loses no record, while a truncated annotation is durable and +wrong (ADR-008). + +## Shutdown and eviction are two transitions + +| Run state at shutdown/eviction | Marker | Server-visible replacement | +|---|---|---| +| Workflow Task open — retained by the wait set, or open with an unfinished activation | Core issues `FinalizeExternalStreams{SHUTDOWN}` and writes the marker from the finalized annotation | Same completion sets `force_new_wft = true` | +| No open Workflow Task — the window after a command-producing or rolled-over completion | **None.** Nothing is accumulated; see the invariant below | Python's manager sends the reserved wake Signal for each active subscription and awaits acknowledgement, then tears the watcher down | +| No open Workflow Task and the wait set is parked | None | None needed. A confirmed park generation exists, so the producer's wake path applies unchanged | + +Core and Python must classify a Run into the same row — Core when it decides whether to finalize, +Python's sweep when it decides whether a wake is owed — or a Run is handled by both mechanisms or by +neither. + +### The invariant that makes the first row safe + +> An accumulated, unwritten annotation exists only while a Workflow Task is open. + +Deltas arrive only on activation completions, activations exist only under a Workflow Task, and every +completion path writes the accumulated annotation as exactly one marker and clears it. So a +Core-decided boundary either has a Workflow Task to finalize against or has nothing to write; there +is no third state in which Core holds a partial annotation with no way to complete it. + +### Sequencing: teardown cannot precede finalization + +Eviction is produced only once no activation is outstanding, so a `FinalizeExternalStreams` issued +before eviction is always answered before `RemoveFromCache`; Python's per-Run teardown is driven by +`RemoveFromCache` rather than by the shutdown signal; and pending evictions count as pending work, so +shutdown does not finish underneath them. Two consequences constrain the implementation: + +- The marker is written by the **finalization completion**, never by the eviction completion. An + eviction completion reports nothing, so a marker attached there would be silently dropped. +- If finalization cannot be answered, Python fails the activation rather than returning a partial + terminal. Core writes **no** marker and the task is retried from the previous one. Nothing consumed + during the abandoned Workflow Task was committed, so the cost is one repeated Workflow Task. + +### The no-open-WFT transition + +Nothing local can create server-visible work here, so the wake Signal does it. The manager runs this +as an explicit shutdown sweep rather than leaving it to watchers, because watchers only fire on an +append and the point is to hand the Run over whether or not one arrives. An idle cached Run receives +no eviction activation at shutdown at all, so the sweep cannot be folded into the eviction path. + +**The sweep is in two halves, at two points in shutdown**, because the moment the answers still exist +is not the moment the wakes may be sent: + +1. **Before Core's shutdown is initiated**, the manager calls `external_stream_run_status(run_id)` + once for every Run with active subscriptions and **records** the answer — a **read-only** probe, + answered on the same serialized local-input lane as readiness, returning + `WftOpen | Parked | NoOpenWorkflowTask | RunNotFound`. It is deliberately not the readiness call: + readiness means "a record is buffered", and probing with it would be a false claim that + manufactures a spurious activation on the way out. +2. **After the pollers have stopped and every activation has been answered**, the manager acts on the + recorded answers. `WftOpen` — the finalization is already answered and the first table row + applies, so no wake is owed. `Parked` — nothing to do. `NoOpenWorkflowTask` or `RunNotFound` — + send the wake Signal and await the server's acknowledgement. A Run the first half never reached is + probed again here, on whatever Core has left to say. +3. Teardown of a Run happens only after step 2 resolves for it, or after the graceful-shutdown grace + period expires. + +Neither half can move to where the other is. Core's workflow-state lane ends when shutdown is +initiated, so every later probe answers `RunNotFound` — which owes a wake exactly as +`NoOpenWorkflowTask` does, so a late-probing sweep still sends its wake and still looks correct while +the two answers that mean *do not send one* have silently stopped being reachable. And a wake sent +before the pollers stop offers the Run to a task queue this Worker will answer itself. What the split +gives up is the guarantee that no Run acquires a Workflow Task after the probe — costing at most one +extra empty Workflow Task, against an ordering that would buy the guarantee with an answer describing +nothing. + +A wake still unacknowledged when the grace period expires is counted on a distinct +`temporal_external_stream_shutdown_wake_failed` metric and shutdown proceeds. Retries within the period reuse +the derived request ID and deduplicate server-side; past it, the Run falls back to the durability +boundary below rather than blocking shutdown or pretending the wake happened. + +**Every subscription the sweep does not resolve is counted, not only the wakes it attempted.** The +sweep starts by treating each subscription as unaccounted for and resolves it exactly once: when its +Run's status says nothing is owed, or when a wake has been attempted and its result recorded. Whatever +is left when the sweep stops is counted on the same metric. Three cases reach that point and all three +are silent otherwise: + +- the grace period expiring **inside** a hanging wake send, which cancels the sweep where it stands; +- every subscription **after** that one in the serial loop, which is never visited; +- a Run whose **status probe could not answer**, which the sweep deliberately sends no wake for — it + might have an open Workflow Task — but which may equally be holding a buffered record with nowhere + to announce it. "We could not tell" is not "nothing was owed." + +Accounting only where the sweep reached reported `shutdown_wake_failures == 0` for a Worker that had +just abandoned every one of its handoffs, which is exactly the silence the metric exists to break: a +dropped wake looks identical to a producer with nothing to say. + +The same metric counts a **cleanup handoff** the exit could not wait out. A stale intent retired by +the final owed-removal pass owes an unparked wake for the record it silenced, and that pass runs after +the sweep, so its handoff is as much a part of this shutdown as a wake the sweep sent itself — and as +silent if abandoned. What the grace period cuts off there is counted, not dropped. + +Three cases are deliberately *not* counted, because a metric that fires on a clean shutdown is not +alertable. A Run reported `Parked` or `WftOpen` owes nothing — the first is woken by a producer's +append through the ordinary path, the second is Core's to finish. A manager with **no status probe +wired** has no sweep at all: the mechanism is defined in terms of what Core answers, so there is no +obligation it could have failed to discharge. And a subscription the **live path removed** while the +sweep ran has had its handoff made: readiness answered `RunNotFound` owes a wake, sends it, and drops +the subscription only afterwards — a likely answer during shutdown, with watchers running for the whole +grace window and awaits inside the sweep for them to interleave with. The sweep then reaches that Run, +finds no subscriptions, and moves on; counting what it skipped there reports a loss that did not +happen. + +## The durability boundary, stated honestly + +Mechanisms (1) and (3) are **not durable**. Both depend on the consuming Worker being alive — (1) with +the Run cached and a watcher running, (3) for as long as the shutdown grace period lasts. Once the +Workflow is parked and the Run is evicted or the Worker restarts, no watcher exists, and a producer +that crashed between its append and its wake Signal leaves the Workflow parked with data available +and nothing to wake it. Nothing in the consumer can repair that, because the consumer is not running. + +This design does **not** claim to solve that case implicitly. One of the following must be chosen per +deployment, and the choice is explicit: + +- **Durable producer (default expectation).** `publish()` is complete only when its wake step has been + acknowledged. Producers that are themselves durable — an Activity, which Temporal retries — satisfy + this for free. A plain external process must retry until acknowledged, and the API surfaces the + un-acknowledged state rather than hiding it. +- **Backend outbox.** The provider durably records the pending wake and a relay delivers it. This + removes the requirement on the producer at the cost of a component that must itself be operated. + Out of scope; future work. +- **Consumer-side sweep.** A periodic Workflow Timer rechecks parked subscriptions. This bounds the + stall by the sweep interval instead of eliminating it, and costs History events. + +## Multiple subscriptions to one stream + +Cursors are per **subscription**, not per stream. Two subscriptions in one Workflow to the same stream +name have independent wait IDs, cursors, park intents keyed by `(stream key, wait_id)`, +`wait_generation`s, and entries in the annotation header and the Continue-As-New continuation state. +Delivery is **broadcast**: each sees every record from its own cursor, and work-sharing between two +subscriptions inside one Workflow is not supported (ADR-021). + +Cancelling a subscription commits its cursor at the next marker and removes it from the wait set. +Re-subscribing to the same stream name creates a *new* subscription with a new wait ID rather than +resuming the cancelled one. Resumption across Continue-As-New is by wait ID. + +## What a failure costs + +**Marker recording commits the cursor. Reading or delivering a record does not.** So a backend read or +coordination failure fails the current Workflow Task for normal Temporal retry, and a Worker crash +discards uncommitted reads and replays from the last committed marker — in both cases the cost is a +repeated Workflow Task and no record is lost. Spurious wakeups and empty Workflow Tasks are permitted +correctness-wise, which is what makes stale readiness and stale wake Signals safe to ignore rather +than something to resolve exactly. Backend unavailability, retention loss, and integrity violations +must surface distinctly — see `failure-taxonomy.md`. + +### A wake can race a terminal command + +A wake Signal may enter History while the Workflow Task is issuing its terminal command. The server +can reject that task with `UnhandledCommand` so the external event is not lost, then schedule replay. +That retry is a normal Temporal ordering outcome: the Workflow may execute again and still complete +with the same stream observations. Tests for terminal wake races therefore allow +`UnhandledCommand` specifically, reject every other Workflow Task failure cause, and compare the +observations of every execution rather than requiring an exact workflow-body start count. diff --git a/arch_docs/streaming-poc-docs/verification-hazards.md b/arch_docs/streaming-poc-docs/verification-hazards.md new file mode 100644 index 000000000..61506274c --- /dev/null +++ b/arch_docs/streaming-poc-docs/verification-hazards.md @@ -0,0 +1,202 @@ +# Verification hazards + +Seven ways a test result in this repository can be confidently wrong. Each is a current constraint +on trustworthy validation and is cheap to check once it is named. + +A result is evidence about the system *plus* its harness. Broken apparatus can produce a plausible, +repeatable failure against correct code; a gate that never armed its case can produce a clean pass +while checking nothing. + +## 1. A stale native extension + +**Symptom.** Rust tests pass. Python end-to-end tests fail with a nondeterminism error naming a +mechanism that was just implemented — as though Core were missing the feature entirely. Which it +was: the process had loaded a Core built before it. + +**Why it happens.** `pyproject.toml` sets + +```toml +[tool.maturin] +manifest-path = "temporalio/bridge/Cargo.toml" +module-name = "temporalio.bridge.temporal_sdk_bridge" +``` + +so the extension must land at `temporalio/bridge/temporal_sdk_bridge.abi3.so`, which is what +`import temporalio.bridge.temporal_sdk_bridge` resolves. Running `maturin develop` **from +`temporalio/bridge/`** instead of the repo root uses that crate's own packaging and installs a +top-level `temporal_sdk_bridge` package into the virtualenv. The command succeeds. It prints +`Installed`. Nothing imports the result. + +**Why it is insidious.** A stale extension does not error — it is quietly correct for the *old* +code. Every test that does not exercise the new Core keeps passing, so the suite stays green and +only the newest tests disagree. The natural reading is that the new tests found a bug. + +**The check.** Ask the process what it actually loaded, rather than trusting the build: + +```python +import temporalio.bridge.temporal_sdk_bridge as b +import os, time +print(b.__file__, time.ctime(os.path.getmtime(b.__file__))) +``` + +If the timestamp predates the Rust change, nothing since then has been tested. + +**The fix.** Build from the repository root: `uv run maturin develop --uv`. + +**When to check.** Before believing any Python-side result about Core behaviour, and always before +filing a Core defect from a Python test. + +## 2. A test helper that restarts a producer sequence + +**Symptom.** A Workflow appears to hang, the server accumulates tens of thousands of history +events, and the client's own event loop looks starved. It reads like a livelock between retention +and quiescence. + +**Why it happens.** `(session_id, sequence)` is the append idempotency key, and it is idempotent on +*identity*, not on the key alone. A helper like + +```python +async def publish(backend, key, values): + for i, value in enumerate(values): # restarts at 0 every call + await backend.append(key, StreamRecord(DATA, encode(value), "producer", i)) +``` + +re-uses `producer/0` on its second call with different content. The backend rejects it — +correctly; that rejection is a required conformance case. The record is never appended, the +Workflow waits for data that will never arrive, and the exception surfaces somewhere unrelated to +the Workflow that is now stuck. + +**Why it is insidious.** The visible symptom is on the Workflow side and looks like a coordination +bug, while the cause is an exception in the test's own producer several lines earlier. Two separate +test modules grew the same helper independently, and both were wrong the same way. + +**The fix.** Carry the sequence across calls, per producer session: + +```python +_sequences: dict[StreamKey, int] = {} + +async def publish(backend, key, values): + start = _sequences.get(key, 0) + for i, value in enumerate(values, start=start): + await backend.append(key, StreamRecord(DATA, encode(value), "producer", i)) + _sequences[key] = start + len(values) +``` + +**Related contract.** See `spec/backend-contract.md` on idempotent append, and ADR-020 on why +identity rather than the key alone. + +## 3. A required-test gate reading a stale copy of its own list + +**Symptom.** `test_m1_gate.py` passes, and reports the milestone fully covered, immediately after a +case was added to a required-test list and mapped to a new test. It would report the same thing if +the case had never been added. + +**Why it happens.** `m1_gate.py` resolves `PLAN_DIR` into the **vendored** Core checkout under +`temporalio/bridge/sdk-core`, not into the working tree these documents live in. That is deliberate — +reading the list from a copy inside the Python repo would let the two drift silently — but it means +the list the gate parses is whatever commit the submodule pointer names. Edit `required-tests/` in +the Core working tree and the gate does not see the edit at all. + +**Why it is insidious.** The failure the gate exists to produce is an *unmapped* case. The reverse, +a mapping for a case the parsed list does not contain, is caught by nothing: the count check compares +the parsed list against that same list's own heading, so both sides move together, and the coverage +check only walks cases the list contains. So a case mapped ahead of the pointer is not a pending +gate that will fail until the pointer moves — it is invisible, and the gate stays green without ever +arming the case. Mapping ahead is the normal order of work here, since the two repos commit +separately, which is exactly why the green result cannot be read as coverage. + +**The check.** Ask the gate what it counted, rather than whether it passed: + +```python +from tests.contrib.external_workflow_streams.m1_gate import PLAN_DIR, declared_count +print(PLAN_DIR, declared_count("tests-m1.md")) +``` + +If the count predates the case you just added, that case is not armed. Confirm the same way for the +Core-side lists before trusting a milestone figure. + +**The fix.** Move the submodule pointer to the Core commit that carries the list, then re-run the +gate and check the coverage figures changed. Checking the new lists out into the vendored tree first +is a cheap way to see what the pointer will arm before committing to it. + +**When to check.** Whenever a required-test list and its mapping are edited — which is every fix +that adds a case. + +## 4. A pytest-rewritten module imported into the Workflow sandbox + +**Symptom.** Worker construction fails intermittently during Workflow validation with a `KeyError` +inside CPython module-lock bookkeeping. It is more likely in a concurrent or complete suite than in +the focused test and occurs before the Core worker exists. + +**Why it happens.** A Workflow defined in a test module is re-imported by the sandbox. If pytest's +assertion rewriter transformed that module, the sandbox import also pulls `_pytest` implementation +modules and their process-global import machinery into a path whose isolation assumptions they do +not satisfy. The stack points at real sandbox and import-lock code, but the product behavior the test +intended to exercise has not started. + +**The fix.** A test module that defines sandboxed Workflows and constructs a `Worker` or `Replayer` +declares `PYTEST_DONT_REWRITE` in its module docstring, or moves the Workflow into a helper pytest +does not rewrite. A Workflow may instead be unsandboxed only when the test is not about sandbox +behavior. Do not diagnose a general sandbox defect from the pytest-specific path. + +## 5. Rejecting every Workflow Task failure in a terminal-race test + +**Symptom.** The Workflow returns the expected stream observations, yet a History assertion fails +because one Workflow Task was rejected with `UnhandledCommand`; replay may also add another +workflow-body execution. + +**Why it happens.** A wake Signal can race the terminal command. Temporal rejects the task so it can +preserve the external event, then replays. This is the lifecycle described in +`spec/wft-lifecycle.md`, not evidence that stream replay failed. + +**The fix.** Tests that deliberately create this race permit `UnhandledCommand` specifically while +rejecting every other cause. They compare every execution's observations with the live result and do +not require an exact body-start count. Tests that do not create the race should retain stricter +assertions. + +## 6. A controlled experiment that never observes its control stimulus + +**Symptom.** A fault-injection or wake-suppression variant changes the failure rate dramatically, but +its counters show that the event it was meant to isolate never reached the gate. The apparatus may +have intercepted the legitimate delivery wake or changed scheduling before the target condition. + +**The check.** Assert that the original stimulus reached the server or the exact production boundary +before accepting any experimental run. Count original inputs separately from follow-ups. Apply the +same tracing to control and treatment; instrumentation on only one side makes scheduling part of the +comparison. On timeout, retain History and correlated runtime state before cleanup, and use long +tracebacks so the first failure is not reduced to a one-line table. + +**The rule.** A treatment that did not observe its control stimulus is void, no matter how stable its +result looks. + +## 7. Worker tasks that outlive their test + +**Symptom.** A later test reports pending-task destruction, shutdown warnings, or scheduling-sensitive +failures whose originating Worker was constructed by an earlier case. + +**Why it happens.** Constructing a `Worker` creates a real Core worker and integration fixtures often +start worker and shutdown coroutines separately. Dropping the Python objects does not make those +lifetimes orderly. + +**The fix.** Successful constructor tests use `async with Worker(...)`. Fixtures retain every worker +and shutdown task and await them during teardown. Constructor-failure tests remain direct because no +Worker exists to close when validation raises. + +## Before reporting a defect found by a test + +1. Confirm the binary under test is the one you built (hazard 1). +2. Confirm the test's own setup succeeded — an exception in a producer or fixture can present as a + hang in the system under test (hazard 2). +3. Confirm required-test lists and their mappings come from the same submodule revision (hazard 3). +4. Keep pytest rewriting and leaked Worker lifetimes out of Workflow tests (hazards 4 and 7). +5. Classify `UnhandledCommand` against the event race the test created (hazard 5). +6. Prove a controlled experiment observed its control stimulus (hazard 6). +7. Reproduce at the smallest layer that shows it. If a Core-level test cannot reproduce what an + end-to-end test shows, suspect the apparatus before concluding the layers disagree. +8. Mutation-test the claim: break the code path you believe is at fault and confirm *that* test + fails. If nothing fails, the test was not covering what you thought — and if the test already + passes without the mechanism, the defect is not where you think it is. + +The last step also covers hazard 3, and it is worth stating the general form: **a green +result is a claim about what ran, and nothing checks that for you.** Before reading a pass as +coverage, make the covering mechanism fail once. diff --git a/crates/protos/protos/local/temporal/sdk/core/core_interface.proto b/crates/protos/protos/local/temporal/sdk/core/core_interface.proto index 2173e55d3..d2a1ebcc7 100644 --- a/crates/protos/protos/local/temporal/sdk/core/core_interface.proto +++ b/crates/protos/protos/local/temporal/sdk/core/core_interface.proto @@ -14,6 +14,7 @@ import "temporal/sdk/core/activity_result/activity_result.proto"; import "temporal/sdk/core/activity_task/activity_task.proto"; import "temporal/sdk/core/common/common.proto"; import "temporal/sdk/core/external_data/external_data.proto"; +import "temporal/sdk/core/external_stream/external_stream.proto"; import "temporal/sdk/core/workflow_activation/workflow_activation.proto"; import "temporal/sdk/core/workflow_commands/workflow_commands.proto"; import "temporal/sdk/core/workflow_completion/workflow_completion.proto"; diff --git a/crates/protos/protos/local/temporal/sdk/core/external_data/external_data.proto b/crates/protos/protos/local/temporal/sdk/core/external_data/external_data.proto index 18bee02b8..2e0fee4d3 100644 --- a/crates/protos/protos/local/temporal/sdk/core/external_data/external_data.proto +++ b/crates/protos/protos/local/temporal/sdk/core/external_data/external_data.proto @@ -35,4 +35,97 @@ message PatchedMarkerData { string id = 1; // Whether or not the patch is marked deprecated. bool deprecated = 2; -} \ No newline at end of file +} + +// Why a Workflow Task holding external stream waits ended. Core knows the reason on every +// completion path -- it either decided the boundary itself or was told -- which is why this +// lives here, in the Core-readable marker envelope, and is deliberately *not* duplicated inside +// the opaque replay annotation where a second copy could disagree with this one. +enum ParkReason { + PARK_REASON_UNSPECIFIED = 0; + // The global quiescence timer expired with no record on any active wait. + PARK_REASON_IDLE = 1; + // Every active wait in the quiescent snapshot reached a write fence with no later record + // immediately available, so the idle delay was bypassed. + PARK_REASON_ALL_WRITE_FENCED = 2; + // The Worker is shutting down or the Run is being evicted while a Workflow Task is open. + PARK_REASON_SHUTDOWN = 3; + // The Workflow Task rollover deadline expired. Core decided this one, so the terminal came + // back through FinalizeExternalStreams. + PARK_REASON_ROLLOVER = 4; + // The annotation reached its byte budget high-water mark and lang asked for a rollover. Unlike + // PARK_REASON_ROLLOVER this needs no finalization round trip: the triggering + // WorkflowStreamProgress already carried the terminal. + PARK_REASON_BUDGET_ROLLOVER = 5; + // The completion carried server-bound commands (timer, activity, child workflow, signal), so + // the Workflow Task completed normally with subscriptions left active and unparked. + PARK_REASON_COMMANDS_PRODUCED = 6; + // The completion carried a terminal command -- complete, fail, cancel, or continue-as-new. + PARK_REASON_WORKFLOW_COMPLETED = 7; + // The Workflow Task completed normally with nothing else to report: records were consumed and + // no stream wait remained pending. Distinct from PARK_REASON_COMMANDS_PRODUCED so the marker + // says which of the two it actually was rather than implying commands that never existed. + PARK_REASON_TASK_COMPLETED = 8; + // The Workflow Task's output visibility deadline expired. + PARK_REASON_OUTPUT_LATENCY = 9; + // The staged output reached its deterministic record or logical-byte capacity. + PARK_REASON_OUTPUT_CAPACITY = 10; +} + +// What Core writes into a marker for a Workflow Task that consumed external streams. Exactly one +// of these is written per Workflow Task, however many progress reports that task carried. +message ExternalStreamMarkerData { + uint32 schema_version = 1; + // Identifies the complete blocked snapshot this marker closes. + uint64 quiescence_generation = 2; + repeated ExternalWaitMarker waits = 3; + // The accumulated replay annotation, opaque to Core. Core appends each observation delta to + // this and never parses any of it. + bytes replay_annotation = 4; + ParkReason terminal_boundary = 5; + // The output staged for this Workflow Task, if any. Output shares this marker with input so + // replay has one activation-segment schedule and never drains the Workflow event loop twice. + ExternalOutputStreamManifest output = 6; +} + +message ExternalWaitMarker { + uint32 wait_id = 1; + uint64 generation = 2; +} + +// Compact proof of the externally staged output belonging to one Workflow Task. +// +// This message contains no record payloads. The stage token identifies the immutable pending +// batch in the provider, while the remaining fields let replay validate the deterministic logical +// output and let a cold reader reconcile the pending batch against History. +message ExternalOutputStreamManifest { + uint32 schema_version = 1; + uint32 fingerprint_version = 2; + string stage_token = 3; + // The event immediately preceding this Workflow Task's WorkflowTaskScheduled event in the + // ordered History view used to build its activation. + int64 history_floor_event_id = 4; + string run_id = 5; + repeated ExternalOutputTopicManifest topics = 6; + // One entry per live activation segment, including segments which published no output. + repeated ExternalOutputSegmentManifest segments = 7; + // Stable configured provider binding. Reconciliation refuses to interpret the batch through a + // different provider or provider wire format. + string provider_id = 8; + uint32 provider_format_version = 9; +} + +// Logical, pre-codec identity of one topic sub-batch. +message ExternalOutputTopicManifest { + string topic = 1; + uint32 record_count = 2; + uint64 logical_byte_count = 3; + bytes logical_fingerprint = 4; + bool finished = 5; +} + +// Record counts for one activation segment. Entries correspond positionally to `topics` in the +// enclosing ExternalOutputStreamManifest, retaining empty segments without repeating topic names. +message ExternalOutputSegmentManifest { + repeated uint32 record_counts_by_topic = 1; +} diff --git a/crates/protos/protos/local/temporal/sdk/core/external_stream/external_stream.proto b/crates/protos/protos/local/temporal/sdk/core/external_stream/external_stream.proto new file mode 100644 index 000000000..4204394c0 --- /dev/null +++ b/crates/protos/protos/local/temporal/sdk/core/external_stream/external_stream.proto @@ -0,0 +1,41 @@ +syntax = "proto3"; + +/* + * Wire formats for External Workflow Streams that are neither commands nor activation jobs. + * + * The one member today is the reserved wake Signal's envelope. It lives here, at the protocol + * level, rather than being a user payload: Core must read its fields, and Core has no access to + * the user's DataConverter or codec. A user codec that encrypts payloads would otherwise make the + * envelope unreadable to the component that must read it. + */ +package coresdk.external_stream; +option ruby_package = "Temporalio::Internal::Bridge::Api::ExternalStream"; + +// The payload of the reserved `__temporal_external_stream_wake` Signal. +// +// Sent as a single argument whose Payload uses metadata +// `encoding = "binary/protobuf"` and +// `messageType = "coresdk.external_stream.WakeSignal"`, serialized with the protocol's own +// serialization rather than through the user's DataConverter. +// +// The Signal carries no user data, so bypassing the codec leaks nothing. +message WakeSignal { + // Starts at 1. Core ignores an unknown version harmlessly rather than failing -- an old Core + // must not break because a newer producer learned a new field. + uint32 envelope_version = 1; + // The stream this wake was prompted by. Only a hint: on wakeup the runtime rechecks every + // active subscription, not just this one. + string stream_name = 2; + uint32 wait_id = 3; + // 0 is reserved and means "no park generation" -- an unparked wake, which Core accepts as a + // recheck request after validating chain identity. Park generations are quiescence + // generations, which start at 1, so a *non-zero* generation the current Run does not + // recognize is a claim that turned out to be wrong and is ignored as stale. + uint64 park_generation = 4; + // Chain identity, not the current Run. The Signal is addressed to the Workflow ID without a + // Run ID so it always lands on the current Run of the chain; this field is what lets Core + // tell "same chain, stale generation" from "different chain". + string first_execution_run_id = 5; + // Diagnostics only. Never used for validation or deduplication. + string producer_session_id = 6; +} diff --git a/crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto b/crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto index 7287ab1c1..6462811ac 100644 --- a/crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto +++ b/crates/protos/protos/local/temporal/sdk/core/workflow_activation/workflow_activation.proto @@ -19,6 +19,8 @@ import "temporal/sdk/core/activity_result/activity_result.proto"; import "temporal/sdk/core/child_workflow/child_workflow.proto"; import "temporal/sdk/core/common/common.proto"; import "temporal/sdk/core/nexus/nexus.proto"; +import "temporal/sdk/core/external_data/external_data.proto"; +import "temporal/sdk/core/workflow_commands/workflow_commands.proto"; // An instruction to the lang sdk to run some workflow code, whether for the first time or from // a cached state. @@ -103,6 +105,10 @@ message WorkflowActivation { // the workflow is Pinned. // Experimental. bool target_worker_deployment_version_changed = 12; + // The event immediately preceding this Workflow Task's WorkflowTaskScheduled event in the + // ordered History view Core used to build the activation. Zero means Core could not identify + // that exact predecessor, in which case output must not be staged for this task. + int64 history_floor_event_id = 13; } message WorkflowActivationJob { @@ -140,6 +146,15 @@ message WorkflowActivationJob { ResolveNexusOperationStart resolve_nexus_operation_start = 15; // A nexus operation resolved. ResolveNexusOperation resolve_nexus_operation = 16; + // One or more external stream waits may have data buffered. Runs user workflow code. + ResolveExternalStreamWaits resolve_external_stream_waits = 17; + // Runtime-internal: install park intents and recheck every stream. Runs no user code. + PrepareExternalStreamPark prepare_external_stream_park = 18; + // Replay: deliver the recorded external stream observations from a marker. + ReplayExternalStreams replay_external_streams = 19; + // Runtime-internal: encode the terminal boundary for a marker Core is about to write. + // Runs no user code. + FinalizeExternalStreams finalize_external_streams = 20; // Remove the workflow identified by the [WorkflowActivation] containing this job from the // cache after performing the activation. It is guaranteed that this will be the only job // in the activation if present. @@ -147,6 +162,63 @@ message WorkflowActivationJob { } } +// Tells lang that one or more external stream waits may now have data. +// +// This job contains no records. The listed waits are readiness *hints*, not an exhaustive +// availability claim: on receipt lang probes every active wait, drains all currently available +// inputs, and only then resumes Workflow futures. +message ResolveExternalStreamWaits { + uint64 quiescence_generation = 1; + repeated coresdk.workflow_commands.ExternalStreamWait ready_hints = 2; +} + +// Asks lang to park the complete active wait set through the backend handshake. +// +// Runtime-internal: no user Workflow code runs. Lang installs a park intent per subscription, +// rechecks every stream, and answers with ExternalStreamParkResult. A recheck that finds records +// answers `became_ready` instead, and Core issues a normal resolve activation next. +// +// Requires backend I/O, so lang must not handle this on the synchronous Workflow thread. +message PrepareExternalStreamPark { + uint64 quiescence_generation = 1; + // Always the complete active wait set. + repeated coresdk.workflow_commands.ExternalStreamWait waits = 2; + // One of PARK_REASON_IDLE, PARK_REASON_ALL_WRITE_FENCED, or PARK_REASON_SHUTDOWN. + coresdk.external_data.ParkReason reason = 3; +} + +// Carries a recorded marker's annotation back to lang during replay. +// +// Core issues exactly one of these per marker. Lang's replay driver walks the annotation's +// segments in order, performing one event-loop drain per segment, so the live run's k +// activations become k drains inside this one activation and condition evaluation matches. +// +// Requires backend I/O (the recorded range reads and their integrity validation), so lang must +// prepare it off the Workflow thread and deliver from memory once it is on it. +message ReplayExternalStreams { + uint64 quiescence_generation = 1; + repeated coresdk.workflow_commands.ExternalStreamWait waits = 2; + // Opaque to Core. + bytes replay_annotation = 3; + coresdk.external_data.ParkReason terminal_boundary = 4; + // The output manifest recorded in the same marker, if this Workflow Task published output. + coresdk.external_data.ExternalOutputStreamManifest output = 5; +} + +// Issued before Core writes a marker for a boundary Core itself decided and no park handshake +// will run: rollover-deadline expiry, and shutdown or eviction with a Workflow Task open. +// +// Runs no user Workflow code, cannot resolve futures, and its only legal responses are +// ExternalStreamFinalized or an activation failure. It requires *no* backend I/O: lang reads the +// blocked cursor snapshot from its own in-memory manager state. If lang cannot answer, Core +// writes no marker and the Workflow Task fails for retry -- there is no best-effort path. +message FinalizeExternalStreams { + uint64 quiescence_generation = 1; + repeated coresdk.workflow_commands.ExternalStreamWait waits = 2; + // One of PARK_REASON_ROLLOVER or PARK_REASON_SHUTDOWN. + coresdk.external_data.ParkReason reason = 3; +} + // Initialize a new workflow message InitializeWorkflow { // The identifier the lang-specific sdk uses to execute workflow code diff --git a/crates/protos/protos/local/temporal/sdk/core/workflow_commands/workflow_commands.proto b/crates/protos/protos/local/temporal/sdk/core/workflow_commands/workflow_commands.proto index f36c84cef..20dd18490 100644 --- a/crates/protos/protos/local/temporal/sdk/core/workflow_commands/workflow_commands.proto +++ b/crates/protos/protos/local/temporal/sdk/core/workflow_commands/workflow_commands.proto @@ -19,6 +19,7 @@ import "temporal/api/sdk/v1/event_group_marker.proto"; import "temporal/sdk/core/child_workflow/child_workflow.proto"; import "temporal/sdk/core/nexus/nexus.proto"; import "temporal/sdk/core/common/common.proto"; +import "temporal/sdk/core/external_data/external_data.proto"; message WorkflowCommand { // User metadata that may or may not be persisted into history depending on the command type. @@ -54,9 +55,101 @@ message WorkflowCommand { UpdateResponse update_response = 20; ScheduleNexusOperation schedule_nexus_operation = 21; RequestCancelNexusOperation request_cancel_nexus_operation = 22; + WorkflowStreamProgress workflow_stream_progress = 23; + WorkflowStreamQuiescent workflow_stream_quiescent = 24; + ExternalStreamParkResult external_stream_park_result = 25; + ExternalStreamFinalized external_stream_finalized = 26; + WorkflowOutputStreamCommit workflow_output_stream_commit = 27; + WorkflowOutputStreamBuffered workflow_output_stream_buffered = 28; } } +// Commits an observation delta for external streams. +// +// Emitted on *every* completion path where replay-visible stream state changed -- which includes +// an activation that observed no records at all -- and independent of whether the Workflow Task +// is retained. Consuming a record and committing that consumption are separate steps, and the +// second is not conditional on why the Workflow Task ended. +// +// Ordering is normative: this command precedes every command whose value could depend on the +// consumed data. On replay that guarantees integrity validation for a record runs before the +// command derived from it is matched. +message WorkflowStreamProgress { + // The segments produced since the previous progress report for this Workflow Task. Opaque to + // Core, which appends it to the accumulated annotation and never parses it. + bytes observation_delta = 1; + // Set when the encoder is approaching the annotation byte budget and the runtime wants the + // Workflow Task rolled over rather than the marker grown further. Core takes the rollover + // path minus the finalization round trip, because this command already carries the terminal. + bool request_rollover = 2; +} + +// Asks Core to retain the open Workflow Task. Carries no annotation data -- retention and +// progress are separate questions, and a completion may carry either command, both, or neither. +message WorkflowStreamQuiescent { + // Identifies this complete blocked snapshot. + uint64 quiescence_generation = 1; + // The *complete* set of waits the Workflow is blocked on. A partial set would let one idle + // stream park a Workflow Task another stream is still driving. + repeated ExternalStreamWait waits = 2; + // Already reduced by `min` over the quiescent set in wait_id order. Core rejects a + // non-positive value as a malformed completion rather than coercing it. + google.protobuf.Duration idle_timeout = 3; +} + +// One external stream wait. Shared by the commands above and by the activation jobs, so it takes +// no command tag of its own. +message ExternalStreamWait { + uint32 wait_id = 1; + // Increments each time this subscription re-enters the blocked state. Never leaves the + // Core/lang boundary. + uint64 generation = 2; + // Normally set after a write fence. A single fenced stream does not park the Workflow Task; + // Core bypasses the idle delay only when *every* active wait in the snapshot is set. + bool immediately_parkable = 3; +} + +// Lang's answer to PrepareExternalStreamPark. +message ExternalStreamParkResult { + uint64 quiescence_generation = 1; + oneof outcome { + ParkSetConfirmed confirmed = 2; + StreamSetBecameReady became_ready = 3; + } + // Terminal observation delta for the marker Core is about to write. Present on `confirmed`: + // only lang can encode the boundary, and Core must not write a marker whose annotation has + // no terminal. + bytes final_observation_delta = 4; +} + +// Every stream was still empty after intents were installed and every stream was rechecked. +message ParkSetConfirmed {} + +// A recheck found records, so this parking generation is aborted and Core issues a normal +// resolve activation rather than running user code from inside the park path. +message StreamSetBecameReady {} + +// Lang's answer to FinalizeExternalStreams, for the paths where Core decides the boundary and no +// park handshake runs. +message ExternalStreamFinalized { + uint64 quiescence_generation = 1; + bytes final_observation_delta = 2; +} + +// Reports that the output batch for this Workflow Task has been durably staged outside History. +// Core records only this compact manifest in the shared external-stream marker. +message WorkflowOutputStreamCommit { + coresdk.external_data.ExternalOutputStreamManifest manifest = 1; + // The deterministic output capacity was reached and this task must request a replacement. + bool request_rollover = 2; +} + +// Reports that external output is buffered in lang but has not yet been staged. Core retains the +// open Workflow Task until the earliest reported deadline and then asks lang to finalize it. +message WorkflowOutputStreamBuffered { + google.protobuf.Duration max_publish_latency = 1; +} + message StartTimer { // Lang's incremental sequence number, used as the operation identifier uint32 seq = 1; diff --git a/crates/protos/src/protos/constants.rs b/crates/protos/src/protos/constants.rs index 9d0c800b0..09527ab3f 100644 --- a/crates/protos/src/protos/constants.rs +++ b/crates/protos/src/protos/constants.rs @@ -5,3 +5,10 @@ pub const PATCH_MARKER_NAME: &str = "core_patch"; /// Used as `marker_name` field when recording local activity markers pub const LOCAL_ACTIVITY_MARKER_NAME: &str = "core_local_activity"; + +/// Used as `marker_name` field when recording External Workflow Stream markers +/// +/// Deliberately not in the `core_workflow_stream*` namespace: this feature coexists with the +/// shipped `temporalio.contrib.workflow_streams` rather than replacing it, and the two must not +/// collide in History either. +pub const EXTERNAL_STREAM_MARKER_NAME: &str = "core_external_stream"; diff --git a/crates/protos/src/protos/mod.rs b/crates/protos/src/protos/mod.rs index 39733b600..b09a117c4 100644 --- a/crates/protos/src/protos/mod.rs +++ b/crates/protos/src/protos/mod.rs @@ -1000,6 +1000,48 @@ pub mod coresdk { } pub mod external_data { tonic::include_proto!("coresdk.external_data"); + pub use self::stream_marker_helpers::*; + mod stream_marker_helpers { + use super::ExternalStreamMarkerData; + use crate::protos::temporal::api::common::v1::{Payload, Payloads}; + use prost::Message; + use std::collections::HashMap; + + /// The key the marker's details map stores the envelope under. + const DETAILS_KEY: &str = "external_stream"; + + /// Packs the marker envelope into a `RecordMarker` details map. + /// + /// Encoded as proto binary rather than JSON: the replay annotation inside is opaque + /// bytes of arbitrary size, and JSON-encoding it would inflate every marker for the + /// sake of a UI rendering nobody can read anyway. + pub fn build_external_stream_marker_details( + data: &ExternalStreamMarkerData, + ) -> HashMap { + let payload = Payload { + metadata: HashMap::from([( + "encoding".to_string(), + b"binary/protobuf".to_vec(), + )]), + data: data.encode_to_vec(), + ..Default::default() + }; + HashMap::from([( + DETAILS_KEY.to_string(), + Payloads { + payloads: vec![payload], + }, + )]) + } + + /// Reads the envelope back out, or `None` if this is not one of ours. + pub fn extract_external_stream_marker_data( + details: &HashMap, + ) -> Option { + let payload = details.get(DETAILS_KEY)?.payloads.first()?; + ExternalStreamMarkerData::decode(payload.data.as_slice()).ok() + } + } mod sdk_helpers { use prost_types::{Duration, Timestamp}; use serde::{Deserialize, Deserializer, Serialize, Serializer}; @@ -1081,6 +1123,28 @@ pub mod coresdk { } } } + pub mod external_stream { + tonic::include_proto!("coresdk.external_stream"); + + /// The reserved Signal name external stream wakeups are sent under. + /// + /// Core intercepts this before user Signal dispatch and suppresses it from user handlers + /// whether or not the envelope validates, so an unknown version or a stale generation can + /// never reach Workflow code as an unhandled Signal. + pub const WAKE_SIGNAL_NAME: &str = "__temporal_external_stream_wake"; + + /// The `messageType` metadata a [`WakeSignal`] payload carries. + pub const WAKE_SIGNAL_MESSAGE_TYPE: &str = "coresdk.external_stream.WakeSignal"; + + /// The only envelope version Core understands today. + pub const WAKE_SIGNAL_ENVELOPE_VERSION: u32 = 1; + + /// Reserved `park_generation` meaning "the sender knows of no confirmed park". + /// + /// Park generations are quiescence generations, which start at 1, so this value cannot + /// collide with a real one. + pub const UNPARKED_WAKE_GENERATION: u64 = 0; + } pub mod workflow_activation { tonic::include_proto!("coresdk.workflow_activation"); pub use self::sdk_helpers::*; @@ -1130,6 +1194,7 @@ pub mod coresdk { last_sdk_version: String::new(), suggest_continue_as_new_reasons: vec![], target_worker_deployment_version_changed: false, + history_floor_event_id: 0, } } @@ -1277,6 +1342,38 @@ pub mod coresdk { workflow_activation_job::Variant::ResolveNexusOperation(_) => { write!(f, "ResolveNexusOperation") } + workflow_activation_job::Variant::ResolveExternalStreamWaits(r) => { + write!( + f, + "ResolveExternalStreamWaits({}, {} hint(s))", + r.quiescence_generation, + r.ready_hints.len() + ) + } + workflow_activation_job::Variant::PrepareExternalStreamPark(p) => { + write!( + f, + "PrepareExternalStreamPark({}, {:?})", + p.quiescence_generation, + p.reason() + ) + } + workflow_activation_job::Variant::ReplayExternalStreams(r) => { + write!( + f, + "ReplayExternalStreams({}, {} byte(s))", + r.quiescence_generation, + r.replay_annotation.len() + ) + } + workflow_activation_job::Variant::FinalizeExternalStreams(fin) => { + write!( + f, + "FinalizeExternalStreams({}, {:?})", + fin.quiescence_generation, + fin.reason() + ) + } } } } @@ -1643,6 +1740,88 @@ pub mod coresdk { } } + impl Display for WorkflowStreamProgress { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "WorkflowStreamProgress({} byte(s){})", + self.observation_delta.len(), + if self.request_rollover { + ", rollover requested" + } else { + "" + } + ) + } + } + + impl Display for WorkflowStreamQuiescent { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "WorkflowStreamQuiescent({}, {} wait(s))", + self.quiescence_generation, + self.waits.len() + ) + } + } + + impl Display for ExternalStreamParkResult { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + let outcome = match self.outcome { + Some(external_stream_park_result::Outcome::Confirmed(_)) => "confirmed", + Some(external_stream_park_result::Outcome::BecameReady(_)) => { + "became ready" + } + None => "no outcome", + }; + write!( + f, + "ExternalStreamParkResult({}, {})", + self.quiescence_generation, outcome + ) + } + } + + impl Display for ExternalStreamFinalized { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "ExternalStreamFinalized({}, {} byte(s))", + self.quiescence_generation, + self.final_observation_delta.len() + ) + } + } + + impl Display for WorkflowOutputStreamCommit { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "WorkflowOutputStreamCommit({} topic(s){})", + self.manifest + .as_ref() + .map(|manifest| manifest.topics.len()) + .unwrap_or_default(), + if self.request_rollover { + ", rollover requested" + } else { + "" + } + ) + } + } + + impl Display for WorkflowOutputStreamBuffered { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "WorkflowOutputStreamBuffered({:?})", + self.max_publish_latency + ) + } + } + impl QueryResult { /// Helper to construct the Temporal API query result types. pub fn into_components( diff --git a/crates/sdk-core/CHANGELOG.md b/crates/sdk-core/CHANGELOG.md index 7291644cb..aff9ce949 100644 --- a/crates/sdk-core/CHANGELOG.md +++ b/crates/sdk-core/CHANGELOG.md @@ -54,6 +54,9 @@ relevant information. failure cause alongside the existing failure. * Language SDKs can opt in to recording local activity arguments in the local activity marker's `input` detail. +* Added the Core protocol for replay-safe Workflow-originated external stream output, including + exact Workflow Task History floors, compact staged-output marker proofs, and shared input/output + replay segmentation. * Core console logs can now be emitted as newline-delimited JSON when an SDK selects the JSON log format. Configured log filters continue to apply to JSON output. * Workflow completion-as-cancelled commands can now carry details for recording on the terminal @@ -106,6 +109,8 @@ relevant information. * Workers now warn when autoscaling task polling encounters errors continuously for one minute. Repeated warnings use exponential backoff up to 15-minute intervals and stop after polling recovers. +* Workers now defensively buffer a replacement workflow task if it reaches a run that still owns + one, preserving the outstanding task token in release builds. * Workers no longer send worker heartbeats or appear in centralized heartbeat reports before they begin polling. * Ephemeral server processes no longer leak on failed start. diff --git a/crates/sdk-core/src/core_tests/external_streams.rs b/crates/sdk-core/src/core_tests/external_streams.rs new file mode 100644 index 000000000..296379223 --- /dev/null +++ b/crates/sdk-core/src/core_tests/external_streams.rs @@ -0,0 +1,6258 @@ +//! External Workflow Stream routing and entry points (C3, C4). +//! +//! These drive the real serialized local-input lane through a mock worker, so what is under test +//! is the routing -- that an input reaches the right run, that an acknowledgement comes back, and +//! that the read-only probe is genuinely read-only. The wait set's own transition logic is +//! covered by unit tests next to it. + +use crate::{ + ExternalStreamReadyResult, ExternalStreamRunStatus, PollError, + replay::{DEFAULT_ACTIVITY_TYPE, TestHistoryBuilder, canned_histories}, + test_help::{ + MockPollCfg, PollWFTRespExt, ResponseType, WorkerExt, build_fake_worker, + build_mock_pollers, hist_to_poll_resp, mock_worker, query_ok, schedule_activity_cmd, + start_timer_cmd, + }, + worker::client::{WorkflowTaskCompletion, mocks::mock_worker_client}, +}; +use parking_lot::Mutex; +use prost::Message as _; +use std::{ + collections::{HashMap, HashSet}, + sync::{ + Arc, + atomic::{AtomicBool, AtomicUsize, Ordering}, + }, + time::Duration, +}; +use temporalio_common::{ + protos::{ + constants::EXTERNAL_STREAM_MARKER_NAME, + coresdk::{ + external_data::{ + ExternalOutputSegmentManifest, ExternalOutputStreamManifest, + ExternalOutputTopicManifest, ExternalStreamMarkerData, ParkReason, + extract_external_stream_marker_data, + }, + external_stream::{self, WakeSignal}, + workflow_activation::{WorkflowActivation, workflow_activation_job}, + workflow_commands::{ + ActivityCancellationType, CompleteWorkflowExecution, + ContinueAsNewWorkflowExecution, ExternalStreamFinalized, ExternalStreamParkResult, + ExternalStreamWait, FailWorkflowExecution, ParkSetConfirmed, ScheduleActivity, + StreamSetBecameReady, UpdateResponse, WorkflowOutputStreamBuffered, + WorkflowOutputStreamCommit, WorkflowStreamProgress, WorkflowStreamQuiescent, + external_stream_park_result, update_response::Response as UpdateOutcome, + workflow_command, + }, + workflow_completion::WorkflowActivationCompletion, + }, + temporal::api::{ + command::v1::command, + common::v1::Payload, + enums::v1::{CommandType, EventType}, + query::v1::WorkflowQuery, + workflowservice::v1::RespondWorkflowTaskCompletedResponse, + }, + }, + worker::WorkerTaskTypes, +}; + +/// One recorded external stream marker: the snapshot it closed, its terminal, its annotation. +type StreamMarker = (u64, ParkReason, Vec); +/// Markers collected across a test's completions, in the order they were reported. +type StreamMarkers = Arc>>; + +/// A worker holding one run in its cache, with the first activation left outstanding. +/// +/// Leaving it outstanding is what keeps the run cached for the whole test: completing it would +/// let the mock's poll budget run out and the run be evicted, which is a different state from the +/// one these tests are about. +async fn worker_with_a_cached_run() -> (crate::Worker, String) { + let t = canned_histories::single_timer("1"); + let worker = build_fake_worker("fake_wf_id", t, [1]); + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + (worker, run_id) +} + +/// Completes the outstanding activation and shuts the worker down cleanly. +async fn finish(worker: crate::Worker, run_id: &str) { + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.to_string(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +// --- readiness --------------------------------------------------------------- + +#[tokio::test] +async fn readiness_for_an_unknown_run_reports_run_not_found() { + let (worker, run_id) = worker_with_a_cached_run().await; + + let result = worker + .notify_external_stream_ready("no-such-run", 1, 0) + .await; + + assert_eq!(result, ExternalStreamReadyResult::RunNotFound); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn readiness_for_a_cached_run_with_no_open_task_is_not_run_not_found() { + // The distinction this asserts is the whole reason the two results are separate: a run cached + // between Workflow Tasks is healthy, and telling the watcher it was evicted would make it + // tear itself down while it is still needed. + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, vec![1], None, false) + .await; + + let result = worker.notify_external_stream_ready(&run_id, 1, 0).await; + + assert_eq!(result, ExternalStreamReadyResult::NoOpenWorkflowTask); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn readiness_for_a_confirmed_park_reports_parked() { + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, vec![1], Some(7), false) + .await; + + let result = worker.notify_external_stream_ready(&run_id, 1, 0).await; + + assert_eq!(result, ExternalStreamReadyResult::Parked); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn readiness_at_the_current_generation_is_accepted() { + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, vec![1, 2], None, true) + .await; + + let result = worker.notify_external_stream_ready(&run_id, 1, 0).await; + + assert_eq!(result, ExternalStreamReadyResult::Accepted); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn a_stale_wait_generation_is_reported_as_stale() { + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, vec![1], None, true) + .await; + + let result = worker.notify_external_stream_ready(&run_id, 1, 99).await; + + // Not `NoOpenWorkflowTask`: the wait exists, so the watcher should re-probe rather than + // send a Signal for a block that has already been resolved. + assert_eq!(result, ExternalStreamReadyResult::Stale); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn concurrent_readiness_calls_are_all_acknowledged() { + // The call must be safe from several watcher tasks at once -- one per subscription is the + // normal case. + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, (1..=8).collect(), None, true) + .await; + + let results = futures_util::future::join_all( + (1..=8u32).map(|wait_id| worker.notify_external_stream_ready(&run_id, wait_id, 0)), + ) + .await; + + assert!( + results + .iter() + .all(|r| *r == ExternalStreamReadyResult::Accepted), + "expected every concurrent notification to be accepted, got {results:?}" + ); + finish(worker, &run_id).await; +} + +// --- the read-only status probe ---------------------------------------------- + +#[tokio::test] +async fn status_for_an_unknown_run_reports_run_not_found() { + let (worker, run_id) = worker_with_a_cached_run().await; + + assert_eq!( + worker.external_stream_run_status("no-such-run").await, + ExternalStreamRunStatus::RunNotFound + ); + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn status_distinguishes_the_three_cached_states() { + let (worker, run_id) = worker_with_a_cached_run().await; + + // Cached with no waits at all -- nothing for the sweep to do. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask + ); + + worker + .seed_external_stream_waits(&run_id, vec![1], None, true) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + + worker + .seed_external_stream_waits(&run_id, vec![1], Some(3), false) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked + ); + + finish(worker, &run_id).await; +} + +#[tokio::test] +async fn the_status_probe_leaves_the_run_untouched() { + // Probing must not be usable as a readiness claim by accident: no activation may be + // manufactured and no state may move, however many times it is asked. + let (worker, run_id) = worker_with_a_cached_run().await; + worker + .seed_external_stream_waits(&run_id, vec![1], None, true) + .await; + + for _ in 0..5 { + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + } + + // Readiness still behaves as though nothing had happened -- in particular the wait is still + // at generation 0 and still blocked, not consumed by the probes. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + + finish(worker, &run_id).await; +} + +// --- idle timeout routing (C3) ----------------------------------------------- + +#[tokio::test] +async fn an_idle_timeout_for_an_unknown_run_is_harmless() { + let (worker, run_id) = worker_with_a_cached_run().await; + + worker.notify_external_stream_idle_timeout("no-such-run", 1); + + finish(worker, &run_id).await; +} + +// --- the run-level rollover timer (C13) -------------------------------------- + +/// The gap ADR-017 names, as a test. +/// +/// A worker registering workflows and no activities has no local-activity request sink, and the +/// old `sink_heartbeat_timeout_start` silently returned a handle to a timer that was never +/// started. That is exactly the worker external streams must support, so the rollover deadline +/// has to come from the run rather than from the local-activity subsystem. +#[tokio::test] +async fn the_rollover_deadline_fires_on_a_workflow_only_worker() { + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + let saw_force_new_wft = Arc::new(AtomicBool::new(false)); + let recorder = saw_force_new_wft.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + recorder.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + }); + + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + // No activity task types at all, so `enable_local_activities` is false and there is no + // request sink for a timer to be pushed into. + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + assert!( + !worker.get_config().task_types.enable_local_activities, + "this test is only meaningful without a local-activity sink" + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .start_wft_rollover_timer(&run_id, Duration::from_millis(50)) + .await; + // Long enough for the deadline (80% of 50ms) to have passed. + tokio::time::sleep(Duration::from_millis(200)).await; + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + assert!( + saw_force_new_wft.load(Ordering::Relaxed), + "the rollover deadline expired but the completion did not request a replacement task" + ); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_cancelled_rollover_timer_does_not_force_a_new_task() { + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + let saw_force_new_wft = Arc::new(AtomicBool::new(false)); + let recorder = saw_force_new_wft.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + recorder.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + }); + + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // A deadline far enough out that it cannot have fired by the time we complete. + worker + .start_wft_rollover_timer(&run_id, Duration::from_secs(300)) + .await; + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + assert!( + !saw_force_new_wft.load(Ordering::Relaxed), + "a rollover deadline that has not expired must not force a replacement task" + ); + worker.drain_pollers_and_shutdown().await; +} + +// --- quiescence retains the Workflow Task (C6) ------------------------------- + +fn quiescent_command( + quiescence_generation: u64, + wait_ids: &[u32], + idle_timeout: Duration, +) -> workflow_command::Variant { + workflow_command::Variant::WorkflowStreamQuiescent(WorkflowStreamQuiescent { + quiescence_generation, + waits: wait_ids + .iter() + .map(|id| ExternalStreamWait { + wait_id: *id, + generation: 0, + immediately_parkable: false, + }) + .collect(), + idle_timeout: Some(idle_timeout.try_into().unwrap()), + }) +} + +/// A worker whose completions are all recorded, so a test can assert that a retained task +/// reported *nothing* to the server. +fn worker_recording_completions(completions: Arc) -> crate::Worker { + worker_counting_completions(completions, canned_histories::single_timer("1"), vec![1]) +} + +/// The same, over a history and batch schedule the test chooses. +fn worker_counting_completions( + completions: Arc, + history: TestHistoryBuilder, + batches: Vec, +) -> crate::Worker { + let t = history; + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, batches, mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let counter = completions.clone(); + asserts.then(move |_| { + counter.fetch_add(1, Ordering::Relaxed); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + mock_worker(mock) +} + +#[tokio::test] +async fn quiescence_holds_the_workflow_task_open() { + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + )) + .await + .unwrap(); + + assert_eq!( + completions.load(Ordering::Relaxed), + 0, + "a retained Workflow Task must report nothing to the server" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + // And the wait set really is the one lang described, not an empty stand-in. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 2, 0).await, + ExternalStreamReadyResult::Accepted + ); + + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +/// Polls the resolve activation readiness produced and completes it, leaving no task open. +async fn consume_resolve_activation(worker: &crate::Worker, run_id: &str) -> Vec { + let activation = worker.poll_workflow_activation().await.unwrap(); + let hints = resolve_hints(&activation); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.to_string(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + hints +} + +#[tokio::test] +async fn the_idle_timer_fires_and_starts_the_park_handshake() { + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_millis(30)), + )) + .await + .unwrap(); + + // The timer fires and the set enters `Parking`, which Core expresses by asking lang to run + // the handshake. Nothing was reported to the server: parking is what *ends* the task, and it + // has not been confirmed yet. + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park), vec![(1, ParkReason::Idle, vec![1])]); + assert_eq!( + completions.load(Ordering::Relaxed), + 0, + "asking lang to park must not complete the Workflow Task" + ); + + // Answering `became_ready` releases the set, which is enough to let this test end without + // parking; the handshake's own outcomes are C8's tests. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_became_ready_command(1), + )) + .await + .unwrap(); + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn readiness_does_not_cancel_a_timer_for_a_superseded_snapshot() { + // A timer that fires for a generation the Workflow has already run past must be discarded + // rather than parking the set it finds. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_secs(30)), + )) + .await + .unwrap(); + + // A timer for a stale generation arrives late. + worker.notify_external_stream_idle_timeout(&run_id, 99); + + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "a stale idle timeout must leave the wait set alone" + ); + + // Release the retained task so shutdown can finish. + worker.notify_external_stream_ready(&run_id, 1, 0).await; + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_quiescent_command_with_a_non_positive_idle_timeout_is_malformed() { + // Rejected rather than coerced: zero would park instantly and absent would hold the task + // until it timed out, and neither is something a caller can have meant. The mock's + // `num_expected_fails` is what asserts the failure actually reached the server. + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + mock_cfg.num_expected_fails = 1; + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + workflow_command::Variant::WorkflowStreamQuiescent(WorkflowStreamQuiescent { + quiescence_generation: 1, + waits: vec![ExternalStreamWait { + wait_id: 1, + generation: 0, + immediately_parkable: false, + }], + idle_timeout: None, + }), + )) + .await + .unwrap(); + + // Nothing was retained: no wait set was recorded for the malformed snapshot. + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "a malformed quiescence command must not retain the Workflow Task" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn a_completion_with_server_bound_commands_does_not_retain() { + // Subscriptions stay registered, but the task must be reported so the server can act on the + // timer -- the wake Signal is what covers the window that leaves. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + quiescent_command(1, &[1], Duration::from_secs(30)), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + assert_eq!( + completions.load(Ordering::Relaxed), + 1, + "a completion carrying a server-bound command must be reported, not retained" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_pending_timer_suppresses_retention_but_its_subscriptions_survive_it() { + // Retention is the only thing a pending timer suppresses. The wait set is what a subscription + // *is* on this side, so dropping it along with the retention would leave the Workflow blocked + // on streams Core no longer knows about: every later readiness would answer `RunNotFound`, the + // watcher would tear itself down, and nothing would ever deliver the next record. + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1, 2], + marker_then_timer_history(1, ParkReason::CommandsProduced, b"onetwo"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // A quiescent snapshot registers both subscriptions. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"one", false), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + // The activation that drains a record also starts a timer, and that is what ends the task. + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"two", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![(1, ParkReason::CommandsProduced, b"onetwo".to_vec())], + "the task ended, so its consumption is committed with it" + ); + + // The next Workflow Task finds the same wait set at the same generations: nothing had to be + // re-registered for readiness to be deliverable locally again. The window *between* the two + // tasks is not asserted here -- the replacement arrives on the mock's own schedule, so which + // side of it a probe lands on is a race; `readiness_for_a_cached_run_with_no_open_task_is_not\ + // _run_not_found` pins that state deterministically instead. + let fired = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(fired.run_id, run_id); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the surviving wait set must hold the replacement task open too" + ); + for wait_id in [1, 2] { + assert_eq!( + worker + .notify_external_stream_ready(&run_id, wait_id, 0) + .await, + ExternalStreamReadyResult::Accepted, + "wait {wait_id} must have survived the command-producing completion at generation 0" + ); + } + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +/// A history whose first Workflow Task starts a timer and whose second carries a wake Signal. +/// +/// The task timeout is long on purpose: what is under test is registration, and a park handshake +/// arriving from an idle deadline would be a different activation entirely. +fn timer_then_wake_history() -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + t.add_wfe_started_with_wft_timeout(Duration::from_secs(300)); + t.add_full_wf_task(); + t.add_by_type(EventType::TimerStarted); + t.add_we_signaled( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(0, ""))], + ); + t.add_full_wf_task(); + t.add_workflow_execution_completed(); + t +} + +#[tokio::test] +async fn a_server_bound_command_suppresses_retention_without_dropping_the_registration() { + // Registering a wait set and retaining the Workflow Task for it are two questions, and only + // the second one a timer answers "no" to. Welded together they make an ordinary Workflow -- + // one that starts a timer and first blocks on a stream in the *same* activation -- permanently + // unresumable: the completion registers nothing, so a wake Signal has no wait to mark ready, + // every Workflow Task the Signal produces is empty, and the records sit in the buffer for ever. + // + // Both halves are asserted here deliberately. A test that only checked non-retention is + // exactly what let this through: suppressing retention was already right, and losing the + // registration along with it was the defect. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = + worker_counting_completions(completions.clone(), timer_then_wake_history(), vec![1, 2]); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + // Half one: the task really was reported. The server has to see the timer, so retention is + // correctly refused and the wake Signal covers the window that leaves. + assert_eq!( + completions.load(Ordering::Relaxed), + 1, + "a completion carrying a server-bound command must be reported, not retained" + ); + + // Half two: the subscriptions that same completion described are registered, so the wake + // Signal on the next Workflow Task finds a set to mark ready. Both waits, not only the one the + // Signal names -- the stream in a wake is a hint and lang rechecks every subscription. + let woken = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + resolve_hints(&woken), + vec![1, 2], + "the wake Signal must resolve the registered waits; with nothing registered it marks \ + nothing ready and the activation comes back empty, got {:?}", + woken.jobs + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the registration must hold the replacement task open once the wake resolved it" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn readiness_accepted_against_a_reported_task_is_not_stranded() { + // `Accepted` is a promise that Core will activate, and a task on its way to the server cannot + // keep it. A completion that refuses retention -- one carrying a server-bound command, or a + // replaying one that registers its wait set without holding the task open -- reports the task + // and leaves the accepted readiness with nowhere to ride: the resolve job is never issued, and + // the watcher, told to do nothing, sends no wake Signal either. The record then waits for a + // Workflow Task that never comes, which is the deadlock this asks about. + // + // Requesting the replacement task is what keeps the promise for readiness already accepted. + // The other half of the fix -- ending local delivery with the report, so that readiness + // arriving after it is told `NoOpenWorkflowTask` and signals for itself rather than being + // accepted into the same dead end -- cannot be probed here, because completing the only task + // this mock has budget for evicts the Run; the Python end-to-end replay case covers it. + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + let saw_force_new_wft = Arc::new(AtomicBool::new(false)); + let recorder = saw_force_new_wft.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + recorder.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + }); + + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // A registered wait set with a record buffered against it while lang is still working: the + // shape a resumed Run has when the watcher its replayed `subscribe()` started finds records + // already in the stream. + worker + .seed_external_stream_waits(&run_id, vec![1], None, true) + .await; + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted, + "the seeded set holds an open task, so this readiness is Core's to deliver" + ); + + // The timer is server-bound, so this completion is reported rather than retained. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + assert!( + saw_force_new_wft.load(Ordering::Relaxed), + "the accepted readiness was reported away with the task and nothing asked for the \ + replacement task its resolve job could be issued on" + ); + worker.drain_pollers_and_shutdown().await; +} + +// --- readiness resolves the wait set (C7) ------------------------------------ + +#[tokio::test] +async fn simultaneous_readiness_ships_as_one_coalesced_activation() { + // One activation, not one per wait: there is never more than one outstanding activation per + // Run, and lang probes every active wait on receipt anyway -- the hints are hints. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1, 2, 3], Duration::from_secs(30)), + )) + .await + .unwrap(); + + // The first notification has nothing to coalesce with and ships on its own. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 3, 0).await, + ExternalStreamReadyResult::Accepted + ); + let first = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&first), vec![3]); + + // Everything arriving while that activation is outstanding accumulates -- including a repeat + // for a wait already known ready, which must not ship twice. + for wait_id in [1, 2, 1] { + assert_eq!( + worker + .notify_external_stream_ready(&run_id, wait_id, 0) + .await, + ExternalStreamReadyResult::Accepted + ); + } + assert_eq!( + completions.load(Ordering::Relaxed), + 0, + "readiness must not complete the retained Workflow Task" + ); + + // Completing that activation ships everything accumulated as *one* more activation. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(2, &[1, 2, 3], Duration::from_secs(30)), + )) + .await + .unwrap(); + + let hints = consume_resolve_activation(&worker, &run_id).await; + assert_eq!( + hints, + vec![1, 2], + "three notifications across two waits must coalesce into one activation" + ); + + worker.drain_pollers_and_shutdown().await; +} + +/// The wait ids named by any resolve job in an activation. +fn resolve_hints(activation: &WorkflowActivation) -> Vec { + let mut ids: Vec = activation + .jobs + .iter() + .filter_map(|j| match &j.variant { + Some(workflow_activation_job::Variant::ResolveExternalStreamWaits(r)) => { + Some(r.ready_hints.iter().map(|w| w.wait_id).collect::>()) + } + _ => None, + }) + .flatten() + .collect(); + ids.sort_unstable(); + ids +} + +#[tokio::test] +async fn readiness_arriving_during_an_outstanding_activation_is_accumulated() { + // Readiness that lands while lang is still working cannot ship immediately, and dropping it + // would leave a buffered record with nothing to deliver it. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + )) + .await + .unwrap(); + + // First readiness produces an activation. + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let outstanding = worker.poll_workflow_activation().await.unwrap(); + assert!( + outstanding.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::ResolveExternalStreamWaits(_)) + )), + "expected a resolve activation, got {:?}", + outstanding.jobs + ); + + // Second readiness arrives while that one is still outstanding. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 2, 0).await, + ExternalStreamReadyResult::Accepted + ); + + // Completing the outstanding activation with a fresh quiescent snapshot must surface the + // accumulated readiness rather than losing it. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(2, &[1, 2], Duration::from_secs(30)), + )) + .await + .unwrap(); + worker.notify_external_stream_ready(&run_id, 2, 0).await; + + let hints = consume_resolve_activation(&worker, &run_id).await; + assert_eq!(hints, vec![2]); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn readiness_before_the_idle_timer_expires_cancels_it() { + // If the timer survived readiness it would fire against the *next* quiescent snapshot and + // park a set that had just been told a record was waiting. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_millis(60)), + )) + .await + .unwrap(); + + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + consume_resolve_activation(&worker, &run_id).await; + + // Well past when the cancelled timer would have fired. + tokio::time::sleep(Duration::from_millis(300)).await; + + // Nothing parked: readiness ended that snapshot, and the timer measuring it went with it. + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked, + "a cancelled idle timer must not park a set readiness already resolved" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_stale_generation_produces_no_activation() { + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_secs(30)), + )) + .await + .unwrap(); + + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 42).await, + ExternalStreamReadyResult::Stale + ); + + // Still retained, with no activation to poll -- the notification was for a block that had + // already been resolved, so manufacturing an activation for it would run user code for + // nothing. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + assert_eq!(completions.load(Ordering::Relaxed), 0); + + worker.notify_external_stream_ready(&run_id, 1, 0).await; + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +// --- observation deltas accumulate (C14a) ------------------------------------ + +fn progress_command(delta: &[u8], request_rollover: bool) -> workflow_command::Variant { + workflow_command::Variant::WorkflowStreamProgress(WorkflowStreamProgress { + observation_delta: delta.to_vec(), + request_rollover, + }) +} + +#[tokio::test] +async fn deltas_accumulate_across_a_retained_task() { + // Core is annotation-blind: it appends bytes and hands them back. What this asserts is that + // the concatenation is exactly what lang emitted, in order -- which is the whole reason lang + // can build the marker's annotation by concatenating its own deltas. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"first", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert_eq!(worker.external_stream_annotation(&run_id).await, b"first"); + + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"second", false), + quiescent_command(2, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"firstsecond", + "successive deltas for one Workflow Task must concatenate in order" + ); + assert_eq!( + completions.load(Ordering::Relaxed), + 0, + "accumulating a delta must not complete the retained task" + ); + + worker.notify_external_stream_ready(&run_id, 1, 0).await; + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_empty_delta_accumulates_like_any_other() { + // An activation that observed nothing still observed: it recorded a drain that replay must + // reproduce, and on a subscription's first observation it carries the header without which + // replay has no starting point at all. + let completions = Arc::new(AtomicUsize::new(0)); + let worker = worker_recording_completions(completions.clone()); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + // Accepted without complaint, and it did not disturb what was already there. + assert_eq!(worker.external_stream_annotation(&run_id).await, b""); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + + worker.notify_external_stream_ready(&run_id, 1, 0).await; + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_delta_without_retention_still_accumulates() { + // Progress never implies retention. A completion that consumed records and then produced a + // server-bound command must still commit that consumption, or replay re-delivers the records + // while the command they produced is already durable. + // + // Two task batches, so the run is still cached when the annotation is read -- with one, the + // worker runs out of work and evicts before the assertion. + let completions = Arc::new(AtomicUsize::new(0)); + let counter = completions.clone(); + let t = marker_then_timer_history(0, ParkReason::CommandsProduced, b"consumed"); + let markers: StreamMarkers = Default::default(); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1, 2], mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..2 { + let counter = counter.clone(); + let collected = markers.clone(); + asserts.then(move |wft| { + counter.fetch_add(1, Ordering::Relaxed); + collected.lock().extend(stream_markers(wft)); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"consumed", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + // The next task proves the run is still cached rather than evicted out from under us. + let next = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(next.run_id, run_id); + + assert_eq!( + completions.load(Ordering::Relaxed), + 1, + "a completion with a server-bound command is reported, not retained" + ); + assert_eq!( + *markers.lock(), + vec![(0, ParkReason::CommandsProduced, b"consumed".to_vec())], + "the delta must be committed even though nothing was retained" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "the annotation is cleared once its marker is written" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_progress_command_after_another_command_is_malformed() { + // Ordering is normative. On replay, a record's integrity must be validated before the command + // derived from it is matched -- the other way round, a damaged stream is discovered only after + // its consequences have been accepted as durable. + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + mock_cfg.num_expected_fails = 1; + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + start_timer_cmd(1, Duration::from_secs(10)), + progress_command(b"too late", false), + ], + )) + .await + .unwrap(); + + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "a misordered delta must be rejected, not accumulated" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +// --- rollover transport (C12a) ----------------------------------------------- + +/// A history whose first Workflow Task records a marker and is then replaced. +/// +/// Commands are matched to history events in order, so a rollover that writes a marker needs the +/// marker event present and first -- a history missing it hands the marker machine whatever event +/// happens to be next. +fn marker_then_replacement_history( + quiescence_generation: u64, + terminal: ParkReason, + annotation: &[u8], +) -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker(quiescence_generation, terminal, annotation); + t.add_workflow_task_scheduled_and_started(); + t +} + +/// The `FinalizeExternalStreams` jobs in an activation, as (generation, reason, wait ids). +fn finalization_jobs(activation: &WorkflowActivation) -> Vec<(u64, ParkReason, Vec)> { + activation + .jobs + .iter() + .filter_map(|j| match &j.variant { + Some(workflow_activation_job::Variant::FinalizeExternalStreams(f)) => Some(( + f.quiescence_generation, + f.reason(), + f.waits.iter().map(|w| w.wait_id).collect(), + )), + _ => None, + }) + .collect() +} + +/// Lang's answer to a finalization job: the terminal Core cannot manufacture. +fn finalized_command(quiescence_generation: u64, delta: &[u8]) -> workflow_command::Variant { + workflow_command::Variant::ExternalStreamFinalized(ExternalStreamFinalized { + quiescence_generation, + final_observation_delta: delta.to_vec(), + }) +} + +/// A workflow-only worker recording both markers and `force_new_wft` per completion. +fn worker_recording_rollovers( + markers: StreamMarkers, + forced: Arc>>, + history: TestHistoryBuilder, + batches: Vec, +) -> crate::Worker { + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, batches, mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let collected = markers.clone(); + let forced = forced.clone(); + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + forced.lock().push(wft.force_create_new_workflow_task); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + // Exactly the worker ADR-017 is about: no local activities, so no request sink. + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + mock_worker(mock) +} + +/// Retains a task holding `annotation`, then fires the rollover deadline. +/// +/// Returns the run id. On return the finalization job has not yet been polled. +async fn retain_then_fire_the_rollover_deadline( + worker: &crate::Worker, + annotation: &[u8], +) -> String { + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(annotation, false), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + worker + .start_wft_rollover_timer(&run_id, Duration::from_millis(50)) + .await; + run_id +} + +#[tokio::test] +async fn a_core_decided_boundary_asks_for_a_terminal_before_writing_anything() { + // The C15a protocol, end to end. Core decided this boundary from a timer, so it has no + // terminal for it -- only lang can encode the blocked cursor snapshot. Core must therefore + // ask, and must write nothing at all until the answer arrives. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced.clone(), + marker_then_replacement_history(1, ParkReason::Rollover, b"before-rollover-terminal"), + vec![1, 2], + ); + + let run_id = retain_then_fire_the_rollover_deadline(&worker, b"before-rollover").await; + + // The deadline produced a finalization job covering the *complete* active wait set, not a + // completion. + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::Rollover, vec![1, 2])], + "a rollover deadline must ask lang to finalize, got {:?}", + finalize.jobs + ); + + // Nothing has been written and nothing has been reported: the annotation is still held, + // because a marker without its terminal is durable and wrong. + assert_eq!(*markers.lock(), Vec::new()); + assert_eq!( + *forced.lock(), + Vec::::new(), + "the task must stay open until its terminal exists" + ); + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"before-rollover", + "the accumulated annotation is held across the round trip, not discarded" + ); + + // Lang supplies the terminal, and only now is the marker written. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b"-terminal"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![( + 1, + ParkReason::Rollover, + b"before-rollover-terminal".to_vec() + )], + "the marker carries the accumulated annotation with lang's terminal appended" + ); + assert_eq!( + *forced.lock(), + vec![true], + "the completion carrying the marker is also the one that requests a replacement" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "the annotation is cleared once its marker is written" + ); + + // And the wait set survived onto the replacement task, which is retained exactly as its + // predecessor was -- the durable half of rollover does not cost the transport half. This poll + // lets the replacement in and hands nothing back, which is itself the retention assertion. + assert!( + tokio::time::timeout( + Duration::from_millis(400), + worker.poll_workflow_activation() + ) + .await + .is_err(), + "the replacement task must be retained too, or the rollover undoes itself" + ); + assert_eq!( + worker.notify_external_stream_ready(&run_id, 2, 0).await, + ExternalStreamReadyResult::Accepted, + "wait 2 must still be registered at generation 0 across the rollover" + ); + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![2]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + + worker.drain_pollers_and_shutdown().await; +} + +/// A workflow-only worker recording the markers and the query ids answered per completion. +fn worker_recording_query_answers( + markers: StreamMarkers, + answered: Arc>>, + history: TestHistoryBuilder, + resps: Vec, +) -> crate::Worker { + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, resps, mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let collected = markers.clone(); + let answered = answered.clone(); + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + answered + .lock() + .extend(wft.query_responses.iter().map(|q| q.query_id.clone())); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + mock_worker(mock) +} + +#[tokio::test] +async fn a_query_riding_along_asks_for_a_terminal_rather_than_committing_without_one() { + // A query answer has to be reported, so it refuses retention -- and that makes the boundary + // *Core's*. Lang asked to be held open, so its `WorkflowStreamProgress` deliberately carried + // no terminal: the terminal was going to arrive on a park result or a finalization. Writing + // `TaskCompleted` from that report would commit `header, segment*` with no terminal frame, + // which is durable and wrong and which ADR-008 forbids without exception. Core owes a + // finalization round trip here exactly as the rollover deadline and a teardown do. + // + // The query must survive that round trip, which runs no user Workflow code and so gives lang + // no way to resend what it already answered. + let markers: StreamMarkers = Default::default(); + let answered: Arc>> = Default::default(); + let history = marker_then_replacement_history(1, ParkReason::TaskCompleted, b"observed|end"); + let mut with_query = + hist_to_poll_resp(&history, "fakeid".to_owned(), ResponseType::ToTaskNum(1)); + with_query.queries = HashMap::from([( + "q1".to_string(), + WorkflowQuery { + query_type: "query-type".to_string(), + query_args: Some(b"hi".into()), + header: None, + }, + )]); + let worker = worker_recording_query_answers( + markers.clone(), + answered.clone(), + history, + vec![with_query.into()], + ); + + // Lang consumed a record and asked to be held open, so its report stops short of the terminal. + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"observed", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the task must be retained here, or the query has nothing to interrupt" + ); + + // The query arrives on its own activation -- which is exactly why lang cannot supply the + // terminal itself: the completion answering it produces no stream command at all, and the + // annotation at risk was accumulated on the activation before. + let queried = worker.poll_workflow_activation().await.unwrap(); + assert!( + queried.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::QueryWorkflow(_)) + )), + "the query has to reach lang, or this test proves nothing, got {:?}", + queried.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + query_ok("q1", "the-answer"), + )) + .await + .unwrap(); + + // Nothing was written and nothing was reported: Core asks first. + assert_eq!( + *markers.lock(), + Vec::new(), + "a query answer must not commit an annotation Core never obtained a terminal for" + ); + assert_eq!( + *answered.lock(), + Vec::::new(), + "the task stays open until its terminal exists, so the query has not been reported yet" + ); + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::TaskCompleted, vec![1])], + "a query-refused retention must ask lang to finalize the boundary it created, got {:?}", + finalize.jobs + ); + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"observed", + "the accumulated annotation is held across the round trip, not discarded" + ); + + // Lang supplies the terminal, and only now is the marker written -- on the same completion + // that finally carries the query answer. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b"|end"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![(1, ParkReason::TaskCompleted, b"observed|end".to_vec())], + "the marker carries the accumulated annotation with lang's terminal appended" + ); + assert_eq!( + *answered.lock(), + vec!["q1".to_string()], + "the query answer must ride the finalization round trip out: the round trip runs no user \ + Workflow code, so lang cannot resend it" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_finalization_answered_without_a_terminal_writes_no_marker() { + // There is no best-effort path. If the terminal cannot be obtained, Core writes nothing and + // the Workflow Task is retried -- an abandoned task commits no cursor and loses no record, + // while a truncated annotation is durable and wrong. + // + // "Writes no marker" is asserted as *zero successful completions*: a failed Workflow Task + // reports no commands at all, so a marker could only have escaped through a completion that + // must not exist. Both counts are verified by the mock when it drops. + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + marker_then_replacement_history(1, ParkReason::Rollover, b"never-written"), + [1], + mock_worker_client(), + ); + mock_cfg.num_expected_fails = 1; + mock_cfg.num_expected_completions = Some(0.into()); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let run_id = retain_then_fire_the_rollover_deadline(&worker, b"before-rollover").await; + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize).len(), + 1, + "the deadline must have asked for a terminal, or this test proves nothing" + ); + + // Lang answers the finalization job with anything other than a terminal. + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "a finalization that produced no terminal must fail the Workflow Task" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn a_finalized_command_with_no_job_outstanding_is_refused() { + // The paired negative. Without it, accepting `ExternalStreamFinalized` unconditionally would + // let lang append a terminal to an annotation Core never asked it to close -- and the + // "answered correctly" case above would pass either way. + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + canned_histories::single_timer("1"), + [1], + mock_worker_client(), + ); + mock_cfg.num_expected_fails = 1; + mock_cfg.num_expected_completions = Some(0.into()); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"unprompted", false), + finalized_command(1, b"-terminal"), + ], + )) + .await + .unwrap(); + + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "an unprompted terminal must fail the Workflow Task rather than be accepted" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn a_rollover_with_nothing_accumulated_asks_for_no_terminal() { + // C12a's half, still true: with no annotation there is no marker to write, so there is + // nothing to finalize either. The task is still replaced -- rollover is about the deadline, + // not about the stream having produced anything. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced.clone(), + canned_histories::single_timer("1"), + vec![1, 1], + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + // Retention with no progress at all: nothing was ever observed. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_secs(30)), + )) + .await + .unwrap(); + worker + .start_wft_rollover_timer(&run_id, Duration::from_millis(50)) + .await; + + // No finalization job is issued, so the replacement task is simply retained again and the + // poll has nothing to hand back. + assert!( + tokio::time::timeout( + Duration::from_millis(600), + worker.poll_workflow_activation() + ) + .await + .is_err(), + "with nothing to finalize the replacement task is retained, not activated" + ); + assert_eq!(*markers.lock(), Vec::new()); + assert_eq!( + *forced.lock(), + vec![true], + "the deadline still requests a replacement even with no marker to write" + ); + + // The wait set survived onto the replacement exactly as C12a requires. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted, + "wait 1 must still be registered at generation 0 across the rollover" + ); + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_budget_rollover_forces_a_replacement_without_a_deadline() { + // Lang decided this boundary, so it needs no finalization round trip -- the very command + // carrying the request already carried the terminal. + let saw_force_new_wft = Arc::new(AtomicBool::new(false)); + let recorder = saw_force_new_wft.clone(); + let t = marker_then_timer_history(0, ParkReason::BudgetRollover, b"at-the-budget"); + let markers: StreamMarkers = Default::default(); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1, 2], mock_worker_client()); + let collected = markers.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + recorder.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + collected.lock().extend(stream_markers(wft)); + }); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"at-the-budget", true), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + let next = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(next.run_id, run_id); + assert_eq!( + finalization_jobs(&next), + Vec::new(), + "a budget rollover needs no finalization round trip -- the command that asked for it \ + already carried the terminal" + ); + + assert!( + saw_force_new_wft.load(Ordering::Relaxed), + "request_rollover must force a replacement task" + ); + assert_eq!( + *markers.lock(), + vec![(0, ParkReason::BudgetRollover, b"at-the-budget".to_vec())], + "a budget rollover writes its marker without a finalization round trip -- the command \ + that asked for it already carried the terminal" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn two_consecutive_rollovers_produce_two_markers_that_reassemble_in_order() { + // A batch split across two Workflow Tasks must be recoverable, and the only thing that makes + // it recoverable is that each marker carries its own task's deltas and the two concatenate in + // task order. A marker that repeated or dropped a stretch would leave replay unable to say + // what the original run actually saw. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker(1, ParkReason::Rollover, b"first-half.terminal-one"); + t.add_full_wf_task(); + t.add_external_stream_marker(2, ParkReason::Rollover, b"second-half.terminal-two"); + t.add_workflow_task_scheduled_and_started(); + + let worker = worker_recording_rollovers(markers.clone(), forced.clone(), t, vec![1, 2]); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // --- rollover one, on the task the run started with --- + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"first-half", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + worker + .start_wft_rollover_timer(&run_id, Duration::from_millis(50)) + .await; + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(finalization_jobs(&finalize).len(), 1); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b".terminal-one"), + )) + .await + .unwrap(); + + // --- rollover two, on the replacement task --- + // The replacement is retained the moment it arrives, so this poll hands nothing back; it + // exists to let the task in. Readiness is what activates lang on it. + assert!( + tokio::time::timeout( + Duration::from_millis(400), + worker.poll_workflow_activation() + ) + .await + .is_err(), + "the replacement task must be retained, or the rollover undoes itself" + ); + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + let resumed = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resumed), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"second-half", false), + quiescent_command(2, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + worker + .start_wft_rollover_timer(&run_id, Duration::from_millis(50)) + .await; + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(finalization_jobs(&finalize).len(), 1); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(2, b".terminal-two"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + assert_eq!( + written.len(), + 2, + "one marker per Workflow Task, not one per run" + ); + assert_eq!( + written[0], + (1, ParkReason::Rollover, b"first-half.terminal-one".to_vec()) + ); + assert_eq!( + written[1], + ( + 2, + ParkReason::Rollover, + b"second-half.terminal-two".to_vec() + ) + ); + + // Reassembled in Workflow Task order the two markers are the whole batch, with nothing + // repeated and nothing lost. + let reassembled: Vec = written.iter().flat_map(|(_, _, a)| a.clone()).collect(); + assert_eq!( + reassembled, + b"first-half.terminal-onesecond-half.terminal-two" + ); + assert_eq!( + *forced.lock(), + vec![true, true], + "each rollover requests its own replacement task" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_continuously_fed_stream_still_rolls_over() { + // The workload the rollover deadline exists for, and the one a deadline re-armed at each + // quiescence silently stops covering. A stream whose gaps stay below the idle timeout never + // reaches the parking path, so nothing but the deadline ends the Workflow Task -- and becoming + // quiescent again is precisely what a delivered record *does*. Anchored at the moment it is + // armed, the deadline is pushed out by every record, the idle timer is clamped below it and so + // cannot fire either, and the retained task runs until the *server* times it out, which is a + // Workflow Task failure rather than a rollover. + // + // The existing deadline test drives a stream with no traffic at all, so it cannot see this: a + // snapshot that is never renewed is one the arming moment and the task's start agree about. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + + // 500ms puts the rollover deadline 400ms after the task started and clamps the idle timeout to + // at most 360ms, so a record every 20ms keeps the idle timer from ever firing. + let mut t = TestHistoryBuilder::default(); + t.add_wfe_started_with_wft_timeout(Duration::from_millis(500)); + t.add_full_wf_task(); + t.add_workflow_task_scheduled_and_started(); + + let worker = worker_recording_rollovers(markers.clone(), forced.clone(), t, vec![1]); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"fed", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + // Feed the stream, one record at a time, until the deadline ends the task. + let started = std::time::Instant::now(); + let mut generation = 1u64; + let finalize = loop { + assert!( + started.elapsed() < Duration::from_secs(5), + "the rollover deadline never fired against a stream that kept feeding it: {generation} \ + quiescent snapshots and no finalization" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted, + "the wait must still be deliverable, or this loop is not feeding anything" + ); + let activation = worker.poll_workflow_activation().await.unwrap(); + if !finalization_jobs(&activation).is_empty() { + break activation; + } + assert_eq!( + park_jobs(&activation), + Vec::new(), + "a fed stream must not reach the idle parking path, got {:?}", + activation.jobs + ); + generation += 1; + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"fed", false), + quiescent_command(generation, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + }; + + // Which snapshot the deadline caught is Core's business, not the loop's -- a completion that + // arrives with the rollover already pending is not retained, so the snapshot it carried is + // never recorded and the loop's own count runs ahead. What matters is that the boundary is a + // rollover, that it covers the wait set, and that the stream really did re-establish + // quiescence many times over before it fired. + let jobs = finalization_jobs(&finalize); + assert_eq!(jobs.len(), 1, "got {:?}", finalize.jobs); + let (closed, reason, waits) = jobs[0].clone(); + assert_eq!(reason, ParkReason::Rollover); + assert_eq!(waits, vec![1]); + assert!( + closed >= 3, + "the stream must have re-established quiescence several times before the deadline fired, \ + or the case under test never arose; it closed generation {closed}" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + finalized_command(closed, b"|end"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + assert_eq!(written.len(), 1, "one Workflow Task gets one marker"); + assert_eq!(written[0].1, ParkReason::Rollover); + assert!(written[0].2.ends_with(b"|end"), "got {:?}", written[0].2); + assert_eq!( + *forced.lock(), + vec![true], + "a rollover hands the work to a replacement task" + ); + + worker.drain_pollers_and_shutdown().await; +} + +/// A stand-in for the encoder's `MAX_ANNOTATION_BYTES`. +/// +/// The real constant lives in lang's codec, on the far side of an annotation Core cannot read. +/// Core's half of the mechanism is `request_rollover`, and what the budget means *to Core* is the +/// bound below: no marker Core writes may exceed it, however many bytes the batch as a whole ran +/// to. +const ANNOTATION_BUDGET: usize = 1200; +/// The fraction of the budget at which lang asks for a rollover on its next report. +const ANNOTATION_HIGH_WATER: usize = ANNOTATION_BUDGET / 2; +/// One activation's worth of observed bytes. +const OBSERVED_CHUNK: usize = 500; + +#[tokio::test] +async fn a_budget_driven_split_writes_two_markers_rather_than_one_oversized_one() { + // The byte budget, as the split it exists to force. Both halves are asserted in *bytes*: no + // marker exceeds the budget, and the two together exceed it -- so the batch really could not + // have fitted in one marker and the rollover was not decoration. A run-count assertion would + // say nothing about either. + // + // Budget-driven rather than deadline-driven, which is what makes this a different mechanism + // from `two_consecutive_rollovers_produce_two_markers_that_reassemble_in_order`: lang decided + // both boundaries, so no `FinalizeExternalStreams` may be issued for either. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + // Two quiescent snapshots passed before the first task hit the mark, and two more before the + // second did, so the markers close generations 2 and 4. + t.add_external_stream_marker(2, ParkReason::BudgetRollover, b"first-marker"); + t.add_full_wf_task(); + t.add_external_stream_marker(4, ParkReason::BudgetRollover, b"second-marker"); + t.add_workflow_task_scheduled_and_started(); + + let worker = worker_recording_rollovers(markers.clone(), forced.clone(), t, vec![1, 2]); + let chunk = vec![b'x'; OBSERVED_CHUNK]; + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // --- the first Workflow Task fills up --- + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(&chunk, false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert!( + worker.external_stream_annotation(&run_id).await.len() < ANNOTATION_HIGH_WATER, + "the first chunk must be below the high-water mark, or the split under test is the wrong \ + one" + ); + + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(&chunk, false), + quiescent_command(2, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert!( + worker.external_stream_annotation(&run_id).await.len() > ANNOTATION_HIGH_WATER, + "the accumulated annotation must have passed the high-water mark, which is what lang \ + reacts to on its next report" + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "passing the high-water mark is not itself a boundary -- lang asks on its *next* report" + ); + + // Lang saw the mark go by and asks for the rollover on its next report, which -- because lang + // decided this boundary -- already carries the terminal. + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + progress_command(b"|first-terminal", true), + )) + .await + .unwrap(); + + assert_eq!(markers.lock().len(), 1, "the first marker closes here"); + assert_eq!(*forced.lock(), vec![true]); + + // --- the second Workflow Task, which the split created --- + // The replacement is retained the moment it arrives, so this poll hands nothing back; it + // exists to let the task in and to prove no finalization was asked for. + assert!( + tokio::time::timeout( + Duration::from_millis(400), + worker.poll_workflow_activation() + ) + .await + .is_err(), + "a budget rollover needs no finalization round trip and must retain its replacement task" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "the next Workflow Task starts a fresh annotation, which is what bounds the marker" + ); + + for (generation, delta) in [(3u64, &chunk), (4, &chunk)] { + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(delta, false), + quiescent_command(generation, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + } + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + progress_command(b"|second-terminal", true), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + assert_eq!( + written.len(), + 2, + "the batch was split across two Workflow Tasks, so it is two markers" + ); + for (generation, terminal, annotation) in &written { + assert_eq!( + *terminal, + ParkReason::BudgetRollover, + "both boundaries were lang's, and the marker must say so" + ); + assert!( + annotation.len() <= ANNOTATION_BUDGET, + "the marker closing generation {generation} grew to {} bytes, past the budget the \ + rollover exists to keep it under", + annotation.len() + ); + } + + let reassembled: Vec = written.iter().flat_map(|(_, _, a)| a.clone()).collect(); + assert!( + reassembled.len() > ANNOTATION_BUDGET, + "the batch must not fit in one marker, or nothing here needed splitting" + ); + let mut expected = chunk.repeat(2); + expected.extend_from_slice(b"|first-terminal"); + expected.extend_from_slice(&chunk.repeat(2)); + expected.extend_from_slice(b"|second-terminal"); + assert_eq!( + reassembled, expected, + "reassembled in Workflow Task order the markers are the whole batch, with nothing \ + repeated and nothing lost" + ); + assert_eq!( + *forced.lock(), + vec![true, true], + "each half is its own Workflow Task, so each asks for its own replacement" + ); + + worker.drain_pollers_and_shutdown().await; +} + +// --- shutdown and eviction transitions (C15b) -------------------------------- + +#[tokio::test] +async fn shutdown_with_a_workflow_task_open_writes_its_marker_and_forces_a_replacement() { + // ADR-009's first row. The Run holds a Workflow Task the Worker is about to stop serving, so + // Core closes that boundary itself: it asks lang for the terminal it cannot manufacture, + // writes the one marker for the task, and completes requesting a replacement task -- which is + // what offers the Run back to the task queue for another Worker to pick up. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced.clone(), + marker_then_replacement_history(1, ParkReason::Shutdown, b"before-shutdown-terminal"), + vec![1], + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"before-shutdown", false), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + + worker.initiate_shutdown(); + // The probe lang's own shutdown sweep makes of every Run with active subscriptions. `WftOpen` + // is what tells lang to leave this Run to Core, and it is the same classification Core's sweep + // keys off -- if the two disagreed a Run would be swept twice or not at all. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::Shutdown, vec![1, 2])], + "shutdown with a Workflow Task open must ask lang to finalize the complete wait set, \ + got {:?}", + finalize.jobs + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "nothing may be written before the terminal arrives" + ); + assert_eq!( + *forced.lock(), + Vec::::new(), + "the task must stay open until its terminal exists" + ); + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"before-shutdown", + "the accumulated annotation is held across the round trip, not discarded" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b"-terminal"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![( + 1, + ParkReason::Shutdown, + b"before-shutdown-terminal".to_vec() + )], + "the marker carries the accumulated annotation with lang's terminal appended, and says \ + which boundary closed it" + ); + assert_eq!( + *forced.lock(), + vec![true], + "the completion carrying the marker is also the one that hands the Run back" + ); + + // And with the boundary closed the Run owes nothing, so shutdown can finish -- which it could + // not while a Workflow Task was still open, since that counts as pending work. + assert!( + matches!( + worker.poll_workflow_activation().await, + Err(PollError::ShutDown) + ), + "the Run must be released once its boundary is closed" + ); + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn shutdown_with_no_open_workflow_task_writes_no_marker_and_completes_nothing() { + // ADR-009's second row, and the reason the two transitions are separate deliverables. Here + // there is no task token to set `force_new_wft` on and nothing accumulated to write, so the + // *correct* behaviour is to do nothing at all. The server-visible replacement is lang's wake + // sweep; Core reimplementing it here would send Signals for Runs it is not entitled to speak + // for. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced.clone(), + canned_histories::single_timer("1"), + vec![1], + ); + + // The first activation is left outstanding for the whole test, which is what keeps the Run + // cached: a Run that goes idle here is dropped when the mock runs out of work, and an evicted + // Run is a different state from the one this is about. + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .seed_external_stream_waits(&run_id, vec![1, 2], None, false) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask, + "this test is only meaningful with subscriptions active and no Workflow Task" + ); + + worker.initiate_shutdown(); + // The same probe as above, and this time its answer is what keeps Core's sweep off the Run. + // Answering it also drives the stream, so the sweep really did get its chance to run. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask + ); + + // Core asked lang for nothing: the only activation in flight is still the original one, which + // lang now answers on its own terms. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + Vec::new(), + "nothing was accumulated, so no marker is written and none is missing" + ); + assert_eq!( + *forced.lock(), + vec![false], + "lang's own completion must not be turned into a hand-back: with no Workflow Task of its \ + own to close, this transition asks the server for nothing" + ); + assert!( + matches!( + worker.poll_workflow_activation().await, + Err(PollError::ShutDown) + ), + "a Run with no open Workflow Task must produce no activation on the way out" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn evicting_a_run_with_no_open_workflow_task_writes_no_marker() { + // Eviction reaches the same two states as shutdown and must make the same choice. With no + // Workflow Task there is nothing to finalize, so the eviction activation goes out directly. + // + // "Writes no marker" is asserted as *zero completions*, the same way C15a's negative case is: + // a marker reaches History only on a completion, so a mock that refuses every completion + // proves no marker escaped by any route at all. The count is verified when the mock drops. + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + canned_histories::single_timer("1"), + [1], + mock_worker_client(), + ); + mock_cfg.num_expected_completions = Some(0.into()); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + // Retaining keeps the Run cached with its task unreported, which is what lets the wait set be + // put into -- and observed in -- the state under test. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_secs(30)), + )) + .await + .unwrap(); + worker + .seed_external_stream_waits(&run_id, vec![1], None, false) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask, + "this test is only meaningful with subscriptions active and no Workflow Task" + ); + + worker.request_workflow_eviction(&run_id); + + let evict = worker.poll_workflow_activation().await.unwrap(); + assert!( + evict.is_only_eviction(), + "with no Workflow Task open the eviction goes out directly -- nothing may be asked of \ + lang first, got {:?}", + evict.jobs + ); + assert_eq!( + finalization_jobs(&evict), + Vec::new(), + "there is no boundary to finalize, so no terminal may be requested" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::RunNotFound, + "the Run really was evicted, so the assertions above are about the state under test" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn evicting_a_run_holding_a_workflow_task_finalizes_before_the_eviction_activation() { + // The sequencing the whole transition rests on. The marker rides the *finalization* + // completion, never the eviction completion -- an eviction completion reports nothing and may + // carry no commands, so a marker attached there would be dropped without a trace. Issuing the + // finalization first is what guarantees it is answered before `RemoveFromCache` goes out. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + marker_then_replacement_history(1, ParkReason::Shutdown, b"before-eviction-terminal"), + [1], + mock_worker_client(), + ); + let collected = markers.clone(); + let recorder = forced.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let collected = collected.clone(); + let recorder = recorder.clone(); + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + recorder.lock().push(wft.force_create_new_workflow_task); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + // Core's own default, which Python does not override: a pending eviction and its reply + // count as pending work. The test helper flips it, and with it flipped the Run would be + // dropped at shutdown before its eviction activation was ever issued -- which is the very + // ordering under test here. + w.ignore_evicts_on_shutdown = false; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"before-eviction", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + worker.request_workflow_eviction(&run_id); + + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::Shutdown, vec![1])], + "the finalization must be issued before the eviction, got {:?}", + finalize.jobs + ); + assert!( + !finalize.is_only_eviction(), + "the eviction must not have overtaken the finalization" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b"-terminal"), + )) + .await + .unwrap(); + assert_eq!( + *markers.lock(), + vec![( + 1, + ParkReason::Shutdown, + b"before-eviction-terminal".to_vec() + )], + "the marker rides the finalization completion" + ); + assert_eq!(*forced.lock(), vec![true]); + + // Only now does the eviction go out, and it adds nothing to what was already written. + let evict = worker.poll_workflow_activation().await.unwrap(); + assert!( + evict.is_only_eviction(), + "the eviction follows the finalization, got {:?}", + evict.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id)) + .await + .unwrap(); + assert_eq!( + markers.lock().len(), + 1, + "one Workflow Task gets one marker, and the eviction completion writes none" + ); + assert_eq!(*forced.lock(), vec![true]); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +/// Runs a Worker that shuts down holding a Workflow Task, returning the marker it wrote. +/// +/// The transition itself is asserted by +/// `shutdown_with_a_workflow_task_open_writes_its_marker_and_forces_a_replacement`; this exists so +/// a *second* Worker can be handed the very bytes the first one wrote, rather than a marker a test +/// composed by hand and hoped was the same shape. +async fn a_marker_written_on_the_way_out() -> ExternalStreamMarkerData { + let recorded: Arc>> = Default::default(); + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + marker_then_replacement_history(1, ParkReason::Shutdown, b"before-shutdown-terminal"), + [1], + mock_worker_client(), + ); + let collected = recorded.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_marker_data(wft)); + }); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"before-shutdown", false), + quiescent_command(1, &[1, 2], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + worker.initiate_shutdown(); + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::Shutdown, vec![1, 2])], + "shutdown must ask lang to finalize before anything is written, got {:?}", + finalize.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + finalized_command(1, b"-terminal"), + )) + .await + .unwrap(); + worker.shutdown().await; + worker.finalize_shutdown().await; + + let mut written = recorded.lock().clone(); + assert_eq!(written.len(), 1, "one Workflow Task gets one marker"); + written.pop().unwrap() +} + +#[tokio::test] +async fn a_second_worker_reconstructs_the_subscription_from_the_shutdown_marker() { + // The half of the shutdown row that says what `force_new_wft` was *for*. Handing the Run back + // to the task queue is only worth doing if whoever picks it up can carry on, and the + // replacement Worker starts with no wait set, no cursor, and no watcher: everything it resumes + // from has to come out of the marker the first Worker wrote. + // + // The marker is the one the first Worker actually produced, bytes and wait list alike -- a + // hand-composed stand-in would prove only that the reader agrees with the test. + let recorded = a_marker_written_on_the_way_out().await; + assert_eq!(recorded.terminal_boundary(), ParkReason::Shutdown); + assert_eq!( + recorded.replay_annotation, b"before-shutdown-terminal", + "the marker under test must be the finalized one, or the replacement has nothing to \ + reconstruct from" + ); + let waits: Vec<(u32, u64)> = recorded + .waits + .iter() + .map(|w| (w.wait_id, w.generation)) + .collect(); + assert_eq!( + waits, + vec![(1, 0), (2, 0)], + "the marker must name the subscriptions it closed" + ); + + // The replacement task, as a second Worker sees it: the first Workflow Task is history now, + // and the marker sits inside it. + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker_covering( + recorded.quiescence_generation, + recorded.terminal_boundary(), + &recorded.replay_annotation, + &waits, + ); + t.add_workflow_task_scheduled_and_started(); + + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers(markers.clone(), vec![2], t); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + let run_id = replayed.run_id.clone(); + assert!( + replayed.is_replaying, + "the replacement Worker reaches the marker by replaying the task that wrote it, got \ + {replayed:?}" + ); + assert_eq!( + replay_jobs(&replayed), + vec![( + recorded.quiescence_generation, + ParkReason::Shutdown, + waits, + recorded.replay_annotation.clone() + )], + "the second Worker must be handed the recorded snapshot, waits, terminal and annotation \ + unchanged, got {:?}", + replayed.jobs + ); + // And no readiness path is entered on the way: the replacement reconstructs from the marker + // rather than by consulting a backend it has never spoken to. + assert_eq!(resolve_hints(&replayed), Vec::::new()); + assert_eq!(park_jobs(&replayed), Vec::new()); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id)) + .await + .unwrap(); + assert_eq!( + *markers.lock(), + Vec::new(), + "the replacement reads the marker; it must not write a second copy of it" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +/// A workflow-only worker recording every marker, which hands out its eviction activations. +/// +/// `ignore_evicts_on_shutdown = false` is Core's own default, which Python does not override. The +/// test helper flips it, and with it flipped a Run is dropped the moment the mock runs out of work +/// -- before the eviction activation any test about eviction is waiting for. +fn worker_recording_markers_through_eviction( + markers: StreamMarkers, + batches: Vec, + history: TestHistoryBuilder, +) -> crate::Worker { + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, batches, mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let collected = markers.clone(); + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + w.ignore_evicts_on_shutdown = false; + }); + mock_worker(mock) +} + +#[tokio::test] +async fn an_unwritten_annotation_exists_only_while_a_workflow_task_is_open() { + // The invariant the whole eviction split rests on, asserted in both states an eviction can + // find a Run in. It is what makes "no open Workflow Task, so no marker" safe rather than + // lossy: with a task open an unwritten annotation may exist and eviction must finalize it, + // and without one there is provably nothing to finalize -- not because nothing was consumed, + // but because the completion that ended the task took the bytes with it. + // + // --- state one: a Workflow Task is open --- + let held: StreamMarkers = Default::default(); + let worker = worker_recording_markers_through_eviction( + held.clone(), + vec![1], + marker_then_replacement_history(1, ParkReason::Shutdown, b"held-terminal"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"held", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"held", + "an unwritten annotation may exist here, and this is the only state in which it may" + ); + + worker.request_workflow_eviction(&run_id); + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::Shutdown, vec![1])], + "an eviction finding an open task must close its boundary, got {:?}", + finalize.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b"-terminal"), + )) + .await + .unwrap(); + assert_eq!( + *held.lock(), + vec![(1, ParkReason::Shutdown, b"held-terminal".to_vec())], + "the held bytes leave as a marker rather than with the Run" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "and the annotation is cleared by the write, so nothing is held into the eviction" + ); + + let evict = worker.poll_workflow_activation().await.unwrap(); + assert!(evict.is_only_eviction(), "got {:?}", evict.jobs); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + assert_eq!( + held.lock().len(), + 1, + "one boundary gets one marker: the eviction completion writes none of its own" + ); + worker.shutdown().await; + worker.finalize_shutdown().await; + + // --- state two: no Workflow Task is open --- + // A Run that consumed records on an earlier task and is now sitting between tasks. The + // distinction that matters is that this Run has *not* done nothing: an empty annotation here + // is a fact about the completion path, not about there having been nothing to write. + let reported: StreamMarkers = Default::default(); + let worker = worker_recording_markers_through_eviction( + reported.clone(), + vec![1, 2], + marker_then_timer_history(0, ParkReason::CommandsProduced, b"consumed"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"consumed", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + // The next task registers the subscription again and is held open, which is what keeps the Run + // observable while its wait set is put into the state under test. + let fired = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(fired.run_id, run_id); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1], Duration::from_secs(30)), + )) + .await + .unwrap(); + assert_eq!( + *reported.lock(), + vec![(0, ParkReason::CommandsProduced, b"consumed".to_vec())], + "everything this Run consumed left with the task that consumed it" + ); + + // Pinned rather than waited for: seeding clears the open flag without touching the annotation, + // so a completion path that had failed to write its bytes would still show them here. + worker + .seed_external_stream_waits(&run_id, vec![1], None, false) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "with no Workflow Task open there is nothing unwritten, which is what makes writing \ + nothing safe" + ); + + worker.request_workflow_eviction(&run_id); + let evict = worker.poll_workflow_activation().await.unwrap(); + assert!( + evict.is_only_eviction(), + "with nothing to finalize the eviction goes out directly, got {:?}", + evict.jobs + ); + assert_eq!( + finalization_jobs(&evict), + Vec::new(), + "there is no boundary to finalize, so no terminal may be requested" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + assert_eq!( + reported.lock().len(), + 1, + "the eviction wrote no marker, and none was missing" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::RunNotFound, + "the Run really was evicted, so the assertions above are about the state under test" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +// --- the reserved wake Signal (C11) ------------------------------------------ + +/// A `WakeSignal` payload, serialized exactly the way a producer sends one. +/// +/// Built with the protocol's own serialization rather than through a +/// `DataConverter`, because Core is the component that has to read it. +fn wake_payload(wake: WakeSignal) -> Payload { + Payload { + metadata: HashMap::from([ + ("encoding".to_string(), b"binary/protobuf".to_vec()), + ( + "messageType".to_string(), + external_stream::WAKE_SIGNAL_MESSAGE_TYPE + .as_bytes() + .to_vec(), + ), + ]), + data: wake.encode_to_vec(), + ..Default::default() + } +} + +fn wake(park_generation: u64, first_execution_run_id: &str) -> WakeSignal { + WakeSignal { + envelope_version: external_stream::WAKE_SIGNAL_ENVELOPE_VERSION, + stream_name: "tokens".to_string(), + wait_id: 1, + park_generation, + first_execution_run_id: first_execution_run_id.to_string(), + producer_session_id: "producer-a".to_string(), + } +} + +/// The external stream markers a completion recorded, with their terminal boundaries. +/// +/// Reading the envelope back out of the command is what makes "exactly one marker per Workflow +/// Task" and "never without a terminal" checkable rather than assumed. +fn stream_markers(wft: &WorkflowTaskCompletion) -> Vec { + stream_marker_data(wft) + .into_iter() + .map(|d| { + ( + d.quiescence_generation, + d.terminal_boundary(), + d.replay_annotation, + ) + }) + .collect() +} + +/// The whole marker envelope a completion recorded, wait list included. +/// +/// [`stream_markers`] drops the waits because most assertions do not need them. A test that hands +/// a marker one Worker wrote to a *second* Worker does: the wait list is what the replacement +/// reconstructs the subscription from. +fn stream_marker_data(wft: &WorkflowTaskCompletion) -> Vec { + wft.commands + .iter() + .filter_map(|c| match &c.attributes { + Some(command::Attributes::RecordMarkerCommandAttributes(m)) + if m.marker_name == EXTERNAL_STREAM_MARKER_NAME => + { + extract_external_stream_marker_data(&m.details) + } + _ => None, + }) + .collect() +} + +/// A worker whose second Workflow Task carries one Signal. +/// +/// The Workflow Task timeout is long on purpose: a wake that fails validation leaves the seeded +/// wait set retaining the replacement task, which re-arms the idle deadline. With the default +/// five-second timeout that deadline expires inside the test and parks the set, which is correct +/// behaviour but not what these tests are about. +fn worker_with_a_signal(signal_name: &str, payloads: Vec) -> crate::Worker { + let mut t = TestHistoryBuilder::default(); + t.add_wfe_started_with_wft_timeout(Duration::from_secs(300)); + t.add_full_wf_task(); + t.add_we_signaled(signal_name, payloads); + t.add_full_wf_task(); + t.add_workflow_execution_completed(); + + let mut mock = build_mock_pollers(MockPollCfg::from_resp_batches( + "fakeid", + t, + [1, 2], + mock_worker_client(), + )); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + mock_worker(mock) +} + +/// Asserts that a wake Signal which failed validation changed nothing. +/// +/// Two things must hold, and both are asserted against whatever Core actually produces rather +/// than against a particular activation shape: the Signal never reaches a user handler, and no +/// wait is resolved. Whether an activation arrives at all depends on the wait set's state -- a +/// still-blocked set retains the task and produces nothing, a parked one lets it complete -- and +/// neither outcome is the point. +async fn assert_the_wake_changed_nothing(worker: &crate::Worker, run_id: &str) { + for _ in 0..4 { + let polled = tokio::time::timeout( + Duration::from_millis(200), + worker.poll_workflow_activation(), + ) + .await; + let Ok(Ok(activation)) = polled else { + // Nothing further to deliver: the task is retained, or the worker is done. + return; + }; + + assert!( + !activation.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::SignalWorkflow(_)) + )), + "the reserved Signal must be suppressed whether or not it validates, got {:?}", + activation.jobs + ); + assert_eq!( + resolve_hints(&activation), + Vec::::new(), + "a wake Signal that failed validation must resolve no wait" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(activation.run_id)) + .await + .unwrap(); + } + let _ = run_id; +} + +/// Drives a run whose wait set is still retaining its Workflow Task to completion. +/// +/// A wake that failed validation leaves the set blocked and the task retained -- correctly, since +/// nothing resolved it. Left that way the run would hold its idle deadline until the Worker shut +/// down, so the test resolves the set for real and lets the Workflow finish. +async fn release_a_retained_run(worker: &crate::Worker, run_id: &str) { + if worker.external_stream_run_status(run_id).await != ExternalStreamRunStatus::WftOpen { + return; + } + assert_eq!( + worker.notify_external_stream_ready(run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + let resolved = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + resolved.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); +} + +/// Completes the first (initialize) task and seeds the wait set for the second. +async fn advance_to_the_signal_task( + worker: &crate::Worker, + wait_ids: Vec, + parked_at: Option, +) -> String { + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + worker + .seed_external_stream_waits(&run_id, wait_ids, parked_at, false) + .await; + run_id +} + +#[tokio::test] +async fn an_unparked_wake_resumes_the_run() { + // `park_generation = 0` is the unparked wake: the sender knows of no confirmed park and is + // asking for a Workflow Task anyway. Core validates chain identity and otherwise accepts it + // as a recheck request, because the runtime rechecks every subscription on wakeup regardless + // and an unnecessary one costs at most one empty Workflow Task. + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(0, ""))], + ); + advance_to_the_signal_task(&worker, vec![1, 2], None).await; + + let second = worker.poll_workflow_activation().await.unwrap(); + + // The Signal itself never reaches a user handler, and every active wait is named -- the + // Signal's stream is a hint, not an exhaustive claim. + assert!( + !second.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::SignalWorkflow(_)) + )), + "the reserved Signal must be suppressed from user handlers, got {:?}", + second.jobs + ); + assert_eq!(resolve_hints(&second), vec![1, 2]); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + second.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_intercepted_wake_does_not_flag_its_activation_replaying() { + // The wake Signal is suppressed from user handlers, so the Workflow Task it arrives on + // carries no job of its own and the resolve job is Core's alone. `is_replaying` is derived + // from the activation's job list -- "every job is a query" -- and that test is vacuously true + // over an empty list, so an activation whose only job is queued after it is built comes out + // marked as replay. Lang emits neither stream progress nor quiescence while replaying, so the + // wait generation would never advance, every later readiness report would be answered `Stale` + // against a watcher cursor that had already moved past those records, and the Workflow would + // stall with data sitting in its buffer. This asserts the flag over the same history an + // ordinary Signal is not-replaying on, so it catches the flag and not the history. + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(0, ""))], + ); + advance_to_the_signal_task(&worker, vec![1], None).await; + + let second = worker.poll_workflow_activation().await.unwrap(); + + assert_eq!( + resolve_hints(&second), + vec![1], + "the wake must produce the resolve job this test is about, got {:?}", + second.jobs + ); + assert!( + !second.is_replaying, + "an activation carrying a resolve job produced by an intercepted wake Signal is live \ + work, not replay, got jobs {:?}", + second.jobs + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + second.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_ordinary_signal_on_the_same_history_is_not_replaying_either() { + // The control for the test above: identical history, an ordinary Signal in place of the + // reserved one. If this ever starts failing, the replay flag on the wake path is not the + // thing at fault -- the history these tests are built on stopped being live work. + let worker = worker_with_a_signal("a-user-signal", vec![]); + let first = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(first.run_id)) + .await + .unwrap(); + + let second = worker.poll_workflow_activation().await.unwrap(); + assert!( + !second.is_replaying, + "the second Workflow Task of this history is live work, got jobs {:?}", + second.jobs + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + second.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_wake_naming_a_recognized_park_generation_resumes_the_run() { + // The park handshake that would produce this generation live is C8, so it is injected here + // -- what is under test is the classification, not how the generation came to exist. + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(7, ""))], + ); + advance_to_the_signal_task(&worker, vec![1], Some(7)).await; + + let second = worker.poll_workflow_activation().await.unwrap(); + + assert_eq!(resolve_hints(&second), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + second.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_stale_generation_neither_resumes_nor_reaches_a_handler() { + // A *non-zero* generation the Run does not recognise is a claim that turned out to be wrong, + // so it is ignored -- but it is still suppressed, because a Signal that failed validation + // reaching Workflow code as an unhandled Signal would be worse than dropping it. + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(99, ""))], + ); + let run_id = advance_to_the_signal_task(&worker, vec![1], Some(7)).await; + + assert_the_wake_changed_nothing(&worker, &run_id).await; + release_a_retained_run(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_unknown_envelope_version_is_ignored_harmlessly() { + // An old Core must not break because a newer producer learned a new field. + let mut envelope = wake(0, ""); + envelope.envelope_version = 99; + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(envelope)], + ); + let run_id = advance_to_the_signal_task(&worker, vec![1], None).await; + + assert_the_wake_changed_nothing(&worker, &run_id).await; + release_a_retained_run(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_foreign_chain_neither_resumes_nor_reaches_a_handler() { + // Same chain plus an unknown generation is harmless; a *different* chain is a mis-addressed + // message, and honouring it would wake a Workflow on another Workflow's data. + let worker = worker_with_a_signal( + external_stream::WAKE_SIGNAL_NAME, + vec![wake_payload(wake(0, "some-other-chain"))], + ); + let run_id = advance_to_the_signal_task(&worker, vec![1], None).await; + + assert_the_wake_changed_nothing(&worker, &run_id).await; + release_a_retained_run(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_ordinary_signal_still_reaches_its_handler() { + // The interception must be by name and nothing else. + let worker = worker_with_a_signal("a-user-signal", vec![]); + let first = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(first.run_id)) + .await + .unwrap(); + + let second = worker.poll_workflow_activation().await.unwrap(); + assert!( + second.jobs.iter().any(|j| matches!( + &j.variant, + Some(workflow_activation_job::Variant::SignalWorkflow(s)) + if s.signal_name == "a-user-signal" + )), + "an ordinary Signal must reach its handler, got {:?}", + second.jobs + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + second.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +// --- marker emission (C9, C14b) ---------------------------------------------- + +/// A history for a Workflow Task that records a marker and then starts a timer. +/// +/// Commands are matched to history events in order, so the marker event has to be there and has +/// to come first -- a history missing it hands the marker machine whatever event is next. +fn marker_then_timer_history( + quiescence_generation: u64, + terminal: ParkReason, + annotation: &[u8], +) -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker(quiescence_generation, terminal, annotation); + let timer_started = t.add_by_type(EventType::TimerStarted); + t.add_timer_fired(timer_started, "1".to_string()); + t.add_workflow_task_scheduled_and_started(); + t +} + +/// A worker that records the markers on every completion. +fn worker_recording_markers( + markers: StreamMarkers, + batches: Vec, + history: TestHistoryBuilder, +) -> crate::Worker { + let t = history; + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, batches, mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..4 { + let collected = markers.clone(); + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + mock_worker(mock) +} + +fn output_manifest( + run_id: &str, + history_floor_event_id: i64, + stage_token: &str, +) -> ExternalOutputStreamManifest { + ExternalOutputStreamManifest { + schema_version: 1, + fingerprint_version: 1, + stage_token: stage_token.to_string(), + history_floor_event_id, + run_id: run_id.to_string(), + topics: vec![ExternalOutputTopicManifest { + topic: "results".to_string(), + record_count: 2, + logical_byte_count: 7, + logical_fingerprint: vec![b'f'; 32], + finished: false, + }], + segments: vec![ExternalOutputSegmentManifest { + record_counts_by_topic: vec![2], + }], + provider_id: "test-provider".to_string(), + provider_format_version: 1, + } +} + +fn output_commit_command(manifest: ExternalOutputStreamManifest) -> workflow_command::Variant { + workflow_command::Variant::WorkflowOutputStreamCommit(WorkflowOutputStreamCommit { + manifest: Some(manifest), + request_rollover: false, + }) +} + +fn output_capacity_commit_command( + manifest: ExternalOutputStreamManifest, +) -> workflow_command::Variant { + workflow_command::Variant::WorkflowOutputStreamCommit(WorkflowOutputStreamCommit { + manifest: Some(manifest), + request_rollover: true, + }) +} + +fn output_buffered_command(max_publish_latency: Duration) -> workflow_command::Variant { + workflow_command::Variant::WorkflowOutputStreamBuffered(WorkflowOutputStreamBuffered { + max_publish_latency: Some(max_publish_latency.try_into().unwrap()), + }) +} + +fn immediately_parkable_quiescence_command( + quiescence_generation: u64, + idle_timeout: Duration, +) -> workflow_command::Variant { + workflow_command::Variant::WorkflowStreamQuiescent(WorkflowStreamQuiescent { + quiescence_generation, + waits: vec![ExternalStreamWait { + wait_id: 1, + generation: 0, + immediately_parkable: true, + }], + idle_timeout: Some(idle_timeout.try_into().unwrap()), + }) +} + +fn output_only_marker_history() -> (TestHistoryBuilder, ExternalOutputStreamManifest) { + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + let manifest = output_manifest(t.get_orig_run_id(), 1, "stage-token"); + t.add_full_wf_task(); + t.add_external_stream_marker_data(ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation: 0, + waits: vec![], + replay_annotation: vec![], + terminal_boundary: ParkReason::CommandsProduced as i32, + output: Some(manifest.clone()), + }); + let timer_started = t.add_by_type(EventType::TimerStarted); + t.add_timer_fired(timer_started, "1".to_string()); + t.add_workflow_task_scheduled_and_started(); + (t, manifest) +} + +fn replay_outputs(activation: &WorkflowActivation) -> Vec { + activation + .jobs + .iter() + .filter_map(|job| match &job.variant { + Some(workflow_activation_job::Variant::ReplayExternalStreams(replay)) => { + replay.output.clone() + } + _ => None, + }) + .collect() +} + +#[tokio::test] +async fn activation_carries_the_exact_history_floor_before_its_scheduled_event() { + let mut mock = build_mock_pollers(MockPollCfg::from_resp_batches( + "fake_wf_id", + canned_histories::single_timer("1"), + [1, 2], + mock_worker_client(), + )); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + assert_eq!( + first.history_floor_event_id, 1, + "event 1 immediately precedes the first WorkflowTaskScheduled event" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + let second = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + second.history_floor_event_id, 6, + "TimerFired event 6 immediately precedes the second WorkflowTaskScheduled event; got \ + {second:?}" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn exact_output_floor_excludes_the_previous_workflow_task_close() { + let mut history = TestHistoryBuilder::default(); + history.add_by_type(EventType::WorkflowExecutionStarted); + history.add_full_wf_task(); + let previous_wft_close = history.current_event_id(); + history.add_we_signaled("deciding-event", vec![]); + let exact_floor = history.current_event_id(); + history.add_workflow_task_scheduled_and_started(); + + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", history, [2], mock_worker_client()); + mock_cfg.num_expected_fails = 1; + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + let run_id = replayed.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + + let producing = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(producing.history_floor_event_id, exact_floor); + assert!( + producing.history_floor_event_id > previous_wft_close, + "the previous Workflow Task close must be below, not inside, the deciding interval" + ); + + let mut falsely_low = output_manifest(&run_id, previous_wft_close - 1, "false-floor-token"); + falsely_low.topics[0].logical_fingerprint = vec![b'l'; 32]; + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + output_commit_command(falsely_low), + )) + .await + .unwrap(); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn an_output_only_marker_is_emitted_replayed_and_not_rewritten() { + let (history, manifest) = output_only_marker_history(); + let recorded: Arc>> = Default::default(); + let collected = recorded.clone(); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history.clone(), [1, 2], mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| collected.lock().extend(stream_marker_data(wft))); + asserts.then(|_| {}); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + first.history_floor_event_id, + manifest.history_floor_event_id + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + first.run_id.clone(), + vec![ + output_commit_command(manifest.clone()), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + let fired = worker.poll_workflow_activation().await.unwrap(); + assert!(fired.jobs.iter().any(|job| matches!( + job.variant, + Some(workflow_activation_job::Variant::FireTimer(_)) + ))); + let written = recorded.lock().clone(); + assert_eq!(written.len(), 1, "an output-only task writes one marker"); + assert!(written[0].waits.is_empty()); + assert!(written[0].replay_annotation.is_empty()); + assert_eq!(written[0].output.as_ref(), Some(&manifest)); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + fired.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; + + let replay_markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers(replay_markers.clone(), vec![2], history); + let replayed = worker.poll_workflow_activation().await.unwrap(); + assert!(replayed.is_replaying); + assert_eq!(replay_outputs(&replayed), vec![manifest]); + assert_eq!(resolve_hints(&replayed), Vec::::new()); + assert_eq!(park_jobs(&replayed), Vec::new()); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replayed.run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + let fired = worker.poll_workflow_activation().await.unwrap(); + assert!(fired.jobs.iter().any(|job| matches!( + job.variant, + Some(workflow_activation_job::Variant::FireTimer(_)) + ))); + assert!( + replay_markers.lock().is_empty(), + "the marker found by replay lookahead must not be written again" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + fired.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn output_capacity_replacement_enters_lang_instead_of_autocompleting() { + // Capacity backpressure blocks the publishing Workflow in lang. Its staged commit has no + // server command that could create a job on the forced replacement, so absent a Core-carried + // resume intent the replacement is empty and gets autocompleted without ever releasing the + // publisher. The replacement must contain exactly one runtime resume job even when there are + // no input waits to hint. + let mut history = TestHistoryBuilder::default(); + history.add_by_type(EventType::WorkflowExecutionStarted); + let manifest = output_manifest(history.get_orig_run_id(), 1, "capacity-stage-token"); + history.add_full_wf_task(); + history.add_external_stream_marker_data(ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation: 0, + waits: vec![], + replay_annotation: vec![], + terminal_boundary: ParkReason::OutputCapacity as i32, + output: Some(manifest.clone()), + }); + history.add_workflow_task_scheduled_and_started(); + + let expected = manifest.clone(); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, [1, 2], mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + assert!( + wft.force_create_new_workflow_task, + "capacity completion must request its replacement" + ); + let markers = stream_marker_data(wft); + assert_eq!(markers.len(), 1); + assert_eq!(markers[0].terminal_boundary(), ParkReason::OutputCapacity); + assert_eq!(markers[0].output.as_ref(), Some(&expected)); + }); + asserts.then(|_| {}); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + output_capacity_commit_command(manifest), + )) + .await + .unwrap(); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(replacement.run_id, run_id); + assert!(!replacement.is_replaying); + assert_eq!( + replacement.jobs.len(), + 1, + "the otherwise-empty replacement must produce one lang activation" + ); + let Some(workflow_activation_job::Variant::ResolveExternalStreamWaits(resume)) = + &replacement.jobs[0].variant + else { + panic!("capacity replacement did not carry its runtime resume job: {replacement:?}"); + }; + assert!(resume.ready_hints.is_empty()); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + CompleteWorkflowExecution::default().into(), + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +async fn exercise_speculative_output_redelivery(changed_manifest: bool) { + let workflow_id = if changed_manifest { + "speculative-changed" + } else { + "speculative-identical" + }; + let mut base = TestHistoryBuilder::default(); + base.add_by_type(EventType::WorkflowExecutionStarted); + base.add_full_wf_task(); + base.add_activity_task_scheduled("act1"); + + let mut speculative_history = base.clone(); + speculative_history.add_workflow_task_scheduled_and_started(); + let update_id = "speculative-update"; + let mut first_attempt = + hist_to_poll_resp(&speculative_history, workflow_id, ResponseType::OneTask(2)); + first_attempt.add_update_request(update_id, 1); + let mut redelivery = + hist_to_poll_resp(&speculative_history, workflow_id, ResponseType::OneTask(2)); + redelivery.add_update_request(update_id, 1); + + let completions: Arc>>> = Default::default(); + let reset_ids: Arc>> = Default::default(); + let recorded = completions.clone(); + let recorded_resets = reset_ids.clone(); + let client = mock_worker_client(); + let mut mock_cfg = MockPollCfg::from_resp_batches( + workflow_id, + base, + [ + ResponseType::ToTaskNum(1), + first_attempt.into(), + redelivery.into(), + ], + client, + ); + let mut completion_number = 0; + mock_cfg.completion_mock_fn = Some(Box::new(move |wft| { + completion_number += 1; + recorded.lock().push(stream_marker_data(wft)); + let mut response = RespondWorkflowTaskCompletedResponse::default(); + if completion_number == 2 { + response.reset_history_event_id = 3; + } + recorded_resets.lock().push(response.reset_history_event_id); + Ok(response) + })); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let initial = worker.poll_workflow_activation().await.unwrap(); + let run_id = initial.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + ScheduleActivity { + activity_id: "act1".to_string(), + activity_type: DEFAULT_ACTIVITY_TYPE.to_string(), + ..Default::default() + } + .into(), + )) + .await + .unwrap(); + + let speculative = worker.poll_workflow_activation().await.unwrap(); + let floor = speculative.history_floor_event_id; + let discarded = output_manifest(&run_id, floor, "discarded-stage-token"); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_commit_command(discarded.clone()), + UpdateResponse { + protocol_instance_id: update_id.to_string(), + response: Some(UpdateOutcome::Rejected(Default::default())), + } + .into(), + ], + )) + .await + .unwrap(); + + let redelivered = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + redelivered.history_floor_event_id, floor, + "redelivery reused event IDs and therefore the same exact floor" + ); + let mut accepted = output_manifest(&run_id, floor, "accepted-stage-token"); + if changed_manifest { + accepted.topics[0].logical_fingerprint = vec![b'g'; 32]; + } + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![ + output_commit_command(accepted.clone()), + UpdateResponse { + protocol_instance_id: update_id.to_string(), + response: Some(UpdateOutcome::Accepted(())), + } + .into(), + ], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; + + let written = completions.lock(); + assert_eq!(written.len(), 3); + assert!(written[0].is_empty()); + assert_eq!(written[1].len(), 1); + assert_eq!(written[1][0].output.as_ref(), Some(&discarded)); + assert_eq!(written[2].len(), 1); + assert_eq!(written[2][0].output.as_ref(), Some(&accepted)); + assert_eq!( + reset_ids.lock().as_slice(), + &[0, 3, 0], + "only the first token-bearing completion was discarded; the redelivery was accepted" + ); +} + +#[tokio::test] +async fn speculative_output_redelivery_accepts_a_fresh_token_once() { + exercise_speculative_output_redelivery(false).await; + exercise_speculative_output_redelivery(true).await; +} + +#[tokio::test] +async fn buffered_output_uses_the_earliest_reported_deadline() { + let mut history = TestHistoryBuilder::default(); + history.add_by_type(EventType::WorkflowExecutionStarted); + let manifest = output_manifest(history.get_orig_run_id(), 1, "earliest-stage-token"); + history.add_full_wf_task(); + history.add_external_stream_marker_data(ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation: 0, + waits: vec![], + replay_annotation: vec![], + terminal_boundary: ParkReason::OutputLatency as i32, + output: Some(manifest.clone()), + }); + history.add_workflow_task_scheduled_and_started(); + + let recorded: Arc>> = Default::default(); + let forced = Arc::new(AtomicBool::new(false)); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", history, [1], mock_worker_client()); + let collected = recorded.clone(); + let saw_force = forced.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_marker_data(wft)); + saw_force.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + first.run_id.clone(), + vec![ + output_buffered_command(Duration::from_millis(400)), + output_buffered_command(Duration::from_millis(40)), + ], + )) + .await + .unwrap(); + + let flush = tokio::time::timeout( + Duration::from_millis(250), + worker.poll_workflow_activation(), + ) + .await + .expect("the earlier 40ms output deadline must win") + .unwrap(); + assert_eq!( + finalization_jobs(&flush), + vec![(0, ParkReason::OutputLatency, vec![])] + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + first.run_id, + vec![ + finalized_command(0, b""), + output_commit_command(manifest.clone()), + ], + )) + .await + .unwrap(); + + let written = recorded.lock().clone(); + assert_eq!(written.len(), 1); + assert_eq!(written[0].terminal_boundary(), ParkReason::OutputLatency); + assert_eq!(written[0].output.as_ref(), Some(&manifest)); + assert!(forced.load(Ordering::Relaxed)); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn three_output_flush_windows_cost_three_markers_and_workflow_tasks() { + const WINDOWS: usize = 3; + let mut history = TestHistoryBuilder::default(); + history.add_by_type(EventType::WorkflowExecutionStarted); + let mut next_floor = history.current_event_id(); + let mut manifests = Vec::with_capacity(WINDOWS); + for window in 0..WINDOWS { + let manifest = output_manifest( + history.get_orig_run_id(), + next_floor, + &format!("window-{window}-stage-token"), + ); + history.add_full_wf_task(); + history.add_external_stream_marker_data(ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation: (window + 1) as u64, + waits: vec![ + temporalio_common::protos::coresdk::external_data::ExternalWaitMarker { + wait_id: 1, + generation: 0, + }, + ], + replay_annotation: vec![], + terminal_boundary: ParkReason::OutputLatency as i32, + output: Some(manifest.clone()), + }); + manifests.push(manifest); + if window + 1 < WINDOWS { + history.add_we_signaled("next-output-window", vec![]); + next_floor = history.current_event_id(); + } + } + + let lifecycle: Arc, bool)>>> = Default::default(); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, [1, 2, 3], mock_worker_client()); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + for _ in 0..WINDOWS { + let lifecycle = lifecycle.clone(); + asserts.then(move |wft| { + lifecycle + .lock() + .push((stream_marker_data(wft), wft.force_create_new_workflow_task)); + }); + } + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + for (window, manifest) in manifests.iter().enumerate() { + let activation = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + activation.history_floor_event_id, manifest.history_floor_event_id, + "window {window} staged against the wrong Workflow Task" + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id.clone(), + vec![ + output_buffered_command(Duration::from_secs(90)), + output_buffered_command(Duration::from_secs(60)), + output_buffered_command(Duration::from_secs(30)), + quiescent_command((window + 1) as u64, &[1], Duration::from_secs(300)), + ], + )) + .await + .unwrap(); + + worker.notify_external_output_flush_deadline(&activation.run_id); + assert_eq!( + worker.external_stream_run_status(&activation.run_id).await, + ExternalStreamRunStatus::WftOpen + ); + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![((window + 1) as u64, ParkReason::OutputLatency, vec![1])] + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id, + vec![ + finalized_command((window + 1) as u64, b""), + output_commit_command(manifest.clone()), + ], + )) + .await + .unwrap(); + } + worker.drain_pollers_and_shutdown().await; + + let lifecycle = lifecycle.lock(); + assert_eq!( + lifecycle.len(), + WINDOWS, + "each latency window must complete exactly one Workflow Task" + ); + for (window, ((markers, forced), manifest)) in + lifecycle.iter().zip(manifests.iter()).enumerate() + { + assert_eq!(markers.len(), 1, "window {window} wrote extra markers"); + assert_eq!(markers[0].output.as_ref(), Some(manifest)); + assert_eq!(markers[0].terminal_boundary(), ParkReason::OutputLatency); + assert!( + *forced, + "window {window} did not request its replacement task" + ); + } +} + +fn output_park_boundary_history( + terminal: ParkReason, + wait_generation: u64, + stage_token: &str, + add_signal: bool, +) -> (TestHistoryBuilder, ExternalOutputStreamManifest) { + let mut history = TestHistoryBuilder::default(); + history.add_wfe_started_with_wft_timeout(Duration::from_secs(300)); + let manifest = output_manifest(history.get_orig_run_id(), 1, stage_token); + history.add_full_wf_task(); + history.add_external_stream_marker_data(ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation: 1, + waits: vec![ + temporalio_common::protos::coresdk::external_data::ExternalWaitMarker { + wait_id: 1, + generation: wait_generation, + }, + ], + replay_annotation: vec![], + terminal_boundary: terminal as i32, + output: Some(manifest.clone()), + }); + if add_signal { + history.add_we_signaled("keep-the-run-cached", vec![]); + } + history.add_workflow_task_scheduled_and_started(); + (history, manifest) +} + +#[tokio::test] +async fn confirmed_park_flushes_output_in_its_marker_without_forcing_a_task() { + let (history, manifest) = + output_park_boundary_history(ParkReason::AllWriteFenced, 0, "park-stage-token", true); + let recorded: Arc>> = Default::default(); + let forced = Arc::new(AtomicBool::new(true)); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, [1, 2], mock_worker_client()); + let collected = recorded.clone(); + let saw_force = forced.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_marker_data(wft)); + saw_force.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + asserts.then(|_| {}); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_buffered_command(Duration::from_secs(30)), + immediately_parkable_quiescence_command(1, Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + park_jobs(&park), + vec![(1, ParkReason::AllWriteFenced, vec![1])] + ); + assert_eq!(finalization_jobs(&park), Vec::new()); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_commit_command(manifest.clone()), + park_confirmed_command(1, b""), + ], + )) + .await + .unwrap(); + + let written = recorded.lock().clone(); + assert_eq!(written.len(), 1); + assert_eq!(written[0].terminal_boundary(), ParkReason::AllWriteFenced); + assert_eq!(written[0].output.as_ref(), Some(&manifest)); + assert!( + !forced.load(Ordering::Relaxed), + "a successful park must not force a replacement merely to commit its output" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked + ); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replacement.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn readiness_winner_finalizes_output_already_staged_for_park() { + let (history, manifest) = output_park_boundary_history( + ParkReason::TaskCompleted, + 0, + "stale-park-stage-token", + false, + ); + let recorded: Arc>> = Default::default(); + let forced = Arc::new(AtomicBool::new(false)); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, [1, 2], mock_worker_client()); + let collected = recorded.clone(); + let saw_force = forced.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_marker_data(wft)); + saw_force.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + asserts.then(|_| {}); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_buffered_command(Duration::from_secs(30)), + immediately_parkable_quiescence_command(1, Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park).len(), 1); + + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_commit_command(manifest.clone()), + park_confirmed_command(1, b"stale-terminal-must-not-be-recorded"), + ], + )) + .await + .unwrap(); + + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&finalize), + vec![(1, ParkReason::TaskCompleted, vec![1])] + ); + assert_eq!(resolve_hints(&finalize), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b""), + )) + .await + .unwrap(); + + let written = recorded.lock().clone(); + assert_eq!( + written.len(), + 1, + "the staged batch is recorded exactly once" + ); + assert_eq!(written[0].terminal_boundary(), ParkReason::TaskCompleted); + assert_eq!(written[0].output.as_ref(), Some(&manifest)); + assert!(written[0].replay_annotation.is_empty()); + assert!( + forced.load(Ordering::Relaxed), + "the replacement task must deliver readiness that defeated the park" + ); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&replacement), vec![1]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replacement.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn output_deadline_invalidates_an_issued_park_and_flushes_once() { + let (history, manifest) = + output_park_boundary_history(ParkReason::OutputLatency, 1, "latency-stage-token", false); + + let recorded: Arc>> = Default::default(); + let forced = Arc::new(AtomicBool::new(false)); + let mut mock_cfg = + MockPollCfg::from_resp_batches("fakeid", history, [1, 2], mock_worker_client()); + let collected = recorded.clone(); + let saw_force = forced.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_marker_data(wft)); + saw_force.store(wft.force_create_new_workflow_task, Ordering::Relaxed); + }); + asserts.then(|_| {}); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let first = worker.poll_workflow_activation().await.unwrap(); + let run_id = first.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_buffered_command(Duration::from_millis(400)), + output_buffered_command(Duration::from_millis(40)), + immediately_parkable_quiescence_command(1, Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + park_jobs(&park), + vec![(1, ParkReason::AllWriteFenced, vec![1])], + "buffered output and parking must be allowed to race" + ); + + worker.notify_external_output_flush_deadline(&run_id); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the serialized status probe is a barrier behind the deadline input" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + output_commit_command(manifest.clone()), + park_confirmed_command(1, b"losing-park-terminal"), + ], + )) + .await + .unwrap(); + + let flush = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + finalization_jobs(&flush), + vec![(1, ParkReason::OutputLatency, vec![1])] + ); + assert_eq!(resolve_hints(&flush), vec![1]); + assert_eq!(park_jobs(&flush), Vec::new()); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + finalized_command(1, b""), + )) + .await + .unwrap(); + + let written = recorded.lock().clone(); + assert_eq!(written.len(), 1); + assert_eq!(written[0].terminal_boundary(), ParkReason::OutputLatency); + assert_eq!(written[0].output.as_ref(), Some(&manifest)); + assert!( + forced.load(Ordering::Relaxed), + "a latency flush hands the run to a replacement Workflow Task" + ); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&replacement), vec![1]); + assert_eq!(park_jobs(&replacement), Vec::new()); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replacement.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn several_progress_reports_collapse_into_one_marker() { + // The claim the design's History cost rests on: a retained Workflow Task may span many + // activations, and it writes **one** marker however many of them reported progress. + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1, 2], + // Generation 3: the task became quiescent three times before it ended, and the marker + // closes the last of those snapshots. + marker_then_timer_history(3, ParkReason::CommandsProduced, b"onetwothreefour"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + // Three activations under one Workflow Task, each reporting progress. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"one", false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + for delta in [b"two".as_slice(), b"three".as_slice()] { + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(delta, false), + quiescent_command(1, &[1], Duration::from_secs(30)), + ], + )) + .await + .unwrap(); + } + assert_eq!( + *markers.lock(), + Vec::new(), + "a retained task writes nothing" + ); + + // Now the task ends. + worker.notify_external_stream_ready(&run_id, 1, 0).await; + let _ = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"four", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + let _ = worker.poll_workflow_activation().await.unwrap(); + + let written = markers.lock().clone(); + assert_eq!(written.len(), 1, "exactly one marker per Workflow Task"); + let (_, terminal, annotation) = &written[0]; + assert_eq!(*terminal, ParkReason::CommandsProduced); + assert_eq!( + annotation, b"onetwothreefour", + "the marker carries every delta the task accumulated, concatenated in order" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_terminal_command_writes_its_marker_ordered_before_it() { + // Command ordering is normative: on replay this guarantees a record's integrity is validated + // before the command derived from it is matched. + let markers: StreamMarkers = Default::default(); + let ordering: Arc>> = Default::default(); + let recorded = ordering.clone(); + + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + let collected = markers.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + recorded + .lock() + .extend(wft.commands.iter().map(|c| c.command_type)); + }); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id, + vec![ + progress_command(b"consumed", false), + CompleteWorkflowExecution::default().into(), + ], + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + assert_eq!(written.len(), 1); + assert_eq!(written[0].1, ParkReason::WorkflowCompleted); + + let commands = ordering.lock().clone(); + let marker_at = commands + .iter() + .position(|c| *c == CommandType::RecordMarker as i32) + .expect("the marker must be in the completion"); + let terminal_at = commands + .iter() + .position(|c| *c == CommandType::CompleteWorkflowExecution as i32) + .expect("the terminal command must be in the completion"); + assert!( + marker_at < terminal_at, + "the marker must precede the terminal command, got {commands:?}" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_normal_completion_commits_its_marker() { + // The table row with nothing else on it: records were consumed, no stream wait remained + // pending, and the completion carries a `WorkflowStreamProgress` and *nothing more*. Nothing + // else on it could have triggered the marker, which is the point -- committing consumption is + // not a side effect of some other command riding along. Its terminal says `TaskCompleted` + // rather than `CommandsProduced`, so the marker does not imply commands that never existed. + let markers: StreamMarkers = Default::default(); + // Only the quiescence generation is reconciled against the recorded marker, so what this + // history's own annotation says does not matter; what is read below is what the *completion* + // carried. The Signal is there to give the second task an activation, which is what keeps the + // run cached long enough to be asked what it is still holding. + let worker = + worker_recording_markers(markers.clone(), vec![1, 2], marker_then_signal_history(0)); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + progress_command(b"consumed", false), + )) + .await + .unwrap(); + + let next = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(next.run_id, run_id); + assert_eq!( + *markers.lock(), + vec![(0, ParkReason::TaskCompleted, b"consumed".to_vec())], + "a completion carrying only progress must still commit its marker" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "the annotation is cleared once its marker is written" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +/// Completes one activation with a consumed record followed by `command`. +/// +/// Returns the markers the completion reported and the command types it reported, in order -- +/// everything an ordering assertion needs and nothing else. +async fn markers_and_command_order( + command: workflow_command::Variant, +) -> (Vec, Vec) { + let markers: StreamMarkers = Default::default(); + let ordering: Arc>> = Default::default(); + + let t = canned_histories::single_timer("1"); + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1], mock_worker_client()); + let collected = markers.clone(); + let recorded = ordering.clone(); + mock_cfg.completion_asserts_from_expectations(|mut asserts| { + asserts.then(move |wft| { + collected.lock().extend(stream_markers(wft)); + recorded + .lock() + .extend(wft.commands.iter().map(|c| c.command_type)); + }); + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id, + vec![progress_command(b"consumed", false), command], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; + + let m = markers.lock().clone(); + let o = ordering.lock().clone(); + (m, o) +} + +/// Asserts the completion's marker was reported ahead of `dependent`. +fn assert_the_marker_came_first(commands: &[i32], dependent: CommandType) { + let marker_at = commands + .iter() + .position(|c| *c == CommandType::RecordMarker as i32) + .expect("the marker must be in the completion"); + let dependent_at = commands + .iter() + .position(|c| *c == dependent as i32) + .unwrap_or_else(|| panic!("{dependent:?} must be in the completion, got {commands:?}")); + assert!( + marker_at < dependent_at, + "the marker must precede {dependent:?}, got {commands:?}" + ); +} + +#[tokio::test] +async fn an_activity_command_writes_its_marker_ordered_before_it() { + // The same normative ordering as the terminal case, on the command that leaves the Workflow + // running. Reversed, replay would match the Activity against history before the record it was + // derived from had been validated, so a damaged stream would be discovered only after its + // consequences were already durable. + let (markers, commands) = markers_and_command_order(schedule_activity_cmd( + 1, + "q", + "act-1", + ActivityCancellationType::TryCancel, + Duration::from_secs(10), + Duration::from_secs(10), + )) + .await; + + assert_eq!( + markers, + vec![(0, ParkReason::CommandsProduced, b"consumed".to_vec())], + "an Activity ends the Workflow Task, so the task's consumption is committed with it" + ); + assert_the_marker_came_first(&commands, CommandType::ScheduleActivityTask); +} + +#[tokio::test] +async fn a_failed_workflow_writes_its_marker_ordered_before_the_failure() { + // `CompleteWorkflowExecution` is not the only terminal command, and the ordering rule is + // stated about all four. A path that ordered the marker only ahead of a *successful* + // completion would leave a failing Workflow's last consumption uncommitted -- and a failure is + // exactly when a Workflow is most likely to be retried and replayed. + let (markers, commands) = markers_and_command_order( + FailWorkflowExecution { + failure: Some( + temporalio_common::protos::temporal::api::failure::v1::Failure { + message: "deliberate".to_string(), + ..Default::default() + }, + ), + } + .into(), + ) + .await; + + assert_eq!( + markers, + vec![(0, ParkReason::WorkflowCompleted, b"consumed".to_vec())], + "a failing Workflow commits its consumption exactly as a completing one does" + ); + assert_the_marker_came_first(&commands, CommandType::FailWorkflowExecution); +} + +#[tokio::test] +async fn a_continue_as_new_writes_its_marker_ordered_before_it() { + // The terminal command that matters most here: the next Run resumes its subscriptions from the + // committed continuation state, so a marker written *after* the Continue-As-New -- or not at + // all -- would hand the successor a cursor that predates records this Run already delivered. + let (markers, commands) = markers_and_command_order( + ContinueAsNewWorkflowExecution { + workflow_type: "successor".to_string(), + ..Default::default() + } + .into(), + ) + .await; + + assert_eq!( + markers, + vec![(0, ParkReason::WorkflowCompleted, b"consumed".to_vec())], + "continue-as-new is a terminal command like any other, and commits its marker like one" + ); + assert_the_marker_came_first(&commands, CommandType::ContinueAsNewWorkflowExecution); +} + +#[tokio::test] +async fn replaying_a_marker_before_an_activity_delivers_its_record_exactly_once() { + // The replay half of the Activity case. The marker written ahead of the Activity is read back + // ahead of it, and *once*: a lookahead that re-issued the recorded observations on the second + // Workflow Task -- the one the Activity's result arrives on -- would deliver every record + // twice while the command they produced is already durable. + // + // Driven through `init_replay_worker`, which is the path a whole-history replay actually + // takes. + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker_covering( + 1, + ParkReason::CommandsProduced, + b"header.segment.terminal", + &[(1, 0)], + ); + let scheduled = t.add_activity_task_scheduled("act-1"); + let started = t.add_activity_task_started(scheduled); + t.add_activity_task_completed(scheduled, started, b"result".into()); + t.add_full_wf_task(); + t.add_workflow_execution_completed(); + + let worker = crate::init_replay_worker(crate::replay::ReplayWorkerInput::new( + crate::test_help::test_worker_cfg().build().unwrap(), + futures_util::stream::iter([crate::replay::HistoryForReplay::from(t)]), + )) + .unwrap(); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + assert!(replayed.is_replaying); + assert_eq!( + replay_jobs(&replayed), + vec![( + 1, + ParkReason::CommandsProduced, + vec![(1, 0)], + b"header.segment.terminal".to_vec() + )], + "the recorded observations must reach lang on the task that consumed them, got {:?}", + replayed.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replayed.run_id.clone(), + vec![schedule_activity_cmd( + 1, + "q", + "act-1", + ActivityCancellationType::TryCancel, + Duration::from_secs(10), + Duration::from_secs(10), + )], + )) + .await + .unwrap(); + + // The Activity resolves on the next Workflow Task, and that task replays no records: they + // belong to the marker of the task before it, which has already been delivered. + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert!( + resolved.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::ResolveActivity(_)) + )), + "the Activity that followed the marker must still match its event, got {:?}", + resolved.jobs + ); + assert_eq!( + replay_jobs(&resolved), + Vec::new(), + "the record was delivered on the task that consumed it, and must not be delivered again \ + on the task its Activity resolves on, got {:?}", + resolved.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + resolved.run_id.clone(), + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + + // The replay worker ends its stream only when the history ran to completion without a + // nondeterminism failure, so this is what makes the whole replay the thing under test. + assert!( + matches!( + worker.poll_workflow_activation().await, + Err(PollError::ShutDown) + ), + "replay must run the history to its end" + ); +} + +#[tokio::test] +async fn a_workflow_task_that_observed_nothing_writes_no_marker() { + // Not an error: a Workflow Task that touched no stream is the ordinary case, and writing an + // empty marker for it would put a History event on every task in the Workflow. + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1, 2], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + let _ = worker.poll_workflow_activation().await.unwrap(); + + assert_eq!(*markers.lock(), Vec::new()); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_annotation_with_no_terminal_is_refused_rather_than_written() { + // The refusal belongs to emission itself and needs no finalization job to state it: an + // annotation without its terminal is durable and wrong, while an abandoned Workflow Task + // commits no cursor and loses no record. So there is no best-effort path. + // + // Driven directly at the emission primitive, because no completion path can produce this -- + // which is the point: the guard exists for a future path that forgets. + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1, 2], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + + assert!( + worker + .emit_terminal_less_external_stream_marker(&run_id) + .await, + "emitting an annotation with no terminal boundary must be refused" + ); + + // And nothing was written: the completion that follows carries no marker at all. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + let _ = worker.poll_workflow_activation().await.unwrap(); + + assert_eq!( + *markers.lock(), + Vec::new(), + "a refused emission must write nothing rather than writing a truncated annotation" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +// --- the complete-set park handshake (C8) ------------------------------------ + +/// The `PrepareExternalStreamPark` jobs in an activation, as (generation, reason, wait ids). +/// +/// Reading the reason back out is what separates the two triggers: an idle expiry and an +/// all-fenced snapshot produce the same shape of job and the same shape of marker, and only this +/// field says which of them actually happened. +fn park_jobs(activation: &WorkflowActivation) -> Vec<(u64, ParkReason, Vec)> { + activation + .jobs + .iter() + .filter_map(|j| match &j.variant { + Some(workflow_activation_job::Variant::PrepareExternalStreamPark(p)) => Some(( + p.quiescence_generation, + p.reason(), + p.waits.iter().map(|w| w.wait_id).collect(), + )), + _ => None, + }) + .collect() +} + +/// Lang's `ParkSetConfirmed`: every stream was still empty after the intents went in. +fn park_confirmed_command( + quiescence_generation: u64, + terminal: &[u8], +) -> workflow_command::Variant { + workflow_command::Variant::ExternalStreamParkResult(ExternalStreamParkResult { + quiescence_generation, + outcome: Some(external_stream_park_result::Outcome::Confirmed( + ParkSetConfirmed {}, + )), + final_observation_delta: terminal.to_vec(), + }) +} + +/// Lang's `StreamSetBecameReady`: the final recheck found records, so this generation is abandoned. +fn park_became_ready_command(quiescence_generation: u64) -> workflow_command::Variant { + workflow_command::Variant::ExternalStreamParkResult(ExternalStreamParkResult { + quiescence_generation, + outcome: Some(external_stream_park_result::Outcome::BecameReady( + StreamSetBecameReady {}, + )), + // Deliberately empty: an abandoned park reached no boundary, so there is no terminal to + // carry and nothing for Core to write. + final_observation_delta: vec![], + }) +} + +/// A quiescent snapshot whose waits carry individual write-fence states. +fn fenced_quiescent_command( + quiescence_generation: u64, + waits: &[(u32, bool)], + idle_timeout: Duration, +) -> workflow_command::Variant { + workflow_command::Variant::WorkflowStreamQuiescent(WorkflowStreamQuiescent { + quiescence_generation, + waits: waits + .iter() + .map(|(wait_id, immediately_parkable)| ExternalStreamWait { + wait_id: *wait_id, + generation: 0, + immediately_parkable: *immediately_parkable, + }) + .collect(), + idle_timeout: Some(idle_timeout.try_into().unwrap()), + }) +} + +/// Reports `annotation` and the given quiescent snapshot, leaving the task retained. +/// +/// This is the state every park begins from: lang has reported what it observed and asked to be +/// held open, Core is holding the task, and no activation is outstanding. +async fn retain_a_quiescent_task( + worker: &crate::Worker, + annotation: &[u8], + quiescence: workflow_command::Variant, +) -> String { + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![progress_command(annotation, false), quiescence], + )) + .await + .unwrap(); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the quiescent snapshot must have been accepted, or this test proves nothing" + ); + run_id +} + +/// A history whose first Workflow Task records a marker and whose second carries a Signal. +/// +/// The marker event has to be present and first, because commands are matched to events in order. +/// The Signal is there only to give the replacement task an activation -- a run with nothing left +/// to do is evicted, and a test cannot ask an evicted run whether it parked. +fn marker_then_signal_history(quiescence_generation: u64) -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + // Long enough that no rollover deadline or clamped idle deadline can fire inside the test. + t.add_wfe_started_with_wft_timeout(Duration::from_secs(300)); + t.add_full_wf_task(); + t.add_external_stream_marker(quiescence_generation, ParkReason::Idle, b"recorded"); + t.add_we_signaled("keep-the-run-cached", vec![]); + t.add_workflow_task_scheduled_and_started(); + t +} + +/// A workflow-only worker recording every completion's markers and `force_new_wft`. +fn worker_for_a_park( + markers: StreamMarkers, + forced: Arc>>, + history: TestHistoryBuilder, + batches: Vec, +) -> crate::Worker { + worker_recording_rollovers(markers, forced, history, batches) +} + +#[tokio::test] +async fn a_confirmed_idle_park_writes_one_marker_and_completes_the_task() { + // The confirmation is the only place Core can obtain this boundary's terminal: park issues no + // finalization job, so if the result's `final_observation_delta` were dropped the marker would + // be a truncated annotation -- durable and wrong. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_for_a_park( + markers.clone(), + forced.clone(), + marker_then_signal_history(1), + vec![1, 2], + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1, 2], Duration::from_millis(30)), + ) + .await; + + // The idle timer expired and Core asked for the complete set, not just the wait that went + // quiet: parking is all-or-nothing. + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + park_jobs(&park), + vec![(1, ParkReason::Idle, vec![1, 2])], + "an idle expiry must ask lang to park the complete set, got {:?}", + park.jobs + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "nothing may be written before the terminal exists" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_confirmed_command(1, b"-parked"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![(1, ParkReason::Idle, b"observed-parked".to_vec())], + "the park's marker carries the accumulated annotation with the confirmation's terminal" + ); + assert_eq!( + *forced.lock(), + vec![false], + "parking is the opposite of asking for a replacement task" + ); + + // The set is parked: readiness can no longer be delivered locally, and a producer now has a + // generation to name in its wake Signal. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked + ); + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Parked + ); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replacement.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn an_all_fenced_snapshot_parks_without_waiting_out_the_idle_timeout() { + // The fence says no later record is coming, so waiting out the idle delay could only ever end + // the same way. The idle timeout here is five minutes: if the park were coming from the timer + // rather than from the fences, this test would hang rather than fail. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_for_a_park( + markers.clone(), + forced.clone(), + marker_then_signal_history(2), + vec![1, 2], + ); + + // One fenced stream out of two does not qualify. The set is retained and nothing is asked. + let run_id = retain_a_quiescent_task( + &worker, + b"half", + fenced_quiescent_command(1, &[(1, true), (2, false)], Duration::from_secs(300)), + ) + .await; + assert!( + tokio::time::timeout( + Duration::from_millis(300), + worker.poll_workflow_activation() + ) + .await + .is_err(), + "one fenced stream must not park a set another stream is still driving" + ); + + // Now the second stream fences too, and the set parks immediately. + worker.notify_external_stream_ready(&run_id, 2, 0).await; + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![2]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"-rest", false), + fenced_quiescent_command(2, &[(1, true), (2, true)], Duration::from_secs(300)), + ], + )) + .await + .unwrap(); + + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + park_jobs(&park), + vec![(2, ParkReason::AllWriteFenced, vec![1, 2])], + "an all-fenced snapshot must park on its own, got {:?}", + park.jobs + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_confirmed_command(2, b"-fenced"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + vec![(2, ParkReason::AllWriteFenced, b"half-rest-fenced".to_vec())], + "the marker must say which trigger actually parked the set" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked + ); + + let replacement = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replacement.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn readiness_accepted_before_the_confirmation_wins_and_no_marker_is_written() { + // The first ordering of the race. A record arrived while the handshake was in flight, so the + // confirmation that follows is closing a boundary that was never reached. Writing its marker + // would commit a cursor past a record the Workflow has not seen. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_for_a_park( + markers.clone(), + forced.clone(), + canned_histories::single_timer("1"), + vec![1], + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park).len(), 1); + + // Readiness lands while lang is still installing its intents. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted, + "readiness during the handshake must still be accepted -- accepting it is the abort" + ); + + // Lang's recheck saw nothing and confirms anyway. Core knows better. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_confirmed_command(1, b"-parked"), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + Vec::new(), + "a confirmation readiness already beat must write no marker" + ); + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked, + "the set must not be parked with a record buffered for it" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "the task stays open so the resolve Core owes lang has somewhere to arrive" + ); + // And the annotation is still held, unwritten, for whatever boundary does eventually close it. + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"observed" + ); + + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![1]); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_recheck_that_became_ready_resolves_the_waits_rather_than_running_user_code() { + // The other way the handshake can end. `PrepareExternalStreamPark` runs no user Workflow + // code, so a recheck that found records cannot deliver them from inside the park path: it + // abandons the generation and Core issues an ordinary resolve, which is the job that does run + // user code. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_for_a_park( + markers.clone(), + forced.clone(), + canned_histories::single_timer("1"), + vec![1], + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1, 2], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park).len(), 1); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_became_ready_command(1), + )) + .await + .unwrap(); + + assert_eq!( + *markers.lock(), + Vec::new(), + "an aborted park reached no boundary, so it writes no marker" + ); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen + ); + + // A normal resolve activation naming *every* wait: the recheck covered all of them, so any of + // them may now have data. + let resolved = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(resolve_hints(&resolved), vec![1, 2]); + assert_eq!( + park_jobs(&resolved), + Vec::new(), + "the abort must not start a second handshake" + ); + + // The abort bumped every wait generation, which is what makes a readiness notification issued + // against the abandoned block stale rather than resolving against the new one. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Stale + ); + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 1).await, + ExternalStreamReadyResult::Accepted + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_confirmation_naming_a_generation_that_is_not_parking_is_discarded() { + // A stale confirmation is neither a park nor an abort. Without the generation check it would + // park a set whose snapshot has moved on, stranding whatever the current one is holding. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_for_a_park( + markers.clone(), + forced.clone(), + canned_histories::single_timer("1"), + vec![1], + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park), vec![(1, ParkReason::Idle, vec![1])]); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_confirmed_command(99, b"-parked"), + )) + .await + .unwrap(); + + assert_eq!(*markers.lock(), Vec::new()); + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::Parked, + "a confirmation for a generation that is not parking must park nothing" + ); + assert_eq!( + worker.external_stream_annotation(&run_id).await, + b"observed" + ); + + // Still retained, so the wait is still deliverable locally -- the discarded confirmation + // changed nothing at all. + assert_eq!( + worker.notify_external_stream_ready(&run_id, 1, 0).await, + ExternalStreamReadyResult::Accepted + ); + consume_resolve_activation(&worker, &run_id).await; + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_park_job_answered_without_a_result_writes_no_marker() { + // The paired negative of the confirmed case. Core asked for a terminal and did not get one, + // and there is no best-effort path from there: an abandoned Workflow Task commits no cursor + // and loses no record, while a truncated annotation is durable and wrong. + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + canned_histories::single_timer("1"), + [1], + mock_worker_client(), + ); + mock_cfg.num_expected_fails = 1; + mock_cfg.num_expected_completions = Some(0.into()); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park).len(), 1); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "a park answered without a result must fail the Workflow Task" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn an_unprompted_park_result_is_refused() { + // Without this the "answered correctly" case above would pass either way: accepting an + // unrequested result would let lang append a terminal to an annotation Core never asked it to + // close, and park a set Core never asked it to park. + let mut mock_cfg = MockPollCfg::from_resp_batches( + "fakeid", + canned_histories::single_timer("1"), + [1], + mock_worker_client(), + ); + mock_cfg.num_expected_fails = 1; + mock_cfg.num_expected_completions = Some(0.into()); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"unprompted", false), + park_confirmed_command(1, b"-parked"), + ], + )) + .await + .unwrap(); + + assert_ne!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "an unprompted park result must fail the Workflow Task rather than be accepted" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +// --- replay marker lookahead (C10) ------------------------------------------- + +/// The `ReplayExternalStreams` jobs in an activation, as (generation, terminal, waits, annotation). +/// +/// Everything a replayed Workflow Task needs to reproduce what the live one saw. Reading it back +/// whole is what makes "the recorded observations reach lang unchanged" checkable rather than +/// assumed -- Core is annotation-blind, so a copy that mangled the bytes would look like success. +#[allow(clippy::type_complexity)] +fn replay_jobs( + activation: &WorkflowActivation, +) -> Vec<(u64, ParkReason, Vec<(u32, u64)>, Vec)> { + activation + .jobs + .iter() + .filter_map(|j| match &j.variant { + Some(workflow_activation_job::Variant::ReplayExternalStreams(r)) => Some(( + r.quiescence_generation, + r.terminal_boundary(), + r.waits.iter().map(|w| (w.wait_id, w.generation)).collect(), + r.replay_annotation.clone(), + )), + _ => None, + }) + .collect() +} + +/// A history whose first Workflow Task parked on a marker and whose second fires a timer. +fn replayable_marker_history() -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker_covering( + 2, + ParkReason::Idle, + b"header.segment.terminal", + &[(1, 3), (2, 0)], + ); + let timer_started = t.add_by_type(EventType::TimerStarted); + t.add_timer_fired(timer_started, "1".to_string()); + t.add_workflow_task_scheduled_and_started(); + t +} + +#[tokio::test] +async fn a_replayed_marker_resolves_the_wait_set_with_no_readiness_path() { + // The whole point of the marker. On replay the backend is not consulted for *what* was seen -- + // no watcher fires, no readiness is accepted, no idle timer runs and no park is proposed -- + // and yet lang is handed the observations, in the same activation that consumed them live. + let markers: StreamMarkers = Default::default(); + // One batch containing the whole history, so the first Workflow Task is replayed rather than + // executed. That is what puts the marker in the *next* sequence, where lookahead finds it. + let worker = worker_recording_markers(markers.clone(), vec![2], replayable_marker_history()); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + let run_id = replayed.run_id.clone(); + assert!( + replayed.is_replaying, + "this test is only meaningful while replaying, got {replayed:?}" + ); + assert_eq!( + replay_jobs(&replayed), + vec![( + 2, + ParkReason::Idle, + vec![(1, 3), (2, 0)], + b"header.segment.terminal".to_vec() + )], + "the recorded snapshot, waits, terminal and annotation must all reach lang, got {:?}", + replayed.jobs + ); + // Not a live resolve and not a park: replay reproduces the recorded boundary rather than + // re-deciding it, so neither of the paths that need a backend is entered. + assert_eq!(resolve_hints(&replayed), Vec::::new()); + assert_eq!(park_jobs(&replayed), Vec::new()); + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask, + "replay must register no wait set, so no idle or rollover deadline can be running" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + // The marker in history settled the machine lookahead created for it, and nothing was written + // back: re-issuing the command would be matched against the very event it was read from. + let fired = worker.poll_workflow_activation().await.unwrap(); + assert!( + fired.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::FireTimer(_)) + )), + "history after the marker must keep matching, got {:?}", + fired.jobs + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "a replayed marker must not be written a second time" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_completion_while_replaying_writes_no_second_marker() { + // The paired negative for the assertion above. If lang reports progress on a replayed + // activation -- a bug, but a plausible one -- Core must not turn it into a command: the marker + // for that Workflow Task is already durable, and a second one would be matched against the + // event the first produced. + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers(markers.clone(), vec![2], replayable_marker_history()); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + let run_id = replayed.run_id.clone(); + assert_eq!(replay_jobs(&replayed).len(), 1); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id.clone(), + vec![ + progress_command(b"re-observed", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + let fired = worker.poll_workflow_activation().await.unwrap(); + assert!( + fired.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::FireTimer(_)) + )), + "a replayed progress report must not disturb command matching, got {:?}", + fired.jobs + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "replay writes nothing: the marker it would write is the one it just read" + ); + assert!( + worker.external_stream_annotation(&run_id).await.is_empty(), + "and the delta is not left accumulated to leak into the next task's marker" + ); + + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; +} + +/// A replayable history whose recorded boundary is followed by a Signal, on a short task timeout. +/// +/// Two things matter. The Signal rather than a timer means the replayed Workflow Task can be +/// completed with **no** server-bound command, so retention is refused for one reason only -- that +/// the activation was a replay -- and a test can tell that reason from the others. The short task +/// timeout puts both deadlines a retained snapshot would arm (the idle timeout and the rollover +/// deadline at 80% of the task timeout) inside the test's own wait rather than beyond its end. +fn replayable_marker_then_signal_history() -> TestHistoryBuilder { + let mut t = TestHistoryBuilder::default(); + t.add_wfe_started_with_wft_timeout(Duration::from_millis(200)); + t.add_full_wf_task(); + t.add_external_stream_marker_covering( + 2, + ParkReason::Idle, + b"header.segment.terminal", + &[(1, 3), (2, 0)], + ); + t.add_we_signaled("keep-the-run-going", vec![]); + t.add_full_wf_task(); + t.add_workflow_execution_completed(); + t +} + +#[tokio::test] +async fn a_replaying_quiescence_registers_its_waits_and_arms_no_timer() { + // A replayed Run has to rebuild its wait set: that set is per-Worker runtime state, not + // History, so nothing else recreates it and lang reports its quiescent snapshot while + // replaying too. But Core must run no timer for it. The recorded marker is what reproduces the + // boundary, and a replay slower than the idle timeout would otherwise queue a park handshake + // in between replay activations -- proposing a park for a boundary History already settled, + // against wall-clock time that has nothing to do with the recorded one. + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced.clone(), + replayable_marker_then_signal_history(), + // One batch carrying both Workflow Tasks, so the first is replayed rather than executed. + vec![2], + ); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + let run_id = replayed.run_id.clone(); + assert!( + replayed.is_replaying, + "this test is only meaningful while replaying, got {replayed:?}" + ); + assert_eq!( + replay_jobs(&replayed).len(), + 1, + "the recorded boundary must reach lang, got {:?}", + replayed.jobs + ); + + // Lang reports the snapshot it rebuilt, and nothing else -- no server-bound command rides + // along, so replay is the only thing standing between this and retention. + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + quiescent_command(1, &[1, 2], Duration::from_millis(30)), + )) + .await + .unwrap(); + + // Registered: an empty wait set answers `NoOpenWorkflowTask` here whatever the task is doing. + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::WftOpen, + "a replayed Run must rebuild the wait set nothing else recreates" + ); + + // Hold an activation outstanding across the wait, so a timer that did fire could only queue + // its job rather than racing this poll, and the job is then delivered on the next one. + let signalled = worker.poll_workflow_activation().await.unwrap(); + tokio::time::sleep(Duration::from_millis(400)).await; + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + signalled.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + + // Whatever comes next -- an eviction, or nothing at all -- must not be a park handshake or a + // finalization request, because neither deadline was ever armed. + if let Ok(Ok(after)) = tokio::time::timeout( + Duration::from_millis(300), + worker.poll_workflow_activation(), + ) + .await + { + assert_eq!( + park_jobs(&after), + Vec::new(), + "replay must run no idle timer: the recorded marker is the boundary, got {:?}", + after.jobs + ); + assert_eq!( + finalization_jobs(&after), + Vec::new(), + "and no rollover deadline either -- a replayed task holds no retention to roll over, \ + got {:?}", + after.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(after.run_id)) + .await + .unwrap(); + } + + assert!( + !forced.lock().iter().any(|f| *f), + "no completion may ask for a replacement task on behalf of a boundary History already \ + recorded, got {:?}", + forced.lock() + ); + assert_eq!( + *markers.lock(), + Vec::new(), + "and replay writes no marker: the one it would write is the one it just read" + ); + + worker.drain_pollers_and_shutdown().await; +} + +#[tokio::test] +async fn a_marker_no_machine_expects_is_reported_as_nondeterminism() { + // Handled the way local activities handle the same case: history and the machines disagree + // about what this run did, and that is nondeterminism, not something to skip past. Core writes + // exactly one external stream marker per Workflow Task, so a marker that neither a command nor + // a lookahead accounts for cannot be reconciled with anything. + // + // Delivered in two batches so the first task is *executed* rather than replayed: with only the + // first task's events in hand, lookahead has nothing to find, and the marker then arrives on + // the next task with no machine expecting it. + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker(1, ParkReason::Idle, b"written-by-nobody"); + t.add_workflow_task_scheduled_and_started(); + + let mut mock_cfg = MockPollCfg::from_resp_batches("fakeid", t, [1, 2], mock_worker_client()); + mock_cfg.num_expected_fails = 1; + // Asserting the *reason* the task failed, not merely that it did: without this the test would + // pass on any failure at all, including the generic "no command scheduled for event" a marker + // would hit if Core simply had no idea what an external stream marker was. + let saw_the_right_failure = Arc::new(AtomicBool::new(false)); + let recorder = saw_the_right_failure.clone(); + mock_cfg.expect_fail_wft_matcher = Box::new(move |_, cause, failure| { + let message = failure + .as_ref() + .map(|f| f.message.clone()) + .unwrap_or_default(); + recorder.store( + message.contains("no state machine expecting it") + && matches!( + cause, + temporalio_common::protos::temporal::api::enums::v1::WorkflowTaskFailedCause::NonDeterministicError + ), + Ordering::Relaxed, + ); + true + }); + let mut mock = build_mock_pollers(mock_cfg); + mock.worker_cfg(|w| { + w.task_types = WorkerTaskTypes::workflow_only(); + w.max_cached_workflows = 1; + }); + let worker = mock_worker(mock); + + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + assert_eq!( + replay_jobs(&activation), + Vec::new(), + "lookahead cannot have found the marker, or this test proves nothing" + ); + + // Lang produces nothing, so no marker command exists for the event that follows. + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(run_id.clone())) + .await + .unwrap(); + + // The next task carries the unexplained marker and fails. `num_expected_fails` is what asserts + // the failure actually reached the server. + let next = tokio::time::timeout( + Duration::from_millis(500), + worker.poll_workflow_activation(), + ) + .await; + if let Ok(Ok(act)) = next { + assert!( + act.is_only_eviction(), + "the unexplained marker must fail the task rather than activate lang, got {:?}", + act.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::empty(act.run_id)) + .await + .unwrap(); + } + + assert!( + saw_the_right_failure.load(Ordering::Relaxed), + "the marker must fail the task as nondeterminism naming the missing machine" + ); + + worker.shutdown().await; + worker.finalize_shutdown().await; +} + +#[tokio::test] +async fn the_replay_worker_claims_a_marker_followed_by_langs_own_command() { + // The shape a real Worker actually writes, driven through the entry point lang's `Replayer` + // actually uses. + // + // Core emits the marker command *before* lang's commands are pushed into the machines, so in + // History the marker sits between the Workflow Task's completion and the command that task + // produced. On replay the lookahead has to claim that marker; if it does not, the marker event + // reaches the next machine in the command queue -- lang's timer -- and replay fails with a + // nondeterminism error naming a machine that has nothing to do with streams. + // + // `init_replay_worker` rather than a mock poller because that is the path a whole-history + // replay takes: one poll response carrying every event, a previous-started id from the *last* + // Workflow Task, and no live task ever arriving. The mock-poller tests reach the same machines + // by a different route and would not have caught a difference between the two. + let mut t = TestHistoryBuilder::default(); + t.add_by_type(EventType::WorkflowExecutionStarted); + t.add_full_wf_task(); + t.add_external_stream_marker_covering( + 2, + ParkReason::CommandsProduced, + b"header.segment.terminal", + &[(1, 0)], + ); + let timer_started = t.add_by_type(EventType::TimerStarted); + t.add_timer_fired(timer_started, "1".to_string()); + t.add_full_wf_task(); + t.add_workflow_execution_completed(); + + let worker = crate::init_replay_worker(crate::replay::ReplayWorkerInput::new( + crate::test_help::test_worker_cfg().build().unwrap(), + futures_util::stream::iter([crate::replay::HistoryForReplay::from(t)]), + )) + .unwrap(); + + let replayed = worker.poll_workflow_activation().await.unwrap(); + assert!(replayed.is_replaying); + assert_eq!( + replay_jobs(&replayed), + vec![( + 2, + ParkReason::CommandsProduced, + vec![(1, 0)], + b"header.segment.terminal".to_vec() + )], + "the lookahead must claim the marker even though lang's own command follows it in the \ + same Workflow Task, got {:?}", + replayed.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + replayed.run_id.clone(), + vec![start_timer_cmd(1, Duration::from_secs(3))], + )) + .await + .unwrap(); + + // Reaching the timer at all is the second half of the assertion: the marker was consumed by + // the machine lookahead created for it, so the command queue still lines up with history. + let fired = worker.poll_workflow_activation().await.unwrap(); + assert!( + fired.jobs.iter().any(|j| matches!( + j.variant, + Some(workflow_activation_job::Variant::FireTimer(_)) + )), + "the timer that followed the marker must still match its event, got {:?}", + fired.jobs + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + fired.run_id.clone(), + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + + // The replay worker ends the stream only when the history ran to completion without a + // nondeterminism failure, so this is what makes the whole replay -- not just the two + // activations above -- the thing under test. + assert!( + matches!( + worker.poll_workflow_activation().await, + Err(PollError::ShutDown) + ), + "replay must run the history to its end" + ); +} + +// --- the finalization-ownership table, as one assertion ---------------------- +// +// Every per-path integration is tested above, in the section that owns it. What none of them can +// say is what the table says as a whole: that *each* completion path writes exactly one marker and +// that each such marker ends with a terminal -- and that the two paths which reach no boundary +// write none. A path that quietly wrote nothing, or wrote a marker whose annotation stops short of +// its terminal, would leave every test above green. + +/// The suffix each path's terminal carries, whichever round trip supplied it. +/// +/// Core is annotation-blind, so "ends with a terminal" is only checkable if the test knows what a +/// terminal looks like. Every path below ends its annotation with this and nothing else does. +const TERMINAL_SUFFIX: &[u8] = b"|end"; + +/// Row one: records consumed, no wait pending, and nothing else on the completion. +async fn markers_from_a_normal_completion() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + activation.run_id, + progress_command(b"observed|end", false), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row two: a server-bound command rides along, so the task is reported rather than retained. +async fn markers_from_a_command_producing_completion() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id, + vec![ + progress_command(b"observed|end", false), + start_timer_cmd(1, Duration::from_secs(10)), + ], + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row three: a terminal command ends the Workflow. +async fn markers_from_a_terminal_command() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + activation.run_id, + vec![ + progress_command(b"observed|end", false), + CompleteWorkflowExecution::default().into(), + ], + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row four: the idle timer expired and lang confirmed the park, carrying the terminal. +async fn markers_from_a_confirmed_park() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers(markers.clone(), vec![1], marker_then_signal_history(1)); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park), vec![(1, ParkReason::Idle, vec![1])]); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + park_confirmed_command(1, b"|end"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row five: every wait was fenced, so the set parked without waiting the idle delay out. +async fn markers_from_an_all_fenced_park() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers(markers.clone(), vec![1], marker_then_signal_history(1)); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + // Five minutes: if this parked from the timer rather than from the fence the helper would + // hang rather than return the wrong answer. + fenced_quiescent_command(1, &[(1, true)], Duration::from_secs(300)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!( + park_jobs(&park), + vec![(1, ParkReason::AllWriteFenced, vec![1])] + ); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + park_confirmed_command(1, b"|end"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row six: the rollover deadline expired, and lang returned the terminal Core cannot encode. +async fn markers_from_a_deadline_rollover() -> Vec { + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced, + marker_then_replacement_history(1, ParkReason::Rollover, b"observed|end"), + vec![1], + ); + + let run_id = retain_then_fire_the_rollover_deadline(&worker, b"observed").await; + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(finalization_jobs(&finalize).len(), 1); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + finalized_command(1, b"|end"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row seven: lang decided the boundary at the byte budget, so its own report carried the terminal. +async fn markers_from_a_budget_rollover() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + let activation = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + activation.run_id, + progress_command(b"observed|end", true), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// Row eight: the Worker shut down while this Run still held its Workflow Task. +async fn markers_from_a_shutdown_with_a_task_open() -> Vec { + let markers: StreamMarkers = Default::default(); + let forced: Arc>> = Default::default(); + let worker = worker_recording_rollovers( + markers.clone(), + forced, + marker_then_replacement_history(1, ParkReason::Shutdown, b"observed|end"), + vec![1], + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_secs(30)), + ) + .await; + worker.initiate_shutdown(); + let finalize = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(finalization_jobs(&finalize).len(), 1); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id, + finalized_command(1, b"|end"), + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.shutdown().await; + worker.finalize_shutdown().await; + written +} + +/// The first row that writes nothing: lang's recheck found records, so no boundary was reached. +async fn markers_from_an_aborted_park() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + let run_id = retain_a_quiescent_task( + &worker, + b"observed", + quiescent_command(1, &[1], Duration::from_millis(30)), + ) + .await; + let park = worker.poll_workflow_activation().await.unwrap(); + assert_eq!(park_jobs(&park).len(), 1); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmd( + run_id.clone(), + park_became_ready_command(1), + )) + .await + .unwrap(); + + // Snapshotted before the run is released: the annotation the abort left unwritten belongs to + // whatever boundary does eventually close it, and that later marker is another row's business. + let written = markers.lock().clone(); + let resolved = worker.poll_workflow_activation().await.unwrap(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + resolved.run_id, + vec![CompleteWorkflowExecution::default().into()], + )) + .await + .unwrap(); + worker.drain_pollers_and_shutdown().await; + written +} + +/// The second row that writes nothing: the Worker shut down between this Run's Workflow Tasks. +async fn markers_from_a_shutdown_with_no_open_task() -> Vec { + let markers: StreamMarkers = Default::default(); + let worker = worker_recording_markers( + markers.clone(), + vec![1], + canned_histories::single_timer("1"), + ); + + // The first activation is left outstanding for the whole helper, which is what keeps the Run + // cached: a Run that goes idle here is dropped when the mock runs out of work. + let activation = worker.poll_workflow_activation().await.unwrap(); + let run_id = activation.run_id.clone(); + worker + .seed_external_stream_waits(&run_id, vec![1], None, false) + .await; + assert_eq!( + worker.external_stream_run_status(&run_id).await, + ExternalStreamRunStatus::NoOpenWorkflowTask + ); + + worker.initiate_shutdown(); + worker + .complete_workflow_activation(WorkflowActivationCompletion::from_cmds( + run_id, + vec![start_timer_cmd(1, Duration::from_secs(10))], + )) + .await + .unwrap(); + + let written = markers.lock().clone(); + worker.shutdown().await; + worker.finalize_shutdown().await; + written +} + +#[tokio::test] +async fn every_completion_path_writes_exactly_one_marker_ending_in_a_terminal() { + // The cross-path gate. Each row is driven end to end and only its *marker* is looked at, so + // what is under test is the property the table as a whole claims rather than any one path's + // mechanics. + let table: Vec<(&str, Option, Vec)> = vec![ + ( + "normal completion", + Some(ParkReason::TaskCompleted), + markers_from_a_normal_completion().await, + ), + ( + "command-producing completion", + Some(ParkReason::CommandsProduced), + markers_from_a_command_producing_completion().await, + ), + ( + "terminal command", + Some(ParkReason::WorkflowCompleted), + markers_from_a_terminal_command().await, + ), + ( + "park confirmation", + Some(ParkReason::Idle), + markers_from_a_confirmed_park().await, + ), + ( + "all-fenced immediate park", + Some(ParkReason::AllWriteFenced), + markers_from_an_all_fenced_park().await, + ), + ( + "deadline rollover", + Some(ParkReason::Rollover), + markers_from_a_deadline_rollover().await, + ), + ( + "budget rollover", + Some(ParkReason::BudgetRollover), + markers_from_a_budget_rollover().await, + ), + ( + "shutdown with a Workflow Task open", + Some(ParkReason::Shutdown), + markers_from_a_shutdown_with_a_task_open().await, + ), + ("aborted park", None, markers_from_an_aborted_park().await), + ( + "shutdown with no open Workflow Task", + None, + markers_from_a_shutdown_with_no_open_task().await, + ), + ]; + + for (path, expected, written) in &table { + match expected { + Some(reason) => { + assert_eq!( + written.len(), + 1, + "{path} must write exactly one marker, got {written:?}" + ); + assert_eq!( + written[0].1, *reason, + "{path} recorded the wrong terminal boundary" + ); + assert!( + written[0].2.ends_with(TERMINAL_SUFFIX), + "{path} wrote a marker whose annotation stops short of its terminal: {:?}", + written[0].2 + ); + } + None => assert!( + written.is_empty(), + "{path} reached no boundary and must write no marker, got {written:?}" + ), + } + } + + // Every marker-writing row records a *different* boundary. Without this the table would still + // pass if two paths collapsed into one -- and the reason is the only thing in the marker that + // says which path closed it. + let reasons: Vec = table.iter().filter_map(|(_, e, _)| *e).collect(); + assert_eq!( + reasons.iter().collect::>().len(), + reasons.len(), + "two completion paths recorded the same terminal boundary: {reasons:?}" + ); + assert_eq!( + reasons.len(), + 8, + "the table has eight paths that write a marker and two that do not" + ); +} diff --git a/crates/sdk-core/src/core_tests/mod.rs b/crates/sdk-core/src/core_tests/mod.rs index 6dd4479bb..0c3b4174f 100644 --- a/crates/sdk-core/src/core_tests/mod.rs +++ b/crates/sdk-core/src/core_tests/mod.rs @@ -1,5 +1,6 @@ mod activity_tasks; mod event_groups; +mod external_streams; mod queries; mod replay_flag; mod updates; diff --git a/crates/sdk-core/src/lib.rs b/crates/sdk-core/src/lib.rs index e7608a603..11ca4d498 100644 --- a/crates/sdk-core/src/lib.rs +++ b/crates/sdk-core/src/lib.rs @@ -43,14 +43,15 @@ pub use temporalio_common::protos::TaskToken; pub use url::Url; pub use worker::{ ActivitySlotKind, CompleteActivityError, CompleteNexusError, CompleteWfError, - FixedSizeSlotSupplier, LocalActivitySlotKind, NamespaceCapabilities, NexusSlotKind, PollError, - PollerBehavior, ResourceBasedSlotsOptions, ResourceBasedSlotsOptionsBuilder, - ResourceBasedTuner, ResourceBasedTunerConfig, ResourceController, ResourceSlotOptions, - SlotInfo, SlotInfoTrait, SlotKind, SlotKindType, SlotMarkUsedContext, SlotReleaseContext, - SlotReservationContext, SlotSupplier, SlotSupplierOptions, SlotSupplierPermit, TunerBuilder, - TunerHolder, TunerHolderOptions, TunerHolderOptionsBuilder, Worker, WorkerConfig, - WorkerConfigBuilder, WorkerTuner, WorkerValidationError, WorkerVersioningStrategy, - WorkflowErrorType, WorkflowSlotKind, + ExternalStreamReadyResult, ExternalStreamRunStatus, FixedSizeSlotSupplier, + LocalActivitySlotKind, NamespaceCapabilities, NexusSlotKind, PollError, PollerBehavior, + ResourceBasedSlotsOptions, ResourceBasedSlotsOptionsBuilder, ResourceBasedTuner, + ResourceBasedTunerConfig, ResourceController, ResourceSlotOptions, SlotInfo, SlotInfoTrait, + SlotKind, SlotKindType, SlotMarkUsedContext, SlotReleaseContext, SlotReservationContext, + SlotSupplier, SlotSupplierOptions, SlotSupplierPermit, TunerBuilder, TunerHolder, + TunerHolderOptions, TunerHolderOptionsBuilder, Worker, WorkerConfig, WorkerConfigBuilder, + WorkerTuner, WorkerValidationError, WorkerVersioningStrategy, WorkflowErrorType, + WorkflowSlotKind, }; use crate::{ diff --git a/crates/sdk-core/src/replay/history_builder.rs b/crates/sdk-core/src/replay/history_builder.rs index 418600403..3cf593483 100644 --- a/crates/sdk-core/src/replay/history_builder.rs +++ b/crates/sdk-core/src/replay/history_builder.rs @@ -6,14 +6,17 @@ use std::{ time::{Duration, SystemTime}, }; use temporalio_common::protos::{ - constants::{LOCAL_ACTIVITY_MARKER_NAME, PATCH_MARKER_NAME}, + constants::{EXTERNAL_STREAM_MARKER_NAME, LOCAL_ACTIVITY_MARKER_NAME, PATCH_MARKER_NAME}, coresdk::{ AsJsonPayloadExt, IntoPayloadsExt, common::{ NamespacedWorkflowExecution, build_has_change_marker_details, build_local_activity_marker_details, }, - external_data::LocalActivityMarkerData, + external_data::{ + ExternalStreamMarkerData, ExternalWaitMarker, LocalActivityMarkerData, ParkReason, + build_external_stream_marker_details, + }, workflow_commands::ScheduleActivity, }, temporal::api::{ @@ -332,6 +335,64 @@ impl TestHistoryBuilder { self.build_and_push_event(EventType::MarkerRecorded, attrs.into()); } + /// Add an External Workflow Stream marker. + /// + /// Needed by any test that runs a Workflow Task *after* one that emitted a marker: commands + /// are matched to history events in order, so a history missing the marker event hands the + /// marker machine whatever event happens to be next. + pub fn add_external_stream_marker( + &mut self, + quiescence_generation: u64, + terminal_boundary: ParkReason, + replay_annotation: &[u8], + ) { + self.add_external_stream_marker_covering( + quiescence_generation, + terminal_boundary, + replay_annotation, + &[], + ); + } + + /// Add an External Workflow Stream marker recording the waits it closed. + /// + /// The wait list is what replay hands back to lang alongside the annotation, so a test that + /// asserts a replayed set was resolved from history needs a marker that actually names one. + pub fn add_external_stream_marker_covering( + &mut self, + quiescence_generation: u64, + terminal_boundary: ParkReason, + replay_annotation: &[u8], + waits: &[(u32, u64)], + ) { + let data = ExternalStreamMarkerData { + schema_version: 1, + quiescence_generation, + waits: waits + .iter() + .map(|(wait_id, generation)| ExternalWaitMarker { + wait_id: *wait_id, + generation: *generation, + }) + .collect(), + replay_annotation: replay_annotation.to_vec(), + terminal_boundary: terminal_boundary as i32, + output: None, + }; + self.add_external_stream_marker_data(data); + } + + /// Add an External Workflow Stream marker with an exact test-provided envelope. + pub fn add_external_stream_marker_data(&mut self, data: ExternalStreamMarkerData) { + let attrs = MarkerRecordedEventAttributes { + marker_name: EXTERNAL_STREAM_MARKER_NAME.to_string(), + details: build_external_stream_marker_details(&data), + workflow_task_completed_event_id: self.previous_task_completed_id, + ..Default::default() + }; + self.build_and_push_event(EventType::MarkerRecorded, attrs.into()); + } + /// Add a local activity result marker with a successful payload. pub fn add_local_activity_result_marker( &mut self, diff --git a/crates/sdk-core/src/worker/mod.rs b/crates/sdk-core/src/worker/mod.rs index 88d20ec48..8d3627156 100644 --- a/crates/sdk-core/src/worker/mod.rs +++ b/crates/sdk-core/src/worker/mod.rs @@ -42,6 +42,7 @@ pub(crate) use wft_poller::WFTPollerShared; #[allow(unreachable_pub)] // re-exported in test_help::integ_helpers pub use workflow::LEGACY_QUERY_ID; +pub use workflow::{ExternalStreamReadyResult, ExternalStreamRunStatus}; use crate::{ ActivityHeartbeat, @@ -1094,6 +1095,7 @@ impl Worker { WorkflowBasics { worker_config: Arc::new(config.clone()), shutdown_token: shutdown_token.child_token(), + worker_shutdown_token: shutdown_token.clone(), metrics, server_capabilities: client.capabilities().unwrap_or_default(), namespace_capabilities: capabilities.clone(), @@ -1419,6 +1421,116 @@ impl Worker { } } + /// Tell the worker a record is buffered for one of a workflow's External Stream waits. + /// + /// Thread-safe and freely callable concurrently. Unlike the local-activity notification path + /// this is **acknowledged**, because the caller is a watcher whose next action depends + /// entirely on the answer -- see [`ExternalStreamReadyResult`]. + /// + /// "Buffered", not "available": a notification for a record the caller has not yet read into + /// its buffer would produce an activation whose drain must block, which is exactly the + /// deadlock hazard the out-of-thread buffer exists to remove. + pub async fn notify_external_stream_ready( + &self, + run_id: &str, + wait_id: u32, + wait_generation: u64, + ) -> ExternalStreamReadyResult { + let Some(workflows) = self.task_subsystems.workflows.as_ref() else { + dbg_panic!("external stream readiness on a worker with workflows disabled"); + return ExternalStreamReadyResult::RunNotFound; + }; + workflows + .notify_external_stream_ready(run_id, wait_id, wait_generation) + .await + } + + /// Ask what state a run's External Stream wait set is in, changing nothing. + /// + /// Answered on the same serialized lane as + /// [`Worker::notify_external_stream_ready`], so the answer is as authoritative as a readiness + /// acknowledgement. It is a separate call on purpose: readiness means "a record is buffered", + /// so using it as a probe would be a false claim that manufactures a spurious activation on + /// the way out of a shutting-down worker. + pub async fn external_stream_run_status(&self, run_id: &str) -> ExternalStreamRunStatus { + let Some(workflows) = self.task_subsystems.workflows.as_ref() else { + dbg_panic!("external stream status on a worker with workflows disabled"); + return ExternalStreamRunStatus::RunNotFound; + }; + workflows.external_stream_run_status(run_id).await + } + + /// Test-only: drive marker emission with no terminal boundary. + #[cfg(test)] + pub(crate) async fn emit_terminal_less_external_stream_marker(&self, run_id: &str) -> bool { + match self.task_subsystems.workflows.as_ref() { + Some(workflows) => { + workflows + .emit_terminal_less_external_stream_marker(run_id) + .await + } + None => false, + } + } + + /// Test-only: read a run's accumulated, unwritten replay annotation. + #[cfg(test)] + pub(crate) async fn external_stream_annotation(&self, run_id: &str) -> Vec { + match self.task_subsystems.workflows.as_ref() { + Some(workflows) => workflows.external_stream_annotation(run_id).await, + None => Vec::new(), + } + } + + /// Test-only: start a run's workflow task rollover deadline. + /// + /// Starting it for real belongs to the retention path (C6), which is what decides a task is + /// being held open in the first place. + #[cfg(test)] + pub(crate) async fn start_wft_rollover_timer(&self, run_id: &str, wft_timeout: Duration) { + if let Some(workflows) = self.task_subsystems.workflows.as_ref() { + workflows + .start_wft_rollover_timer(run_id, wft_timeout) + .await; + } + } + + /// Test-only reach into the idle-timeout input, which Core's own timer will fire (C6). + #[cfg(test)] + pub(crate) fn notify_external_stream_idle_timeout( + &self, + run_id: &str, + quiescence_generation: u64, + ) { + if let Some(workflows) = self.task_subsystems.workflows.as_ref() { + workflows.notify_external_stream_idle_timeout(run_id, quiescence_generation); + } + } + + /// Test-only reach into the output deadline's serialized local input. + #[cfg(test)] + pub(crate) fn notify_external_output_flush_deadline(&self, run_id: &str) { + if let Some(workflows) = self.task_subsystems.workflows.as_ref() { + workflows.notify_external_output_flush_deadline(run_id); + } + } + + /// Test scaffolding: put a cached run's external stream wait set into a given state. + #[cfg(test)] + pub(crate) async fn seed_external_stream_waits( + &self, + run_id: &str, + wait_ids: Vec, + parked_at: Option, + wft_open: bool, + ) { + if let Some(workflows) = self.task_subsystems.workflows.as_ref() { + workflows + .seed_external_stream_waits(run_id, wait_ids, parked_at, wft_open) + .await; + } + } + /// Tell the worker that an activity has finished executing. May (and should) be freely called /// concurrently. #[instrument(skip(self, completion), diff --git a/crates/sdk-core/src/worker/workflow/driven_workflow.rs b/crates/sdk-core/src/worker/workflow/driven_workflow.rs index c9a2b3ed0..c3e57a4d5 100644 --- a/crates/sdk-core/src/worker/workflow/driven_workflow.rs +++ b/crates/sdk-core/src/worker/workflow/driven_workflow.rs @@ -8,7 +8,9 @@ use std::{ sync::mpsc::{self, Receiver, Sender}, }; use temporalio_common::protos::{ - coresdk::workflow_activation::{WorkflowActivationJob, start_workflow_from_attribs}, + coresdk::workflow_activation::{ + WorkflowActivationJob, start_workflow_from_attribs, workflow_activation_job, + }, temporal::api::{ common::v1::{Payload, SearchAttributes}, history::v1::WorkflowExecutionStartedEventAttributes, @@ -50,6 +52,7 @@ impl DrivenWorkflow { debug!(run_id = %attribs.original_execution_run_id, "Driven WF start"); let started_info = WorkflowStartedInfo { workflow_task_timeout: attribs.workflow_task_timeout.try_into_or_none(), + first_execution_run_id: attribs.first_execution_run_id.clone(), memo: attribs.memo.clone(), search_attrs: attribs.search_attributes.clone(), retry_policy: attribs.retry_policy.clone(), @@ -75,6 +78,16 @@ impl DrivenWorkflow { self.outgoing_wf_activation_jobs.as_slice() } + /// Discards a park job which lost a local race before it was delivered to lang. + pub(super) fn discard_pending_external_stream_park(&mut self) { + self.outgoing_wf_activation_jobs.retain(|job| { + !matches!( + &job.variant, + workflow_activation_job::Variant::PrepareExternalStreamPark(_) + ) + }); + } + /// Drain all pending jobs, so that they may be sent to the driven workflow pub(super) fn drain_jobs(&mut self) -> Vec { self.outgoing_wf_activation_jobs diff --git a/crates/sdk-core/src/worker/workflow/external_streams.rs b/crates/sdk-core/src/worker/workflow/external_streams.rs new file mode 100644 index 000000000..96e176068 --- /dev/null +++ b/crates/sdk-core/src/worker/workflow/external_streams.rs @@ -0,0 +1,1007 @@ +//! External Workflow Stream wait tracking (C2). +//! +//! Plain types plus the transition logic that decides whether a readiness notification, a park +//! confirmation, or a timer expiry is acted on or discarded as stale. Nothing here starts a timer, +//! issues an activation, or touches a `ManagedRun` -- the state machine is a pure function so both +//! orderings of the park/readiness race can be tested without a running worker. +//! +//! Three generations exist and are deliberately named separately: +//! +//! | Name | Scope | Increments when | +//! |-------------------------|-----------------------------|------------------------------------------------| +//! | `wait_generation` | one subscription | that subscription re-enters the blocked state | +//! | `quiescence_generation` | one complete blocked snapshot | the Workflow becomes quiescent again | +//! | `park_generation` | one confirmed park of a set | a park set is confirmed | +//! +//! `park_generation` is not a fourth counter: it takes the value of the `quiescence_generation` +//! that was parked. + +use std::{collections::HashMap, time::Duration}; + +/// The reserved `park_generation` meaning "the sender knows of no confirmed park". +/// +/// Quiescence generations start at 1, so this cannot collide with a real one. +pub(crate) const UNPARKED_WAKE_GENERATION: u64 = 0; + +/// Where one subscription's wait sits in the retention/park lifecycle. +/// +/// Only `BlockedWftOpen`, `Ready`, and `Parking` retain the Workflow Task. A `Parked` wait is +/// still logically pending -- the subscription has not gone away -- but it no longer holds the +/// task open, which is the whole point of parking. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ExternalWaitStatus { + /// Blocked, with the Workflow Task retained on its behalf. + BlockedWftOpen, + /// Core has accepted readiness for this wait and owes lang an activation. + Ready, + /// A park handshake is in flight for the set this wait belongs to. + Parking, + /// The park was confirmed through the backend. No longer retains the task. + Parked, +} + +impl ExternalWaitStatus { + /// Whether a wait in this state holds the current Workflow Task open. + pub(crate) fn retains_wft(self) -> bool { + match self { + ExternalWaitStatus::BlockedWftOpen + | ExternalWaitStatus::Ready + | ExternalWaitStatus::Parking => true, + ExternalWaitStatus::Parked => false, + } + } +} + +/// One subscription's wait, as Core tracks it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ExternalWaitState { + pub(crate) wait_id: u32, + /// Increments each time this wait re-enters the blocked state. A readiness notification + /// naming an older value refers to a block that has already been resolved. + pub(crate) wait_generation: u64, + pub(crate) status: ExternalWaitStatus, + /// Normally set after a write fence. One fenced stream does not park the Workflow Task. + pub(crate) immediately_parkable: bool, +} + +impl ExternalWaitState { + pub(crate) fn new(wait_id: u32, wait_generation: u64, immediately_parkable: bool) -> Self { + Self { + wait_id, + wait_generation, + status: ExternalWaitStatus::BlockedWftOpen, + immediately_parkable, + } + } +} + +/// What a readiness notification did, and what the caller must do about it. +/// +/// The five variants are not cosmetic: `NoOpenWorkflowTask` is the healthy state between Workflow +/// Tasks after a command-producing completion or a rollover, and reporting it as a missing Run +/// would both corrupt the metric and tell the watcher to tear itself down while it is still +/// needed. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ReadinessOutcome { + /// Readiness was serialized while the Workflow Task was still open. The caller cancels the + /// idle timer, aborts any in-flight park, and issues or augments a resolve activation. + Accepted, + /// The wait exists but its `wait_generation` moved on. Re-probe; do not signal. + Stale, + /// A confirmed `park_generation` exists for this wait. The watcher sends the wake Signal. + Parked, + /// The Run is cached and its waits are registered, but no Workflow Task is open. + NoOpenWorkflowTask, +} + +/// The read-only answer to "what state is this Run's wait set in?". +/// +/// Deliberately *not* the readiness call: readiness means "a record is buffered", so probing with +/// it would assert something false and manufacture a spurious activation during shutdown. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum RunStreamStatus { + /// A Workflow Task is open and retained by this wait set. + WftOpen, + /// The complete set is parked; a confirmed park generation exists. + Parked, + /// Waits are registered but no Workflow Task is open. + NoOpenWorkflowTask, +} + +/// The acknowledgement [`crate::Worker::notify_external_stream_ready`] returns. +/// +/// Five values, not two, because lang's watcher does something different with each and an +/// operator should conclude something different from each: +/// +/// | Result | Watcher action | Metric | +/// |-----------------------|-----------------------------------------|--------------------------| +/// | `Accepted` | Nothing; Core will activate | local wakeup | +/// | `Stale` | Re-probe; do **not** signal | stale notification | +/// | `Parked` | Send the reserved wake Signal | signal wakeup, parked | +/// | `NoOpenWorkflowTask` | Send the wake Signal; **keep** the watcher | signal wakeup, unparked | +/// | `RunNotFound` | Send the wake Signal, then tear down | signal wakeup, evicted | +/// +/// `NoOpenWorkflowTask` and `RunNotFound` are separate on purpose. A Run cached between Workflow +/// Tasks is the healthy post-completion state; reporting it as a missing Run would both corrupt +/// the metric and tell the watcher to tear itself down while it is still needed. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ExternalStreamReadyResult { + /// Readiness was serialized while the Workflow Task was still open. + Accepted, + /// The wait exists but its `wait_generation` moved on. + Stale, + /// A confirmed park generation exists for this wait. + Parked, + /// The Run is cached and its waits are registered, but no Workflow Task is open. + NoOpenWorkflowTask, + /// The Run is absent from this Core worker's cache. + RunNotFound, +} + +impl From for ExternalStreamReadyResult { + fn from(outcome: ReadinessOutcome) -> Self { + match outcome { + ReadinessOutcome::Accepted => ExternalStreamReadyResult::Accepted, + ReadinessOutcome::Stale => ExternalStreamReadyResult::Stale, + ReadinessOutcome::Parked => ExternalStreamReadyResult::Parked, + ReadinessOutcome::NoOpenWorkflowTask => ExternalStreamReadyResult::NoOpenWorkflowTask, + } + } +} + +/// The answer [`crate::Worker::external_stream_run_status`] returns. +/// +/// Read-only: asking must leave the Run exactly as it was. This exists so lang's shutdown sweep +/// can ask what state a Run is in without claiming a record is buffered, which is what +/// `notify_external_stream_ready` means and which would manufacture a spurious activation on the +/// way out. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ExternalStreamRunStatus { + /// A Workflow Task is open, retained by the wait set. + WftOpen, + /// The complete wait set is parked. + Parked, + /// The Run is cached with active waits, but no Workflow Task is open. + NoOpenWorkflowTask, + /// The Run is absent from this Core worker's cache. + RunNotFound, +} + +impl From for ExternalStreamRunStatus { + fn from(status: RunStreamStatus) -> Self { + match status { + RunStreamStatus::WftOpen => ExternalStreamRunStatus::WftOpen, + RunStreamStatus::Parked => ExternalStreamRunStatus::Parked, + RunStreamStatus::NoOpenWorkflowTask => ExternalStreamRunStatus::NoOpenWorkflowTask, + } + } +} + +/// Why a set of waits is being parked. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ParkTrigger { + /// The global quiescence timer expired. + IdleTimeout, + /// Every wait in the quiescent snapshot is `immediately_parkable`. + AllWriteFenced, +} + +/// The result of asking the set to begin parking. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ParkStartOutcome { + /// Every wait moved to `Parking`; issue `PrepareExternalStreamPark`. + Started(ParkTrigger), + /// The generation named is not the current one, so nothing was done. + StaleGeneration, + /// At least one wait is `Ready`, so parking would race a readiness Core already accepted. + AlreadyReady, +} + +/// The result of applying lang's `ExternalStreamParkResult`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum ParkResolution { + /// The park stands. The caller writes the marker and completes the Workflow Task. + Confirmed, + /// A recheck found records. The caller issues a normal resolve activation instead of running + /// user code from inside the park path. + Aborted, + /// The confirmation named a generation that is no longer parking -- readiness beat it, or a + /// later quiescent snapshot replaced it. Discarded with no effect. + StaleGeneration, +} + +/// Whether the accumulated annotation may be cleared, and the invariant that answers it. +#[derive(Debug, thiserror::Error, PartialEq, Eq)] +#[error( + "external stream annotation of {0} byte(s) is non-empty with no Workflow Task open; an \ + accumulated, unwritten annotation exists only while a Workflow Task is open" +)] +pub(crate) struct UnwrittenAnnotationError(usize); + +/// Core's view of one Run's complete external stream wait set. +/// +/// Core is annotation-blind: `replay_annotation` is an opaque byte buffer it appends to and hands +/// back, and no method here parses any of it. +#[derive(Debug, Default)] +pub(crate) struct ExternalWaitSet { + /// Identifies the current complete blocked snapshot. Starts at 0 meaning "never quiescent"; + /// the first snapshot is 1, which is why `UNPARKED_WAKE_GENERATION` can be 0. + quiescence_generation: u64, + waits: HashMap, + /// Waits known ready but not yet shipped, each with the `wait_generation` its readiness was + /// accepted at. Coalescing lives here so several notifications arriving while an activation + /// is outstanding become one next time. + /// + /// The generation is kept because a new quiescent snapshot can land *after* readiness was + /// accepted: if the snapshot reports the same generation, lang has not re-blocked and has not + /// seen the record, so the readiness must survive. See `become_quiescent`. + ready: HashMap, + idle_timeout: Option, + /// Set when a park is confirmed; cleared when the set becomes quiescent again. + park_generation: Option, + /// True while a Workflow Task is open for this Run. + wft_open: bool, + /// The accumulated, unwritten replay annotation for the current Workflow Task. + replay_annotation: Vec, +} + +impl ExternalWaitSet { + #[cfg(test)] + pub(crate) fn new() -> Self { + Self::default() + } + + pub(crate) fn is_empty(&self) -> bool { + self.waits.is_empty() + } + + pub(crate) fn quiescence_generation(&self) -> u64 { + self.quiescence_generation + } + + /// The confirmed park generation, if the set is parked. + /// + /// Only the unit tests below read this directly; the run's own paths ask + /// [`Self::accepts_wake_generation`] or [`Self::run_status`] instead, both of which answer a + /// question about the set rather than handing out the number. + #[cfg(test)] + pub(crate) fn park_generation(&self) -> Option { + self.park_generation + } + + pub(crate) fn idle_timeout(&self) -> Option { + self.idle_timeout + } + + pub(crate) fn replay_annotation(&self) -> &[u8] { + &self.replay_annotation + } + + pub(crate) fn wait(&self, wait_id: u32) -> Option<&ExternalWaitState> { + self.waits.get(&wait_id) + } + + /// The wait ids Core has accepted readiness for and not yet shipped. + pub(crate) fn take_ready_wait_ids(&mut self) -> Vec { + let mut ids: Vec<_> = self.ready.drain().map(|(id, _)| id).collect(); + ids.sort_unstable(); + ids + } + + pub(crate) fn has_pending_readiness(&self) -> bool { + !self.ready.is_empty() + } + + /// Whether this set holds the current Workflow Task open. + /// + /// A set with no waits retains nothing; a set every member of which is `Parked` retains + /// nothing either. Core must not complete the task merely because *one* member went idle. + pub(crate) fn retains_wft(&self) -> bool { + self.waits.values().any(|w| w.status.retains_wft()) + } + + /// Whether the whole set is parkable without waiting out the idle timeout. + /// + /// A fence on one stream alone does not qualify: the Workflow Task parks early only when + /// *every* active wait is immediately parkable. + pub(crate) fn all_immediately_parkable(&self) -> bool { + !self.waits.is_empty() && self.waits.values().all(|w| w.immediately_parkable) + } + + pub(crate) fn set_wft_open(&mut self, open: bool) { + self.wft_open = open; + } + + /// Records a complete quiescent snapshot, starting a new `quiescence_generation`. + /// + /// The waits given are the complete set the Workflow is blocked on. Any wait not listed is + /// dropped: the subscription was cancelled or the Workflow is no longer blocked on it. + pub(crate) fn become_quiescent( + &mut self, + waits: impl IntoIterator, + idle_timeout: Duration, + ) -> u64 { + self.quiescence_generation += 1; + self.waits = waits.into_iter().map(|w| (w.wait_id, w)).collect(); + + // Readiness accepted before this snapshot was processed is kept only where the snapshot + // reports the *same* `wait_generation`. Same generation means lang has not re-blocked, so + // it never saw the record and dropping the readiness would strand it until the idle timer + // or a producer's Signal. A bumped generation means lang re-blocked after draining, so + // the readiness refers to a block that is already resolved. + let waits = &self.waits; + self.ready.retain(|wait_id, generation| { + waits + .get(wait_id) + .is_some_and(|w| w.wait_generation == *generation) + }); + for wait_id in self.ready.keys() { + if let Some(wait) = self.waits.get_mut(wait_id) { + wait.status = ExternalWaitStatus::Ready; + } + } + + self.idle_timeout = Some(idle_timeout); + // A new quiescent snapshot supersedes any earlier park: the Workflow ran again, so + // whatever generation a producer may still be holding is no longer the live one. + self.park_generation = None; + self.wft_open = true; + self.quiescence_generation + } + + /// Applies a readiness notification, resolving the park/readiness race. + /// + /// This is the pure function both orderings of that race resolve to. Readiness accepted + /// *before* a park is confirmed wins, and the confirmation for that generation is then stale; + /// a confirmation that lands first wins, and readiness after it reports `Parked` so the + /// watcher sends a wake Signal instead of assuming Core will activate. + pub(crate) fn notify_ready(&mut self, wait_id: u32, wait_generation: u64) -> ReadinessOutcome { + let Some(wait) = self.waits.get_mut(&wait_id) else { + // The wait is not registered on this Run at all. That is not "stale" -- there is + // nothing here whose generation could have moved on. + return ReadinessOutcome::NoOpenWorkflowTask; + }; + + if wait_generation != wait.wait_generation { + return ReadinessOutcome::Stale; + } + + match wait.status { + // A confirmed park is the one state local readiness cannot be delivered into: the + // Workflow Task is gone and a producer's wake Signal is the only way back. + ExternalWaitStatus::Parked => ReadinessOutcome::Parked, + ExternalWaitStatus::BlockedWftOpen + | ExternalWaitStatus::Ready + | ExternalWaitStatus::Parking => { + if !self.wft_open { + return ReadinessOutcome::NoOpenWorkflowTask; + } + // Readiness during `Parking` aborts that parking generation -- see + // `resolve_park`, which will report the later confirmation as stale. + wait.status = ExternalWaitStatus::Ready; + self.ready.insert(wait_id, wait_generation); + ReadinessOutcome::Accepted + } + } + } + + /// The read-only status probe, which must leave the set untouched. + pub(crate) fn run_status(&self) -> RunStreamStatus { + if self.park_generation.is_some() + && self + .waits + .values() + .all(|w| w.status == ExternalWaitStatus::Parked) + { + RunStreamStatus::Parked + } else if self.wft_open && self.retains_wft() { + RunStreamStatus::WftOpen + } else { + RunStreamStatus::NoOpenWorkflowTask + } + } + + /// Moves the complete set into `Parking` so `PrepareExternalStreamPark` can be issued. + pub(crate) fn start_parking( + &mut self, + quiescence_generation: u64, + trigger: ParkTrigger, + ) -> ParkStartOutcome { + if quiescence_generation != self.quiescence_generation || self.waits.is_empty() { + return ParkStartOutcome::StaleGeneration; + } + if self + .waits + .values() + .any(|w| w.status == ExternalWaitStatus::Ready) + { + return ParkStartOutcome::AlreadyReady; + } + for wait in self.waits.values_mut() { + wait.status = ExternalWaitStatus::Parking; + } + ParkStartOutcome::Started(trigger) + } + + /// Applies lang's answer to `PrepareExternalStreamPark`. + pub(crate) fn resolve_park( + &mut self, + quiescence_generation: u64, + confirmed: bool, + ) -> ParkResolution { + if quiescence_generation != self.quiescence_generation { + return ParkResolution::StaleGeneration; + } + // Parking is all-or-nothing across the complete set, so readiness accepted while the + // handshake was in flight makes this confirmation stale even though it names the current + // generation -- and even if it moved only *one* wait out of `Parking`. Checking `any` + // here instead of `all` would confirm a park for a set with a ready wait in it, losing + // that wait's record until a producer happened to signal. + if !self + .waits + .values() + .all(|w| w.status == ExternalWaitStatus::Parking) + { + return ParkResolution::StaleGeneration; + } + + if confirmed { + for wait in self.waits.values_mut() { + wait.status = ExternalWaitStatus::Parked; + } + // A park generation *is* the quiescence generation that was parked, not a fourth + // counter, which is what lets a wake Signal carry one number that Core can match. + self.park_generation = Some(quiescence_generation); + // `wft_open` deliberately stays as it was. The Workflow Task is still open at this + // instant -- Core is about to write the park's marker on it and only then complete it + // -- and clearing it here would make `take_annotation` refuse the very annotation the + // confirmation just supplied the terminal for. The task is closed by + // `mark_wft_complete`, which is where every other completion path closes it too. + ParkResolution::Confirmed + } else { + self.abort_parking(); + ParkResolution::Aborted + } + } + + /// Returns every wait in `Parking` to `BlockedWftOpen`, bumping each `wait_generation`. + /// + /// Bumping is what makes an in-flight readiness notification for the aborted attempt stale + /// rather than resolving against the new block. + fn abort_parking(&mut self) { + for wait in self.waits.values_mut() { + if wait.status == ExternalWaitStatus::Parking { + wait.status = ExternalWaitStatus::BlockedWftOpen; + wait.wait_generation += 1; + } + } + } + + /// Invalidates the current park because another local boundary won its race. + pub(crate) fn abort_park_for_boundary(&mut self) { + self.abort_parking(); + self.mark_all_ready_after_aborted_park(); + } + + /// Marks every wait ready because a wake Signal arrived. + /// + /// All of them, not just the one the Signal names: the Signal's stream is a hint, and lang + /// rechecks every active subscription on wakeup regardless. A parked set is also released + /// here -- the whole point of the wake is that the park is over. + pub(crate) fn mark_all_ready_for_wake(&mut self) { + self.park_generation = None; + self.wft_open = true; + self.mark_all_ready(); + } + + /// Marks every wait ready because a park handshake's final recheck found records. + /// + /// All of them, for the same reason parking is all-or-nothing: the recheck covered every + /// stream, so any of them may now have data, and lang probes them all on the resolve anyway. + /// Unlike a wake, this leaves `park_generation` and `wft_open` alone -- no park was ever + /// confirmed and the Workflow Task never stopped being open. + pub(crate) fn mark_all_ready_after_aborted_park(&mut self) { + self.mark_all_ready(); + } + + fn mark_all_ready(&mut self) { + for wait in self.waits.values_mut() { + wait.status = ExternalWaitStatus::Ready; + self.ready.insert(wait.wait_id, wait.wait_generation); + } + } + + /// Validates a wake Signal's `park_generation` against this Run. + /// + /// `park_generation = 0` is the unparked wake and is always accepted as a recheck request: + /// the runtime rechecks every active subscription on wakeup regardless, so an unnecessary one + /// costs at most one empty Workflow Task. A *non-zero* generation the Run does not recognize + /// is a claim that turned out to be wrong, and is ignored as stale. + pub(crate) fn accepts_wake_generation(&self, park_generation: u64) -> bool { + if park_generation == UNPARKED_WAKE_GENERATION { + return true; + } + self.park_generation == Some(park_generation) + } + + /// Injects a confirmed park generation directly. + /// + /// Used by the wake-Signal path's tests before the live park handshake exists, and by + /// nothing else. + #[cfg(test)] + pub(crate) fn force_parked(&mut self, park_generation: u64) { + for wait in self.waits.values_mut() { + wait.status = ExternalWaitStatus::Parked; + } + self.park_generation = Some(park_generation); + self.wft_open = false; + } + + /// Appends an opaque observation delta. + pub(crate) fn accumulate_annotation(&mut self, delta: &[u8]) { + self.replay_annotation.extend_from_slice(delta); + } + + /// Takes the accumulated annotation, asserting the unwritten-annotation invariant. + /// + /// Deltas arrive only on activation completions, activations exist only under a Workflow + /// Task, and every completion path writes the accumulated annotation as exactly one marker + /// and clears it. So a non-empty annotation with no Workflow Task open is a bug in one of + /// those paths, not a state to recover from. + pub(crate) fn take_annotation(&mut self) -> Result, UnwrittenAnnotationError> { + if !self.wft_open && !self.replay_annotation.is_empty() { + return Err(UnwrittenAnnotationError(self.replay_annotation.len())); + } + Ok(std::mem::take(&mut self.replay_annotation)) + } + + /// The marker's wait list: every wait and the generation it was at. + pub(crate) fn marker_waits(&self) -> Vec<(u32, u64)> { + let mut out: Vec<_> = self + .waits + .values() + .map(|w| (w.wait_id, w.wait_generation)) + .collect(); + out.sort_unstable(); + out + } + + /// Every registered wait, for a runtime-internal job that covers the complete active set. + /// + /// Sorted by wait id so the job lang receives is reproducible: a set iterated in hash order + /// would hand lang a different list each run for the same state, which is exactly the kind of + /// difference replay cannot tolerate. + pub(crate) fn wait_snapshot(&self) -> Vec<(u32, u64, bool)> { + let mut out: Vec<_> = self + .waits + .values() + .map(|w| (w.wait_id, w.wait_generation, w.immediately_parkable)) + .collect(); + out.sort_unstable(); + out + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const IDLE: Duration = Duration::from_secs(1); + + fn quiescent_set(wait_ids: &[u32]) -> ExternalWaitSet { + let mut set = ExternalWaitSet::new(); + set.become_quiescent( + wait_ids + .iter() + .map(|id| ExternalWaitState::new(*id, 0, false)), + IDLE, + ); + set + } + + #[test] + fn quiescence_generations_start_at_one() { + // 0 is reserved as the unparked-wake sentinel, so a real generation must never be 0. + let set = quiescent_set(&[1]); + assert_eq!(set.quiescence_generation(), 1); + assert_ne!(set.quiescence_generation(), UNPARKED_WAKE_GENERATION); + } + + #[test] + fn readiness_at_the_current_generation_is_accepted() { + let mut set = quiescent_set(&[1, 2]); + + assert_eq!(set.notify_ready(1, 0), ReadinessOutcome::Accepted); + assert_eq!(set.wait(1).unwrap().status, ExternalWaitStatus::Ready); + assert_eq!(set.take_ready_wait_ids(), vec![1]); + } + + #[test] + fn a_stale_wait_generation_is_rejected() { + let mut set = quiescent_set(&[1]); + + assert_eq!(set.notify_ready(1, 7), ReadinessOutcome::Stale); + assert_eq!( + set.wait(1).unwrap().status, + ExternalWaitStatus::BlockedWftOpen + ); + assert!(!set.has_pending_readiness()); + } + + #[test] + fn readiness_for_an_unregistered_wait_is_not_reported_as_stale() { + // "Stale" tells the watcher to re-probe. There is nothing here to re-probe against, and + // reporting it as stale would spin the watcher forever. + let mut set = quiescent_set(&[1]); + assert_eq!( + set.notify_ready(99, 0), + ReadinessOutcome::NoOpenWorkflowTask + ); + } + + #[test] + fn readiness_with_no_open_workflow_task_reports_the_healthy_state() { + let mut set = quiescent_set(&[1]); + set.set_wft_open(false); + + assert_eq!(set.notify_ready(1, 0), ReadinessOutcome::NoOpenWorkflowTask); + // Not `Parked`: there is no confirmed park generation, so a producer has nothing to + // observe and the watcher must send an *unparked* wake. + assert_eq!(set.park_generation(), None); + } + + #[test] + fn readiness_coalesces_until_taken() { + let mut set = quiescent_set(&[1, 2, 3]); + + set.notify_ready(3, 0); + set.notify_ready(1, 0); + set.notify_ready(1, 0); + + assert_eq!(set.take_ready_wait_ids(), vec![1, 3]); + assert!(!set.has_pending_readiness()); + } + + #[test] + fn readiness_survives_a_snapshot_that_reports_the_same_generation() { + // Lang computed its snapshot from what it had drained, so a notification accepted + // afterwards refers to a record lang never saw. Same generation means lang has not + // re-blocked, so dropping the readiness would strand that record until the idle timer or a + // producer's Signal. + let mut set = quiescent_set(&[1, 2]); + set.notify_ready(1, 0); + + set.become_quiescent( + [ + ExternalWaitState::new(1, 0, false), + ExternalWaitState::new(2, 0, false), + ], + IDLE, + ); + + assert!(set.has_pending_readiness()); + assert_eq!(set.take_ready_wait_ids(), vec![1]); + } + + #[test] + fn readiness_is_dropped_when_the_snapshot_bumps_the_generation() { + // A bumped generation means lang re-blocked after draining, so it *did* see the record and + // the readiness refers to a block that is already resolved. + let mut set = quiescent_set(&[1]); + set.notify_ready(1, 0); + + set.become_quiescent([ExternalWaitState::new(1, 1, false)], IDLE); + + assert!(!set.has_pending_readiness()); + } + + #[test] + fn readiness_is_dropped_for_a_wait_the_snapshot_no_longer_lists() { + // The subscription was cancelled, or the Workflow is no longer blocked on it. + let mut set = quiescent_set(&[1, 2]); + set.notify_ready(2, 0); + + set.become_quiescent([ExternalWaitState::new(1, 0, false)], IDLE); + + assert!(!set.has_pending_readiness()); + } + + #[test] + fn a_surviving_ready_wait_is_restored_to_ready_not_blocked() { + // Otherwise the set would look fully blocked and the idle timer could park it while a + // record it has not delivered is sitting in the buffer. + let mut set = quiescent_set(&[1]); + set.notify_ready(1, 0); + + set.become_quiescent([ExternalWaitState::new(1, 0, false)], IDLE); + + assert_eq!(set.wait(1).unwrap().status, ExternalWaitStatus::Ready); + let generation = set.quiescence_generation(); + assert_eq!( + set.start_parking(generation, ParkTrigger::IdleTimeout), + ParkStartOutcome::AlreadyReady + ); + } + + // --- the park/readiness race, both orderings --------------------------- + + #[test] + fn readiness_before_confirmation_wins_and_the_confirmation_is_stale() { + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + + assert_eq!( + set.start_parking(quiesc, ParkTrigger::IdleTimeout), + ParkStartOutcome::Started(ParkTrigger::IdleTimeout) + ); + // Readiness lands while the handshake is in flight. + assert_eq!(set.notify_ready(1, 0), ReadinessOutcome::Accepted); + + // The confirmation names the current generation but no wait is `Parking` any more. + assert_eq!( + set.resolve_park(quiesc, true), + ParkResolution::StaleGeneration + ); + assert_eq!(set.park_generation(), None); + assert!(set.retains_wft()); + } + + #[test] + fn confirmation_before_readiness_wins_and_readiness_reports_parked() { + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + + set.start_parking(quiesc, ParkTrigger::IdleTimeout); + assert_eq!(set.resolve_park(quiesc, true), ParkResolution::Confirmed); + + assert_eq!(set.notify_ready(1, 0), ReadinessOutcome::Parked); + assert_eq!(set.park_generation(), Some(quiesc)); + assert!(!set.retains_wft()); + } + + #[test] + fn a_confirmed_park_leaves_its_own_marker_writable() { + // The confirmation is what supplies the terminal, and the marker carrying it is written + // on the still-open Workflow Task. If confirming closed the task here, the annotation the + // confirmation just completed would be refused by the unwritten-annotation invariant and + // the park could never write anything. + let mut set = quiescent_set(&[1]); + let quiesc = set.quiescence_generation(); + set.accumulate_annotation(b"observed"); + set.start_parking(quiesc, ParkTrigger::IdleTimeout); + + assert_eq!(set.resolve_park(quiesc, true), ParkResolution::Confirmed); + + set.accumulate_annotation(b"-terminal"); + assert_eq!( + set.take_annotation().unwrap(), + b"observed-terminal".to_vec() + ); + } + + #[test] + fn an_aborted_park_makes_every_wait_ready_at_its_new_generation() { + // The recheck that abandoned the park covered every stream, so every wait is re-probed. + // The ids must be published at the *bumped* generations, or the resolve activation would + // name blocks that the abort has already superseded. + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + set.start_parking(quiesc, ParkTrigger::AllWriteFenced); + assert_eq!(set.resolve_park(quiesc, false), ParkResolution::Aborted); + + set.mark_all_ready_after_aborted_park(); + + assert_eq!(set.take_ready_wait_ids(), vec![1, 2]); + assert_eq!(set.wait(1).unwrap().wait_generation, 1); + assert_eq!(set.wait(1).unwrap().status, ExternalWaitStatus::Ready); + // Nothing was parked, so no generation is left for a producer to claim. + assert_eq!(set.park_generation(), None); + } + + #[test] + fn a_boundary_winner_invalidates_the_park_and_late_confirmation() { + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + set.start_parking(quiesc, ParkTrigger::AllWriteFenced); + + set.abort_park_for_boundary(); + + assert_eq!(set.take_ready_wait_ids(), vec![1, 2]); + assert_eq!(set.wait(1).unwrap().wait_generation, 1); + assert_eq!(set.wait(2).unwrap().status, ExternalWaitStatus::Ready); + assert_eq!( + set.resolve_park(quiesc, true), + ParkResolution::StaleGeneration, + "a late result for the losing park must not create a second boundary" + ); + assert_eq!(set.park_generation(), None); + } + + #[test] + fn a_confirmation_for_an_old_generation_is_discarded() { + let mut set = quiescent_set(&[1]); + let first = set.quiescence_generation(); + set.start_parking(first, ParkTrigger::IdleTimeout); + + // The Workflow ran again and became quiescent at a new snapshot. + set.become_quiescent([ExternalWaitState::new(1, 1, false)], IDLE); + + assert_eq!( + set.resolve_park(first, true), + ParkResolution::StaleGeneration + ); + assert_eq!(set.park_generation(), None); + } + + #[test] + fn an_aborted_park_bumps_wait_generations() { + // Otherwise a readiness notification issued for the aborted attempt would resolve + // against the new block and deliver twice. + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + set.start_parking(quiesc, ParkTrigger::IdleTimeout); + + assert_eq!(set.resolve_park(quiesc, false), ParkResolution::Aborted); + + assert_eq!(set.wait(1).unwrap().wait_generation, 1); + assert_eq!(set.notify_ready(1, 0), ReadinessOutcome::Stale); + assert_eq!(set.notify_ready(1, 1), ReadinessOutcome::Accepted); + } + + #[test] + fn parking_a_set_with_a_ready_wait_is_refused() { + let mut set = quiescent_set(&[1, 2]); + let quiesc = set.quiescence_generation(); + set.notify_ready(2, 0); + + assert_eq!( + set.start_parking(quiesc, ParkTrigger::IdleTimeout), + ParkStartOutcome::AlreadyReady + ); + } + + #[test] + fn parking_a_stale_generation_is_refused() { + let mut set = quiescent_set(&[1]); + let quiesc = set.quiescence_generation(); + + assert_eq!( + set.start_parking(quiesc + 1, ParkTrigger::IdleTimeout), + ParkStartOutcome::StaleGeneration + ); + } + + // --- retention --------------------------------------------------------- + + #[test] + fn an_empty_set_retains_nothing() { + let set = ExternalWaitSet::new(); + assert!(!set.retains_wft()); + assert!(!set.all_immediately_parkable()); + } + + #[test] + fn one_parked_member_does_not_release_the_task() { + let mut set = quiescent_set(&[1, 2]); + set.waits.get_mut(&1).unwrap().status = ExternalWaitStatus::Parked; + + assert!(set.retains_wft()); + } + + #[test] + fn one_fenced_stream_does_not_make_the_set_parkable() { + let mut set = ExternalWaitSet::new(); + set.become_quiescent( + [ + ExternalWaitState::new(1, 0, true), + ExternalWaitState::new(2, 0, false), + ], + IDLE, + ); + + assert!(!set.all_immediately_parkable()); + + set.waits.get_mut(&2).unwrap().immediately_parkable = true; + assert!(set.all_immediately_parkable()); + } + + // --- the read-only status probe ---------------------------------------- + + #[test] + fn the_status_probe_leaves_the_set_untouched() { + let mut set = quiescent_set(&[1, 2]); + set.notify_ready(1, 0); + let before = set.marker_waits(); + let ready_before = set.has_pending_readiness(); + + assert_eq!(set.run_status(), RunStreamStatus::WftOpen); + assert_eq!(set.run_status(), RunStreamStatus::WftOpen); + + assert_eq!(set.marker_waits(), before); + assert_eq!(set.has_pending_readiness(), ready_before); + assert_eq!(set.wait(1).unwrap().status, ExternalWaitStatus::Ready); + } + + #[test] + fn status_distinguishes_parked_from_no_open_task() { + let mut set = quiescent_set(&[1]); + let quiesc = set.quiescence_generation(); + + set.set_wft_open(false); + assert_eq!(set.run_status(), RunStreamStatus::NoOpenWorkflowTask); + + set.set_wft_open(true); + set.start_parking(quiesc, ParkTrigger::IdleTimeout); + set.resolve_park(quiesc, true); + assert_eq!(set.run_status(), RunStreamStatus::Parked); + } + + // --- wake generations -------------------------------------------------- + + #[test] + fn an_unparked_wake_is_always_accepted() { + let set = quiescent_set(&[1]); + assert!(set.accepts_wake_generation(UNPARKED_WAKE_GENERATION)); + } + + #[test] + fn an_unrecognized_nonzero_wake_generation_is_stale() { + let mut set = quiescent_set(&[1]); + assert!(!set.accepts_wake_generation(42)); + + set.force_parked(7); + assert!(set.accepts_wake_generation(7)); + assert!(!set.accepts_wake_generation(6)); + // Still accepted -- an unparked wake costs at most one empty Workflow Task. + assert!(set.accepts_wake_generation(UNPARKED_WAKE_GENERATION)); + } + + // --- the annotation ---------------------------------------------------- + + #[test] + fn deltas_accumulate_in_order_and_are_taken_once() { + let mut set = quiescent_set(&[1]); + set.accumulate_annotation(b"one"); + set.accumulate_annotation(b"two"); + + assert_eq!(set.replay_annotation(), b"onetwo"); + assert_eq!(set.take_annotation().unwrap(), b"onetwo".to_vec()); + assert!(set.replay_annotation().is_empty()); + } + + #[test] + fn an_empty_annotation_may_be_taken_with_no_task_open() { + let mut set = quiescent_set(&[1]); + set.set_wft_open(false); + assert_eq!(set.take_annotation().unwrap(), Vec::::new()); + } + + #[test] + fn the_unwritten_annotation_invariant_fires() { + let mut set = quiescent_set(&[1]); + set.accumulate_annotation(b"delta"); + set.set_wft_open(false); + + let err = set.take_annotation().expect_err( + "a non-empty annotation with no Workflow Task open is a bug in a completion path", + ); + assert!(err.to_string().contains("no Workflow Task open")); + } + + #[test] + fn a_refused_take_leaves_the_annotation_where_it_was() { + // The invariant is a guard, not a discard. If it fired and cleared anyway, the very data + // the check exists to protect would be gone by the time anyone read the error -- and the + // Workflow Task retry that follows would replay from a cursor that never got committed. + let mut set = quiescent_set(&[1]); + set.accumulate_annotation(b"delta"); + set.set_wft_open(false); + + assert!(set.take_annotation().is_err()); + assert_eq!(set.replay_annotation(), b"delta"); + + // And once a task is open again the same annotation is takeable, unchanged. + set.set_wft_open(true); + assert_eq!(set.take_annotation().unwrap(), b"delta".to_vec()); + } +} diff --git a/crates/sdk-core/src/worker/workflow/machines/external_stream_state_machine.rs b/crates/sdk-core/src/worker/workflow/machines/external_stream_state_machine.rs new file mode 100644 index 000000000..fb686dfab --- /dev/null +++ b/crates/sdk-core/src/worker/workflow/machines/external_stream_state_machine.rs @@ -0,0 +1,364 @@ +//! The External Workflow Stream marker machine (C9). +//! +//! Modeled on [`super::local_activity_state_machine`], and for the same reason: a marker written +//! live must be *matched* by the `MarkerRecorded` event on replay, and on replay the marker must +//! be found by lookahead before the thing that depends on it is resolved. +//! +//! **Exactly one machine per Workflow Task**, however many progress reports that task carried. +//! Accumulating the annotation is the wait set's job -- it is what receives the deltas -- and this +//! machine's job is only to turn one accumulated annotation into one `RecordMarker` command and +//! then to reconcile it against history. +//! +//! The two paths are created by different callers and never by the same one: the live marker comes +//! from a completion that reached a boundary, and the replay marker comes from C10's lookahead, +//! which is the only thing that can find a marker *before* the wait set it closes is resolved. + +use super::{ + EventInfo, HistEventData, NewMachineWithCommand, OnEventWrapper, StateMachine, + TransitionResult, WFMachinesAdapter, WFMachinesError, fsm, workflow_machines::MachineResponse, +}; +use crate::worker::workflow::nondeterminism; +use std::convert::TryFrom; +use temporalio_common::protos::{ + constants::EXTERNAL_STREAM_MARKER_NAME, + coresdk::external_data::{ + ExternalStreamMarkerData, build_external_stream_marker_details, + extract_external_stream_marker_data, + }, + temporal::api::{ + command::v1::{Command as ProtoCommand, RecordMarkerCommandAttributes, command}, + enums::v1::{CommandType, EventType}, + history::v1::{HistoryEvent, MarkerRecordedEventAttributes, history_event}, + }, +}; + +fsm! { + pub(super) name ExternalStreamMachine; + command ExternalStreamCommand; + error WFMachinesError; + shared_state SharedState; + + // Live path: the marker is created from the annotation this Workflow Task accumulated. + Created --(Emit, shared on_emit) --> MarkerCommandCreated; + MarkerCommandCreated --(CommandRecordMarker, on_command_record_marker) --> ResultNotified; + ResultNotified --(MarkerRecorded(ExternalStreamMarkerData), shared on_marker_recorded) + --> MarkerCommandRecorded; + + // Replay path: the marker is found by lookahead before the wait set it closes is resolved, + // and is settled when its `MarkerRecorded` event is reached. Driven by C10. + Replaying --(Emit, shared on_replay_emit) --> WaitingResolveFromMarkerLookAhead; + WaitingResolveFromMarkerLookAhead --(HandleKnownResult(ExternalStreamMarkerData), + shared on_handle_known_result) + --> ResolvedFromMarkerLookAheadWaitingMarkerEvent; + ResolvedFromMarkerLookAheadWaitingMarkerEvent + --(MarkerRecorded(ExternalStreamMarkerData), shared on_marker_recorded) + --> MarkerCommandRecorded; +} + +#[derive(Debug, Clone)] +pub(super) struct SharedState { + /// The envelope this machine will write, or did write. + /// + /// Held rather than rebuilt so the command and the later reconciliation are provably the same + /// data -- Core is annotation-blind, so it has no way to notice a discrepancy otherwise. + data: ExternalStreamMarkerData, +} + +#[derive(Debug, derive_more::Display)] +pub(super) enum ExternalStreamCommand { + /// Write the marker to History. + RecordMarker, + /// The marker already exists in History; nothing to write. + AlreadyRecorded, +} + +#[derive(Default, Clone)] +pub(super) struct Created {} + +#[derive(Default, Clone)] +pub(super) struct Replaying {} + +#[derive(Default, Clone)] +pub(super) struct MarkerCommandCreated {} + +#[derive(Default, Clone)] +pub(super) struct ResultNotified {} + +#[derive(Default, Clone)] +pub(super) struct WaitingResolveFromMarkerLookAhead {} + +#[derive(Default, Clone)] +pub(super) struct ResolvedFromMarkerLookAheadWaitingMarkerEvent {} + +#[derive(Default, Clone)] +pub(super) struct MarkerCommandRecorded {} + +impl Created { + pub(super) fn on_emit( + self, + _state: &mut SharedState, + ) -> ExternalStreamMachineTransition { + TransitionResult::commands(vec![ExternalStreamCommand::RecordMarker]) + } +} + +impl Replaying { + pub(super) fn on_replay_emit( + self, + _state: &mut SharedState, + ) -> ExternalStreamMachineTransition { + // Nothing is written on replay: the marker is already in History, and writing it again + // would produce a command mismatch against the very event it was derived from. + TransitionResult::commands(vec![ExternalStreamCommand::AlreadyRecorded]) + } +} + +impl MarkerCommandCreated { + pub(super) fn on_command_record_marker( + self, + ) -> ExternalStreamMachineTransition { + TransitionResult::default() + } +} + +impl WaitingResolveFromMarkerLookAhead { + pub(super) fn on_handle_known_result( + self, + state: &mut SharedState, + data: ExternalStreamMarkerData, + ) -> ExternalStreamMachineTransition { + state.data = data; + TransitionResult::default() + } +} + +impl ResultNotified { + pub(super) fn on_marker_recorded( + self, + state: &mut SharedState, + data: ExternalStreamMarkerData, + ) -> ExternalStreamMachineTransition { + verify_marker_matches(state, &data) + } +} + +impl ResolvedFromMarkerLookAheadWaitingMarkerEvent { + pub(super) fn on_marker_recorded( + self, + state: &mut SharedState, + data: ExternalStreamMarkerData, + ) -> ExternalStreamMachineTransition { + verify_marker_matches(state, &data) + } +} + +/// Rejects a marker that closes a different quiescent snapshot than this machine's. +/// +/// Core cannot check the annotation -- it is opaque by design -- so the quiescence generation is +/// the only thing it *can* check, and checking it is what turns a reordered marker into a +/// nondeterminism error rather than a silently different stream result. +fn verify_marker_matches( + state: &mut SharedState, + data: &ExternalStreamMarkerData, +) -> ExternalStreamMachineTransition { + if data.quiescence_generation != state.data.quiescence_generation { + return TransitionResult::Err(WFMachinesError::Nondeterminism(format!( + "External stream marker in history closes quiescence generation {} but the machine \ + expecting it closes {}", + data.quiescence_generation, state.data.quiescence_generation + ))); + } + if data.output != state.data.output { + return TransitionResult::Err(WFMachinesError::Nondeterminism( + "External stream marker in history carries a different external output manifest than \ + the machine expecting it" + .to_string(), + )); + } + TransitionResult::default() +} + +impl ExternalStreamMachine { + /// A machine that **writes** one Workflow Task's marker. + /// + /// The live path only. A machine created here issues the `RecordMarker` command and then + /// reconciles it against the `MarkerRecorded` event the server writes back. + pub(super) fn record_marker(data: ExternalStreamMarkerData) -> NewMachineWithCommand { + let mut machine = ExternalStreamMachine { + state: Some(ExternalStreamMachineState::Created(Created {})), + shared_state: SharedState { data: data.clone() }, + }; + OnEventWrapper::on_event_mut(&mut machine, ExternalStreamMachineEvents::Emit) + .expect("Emit is always valid from the initial state"); + NewMachineWithCommand { + command: marker_command(&data), + machine: machine.into(), + } + } + + /// A machine for a marker C10's lookahead found ahead of the wait set it closes. + /// + /// Nothing is written: the marker is already in History, and issuing a command for it would + /// produce a mismatch against the very event it was read from. The machine exists only so that + /// event, when history reaches it, has something to be matched by -- which is what turns a + /// marker Core did not expect into a nondeterminism error rather than a silent skip. + pub(super) fn resolved_from_marker_lookahead(data: ExternalStreamMarkerData) -> Self { + let mut machine = ExternalStreamMachine { + state: Some(ExternalStreamMachineState::Replaying(Replaying {})), + shared_state: SharedState { data: data.clone() }, + }; + OnEventWrapper::on_event_mut(&mut machine, ExternalStreamMachineEvents::Emit) + .expect("Emit is always valid from the initial state"); + OnEventWrapper::on_event_mut( + &mut machine, + ExternalStreamMachineEvents::HandleKnownResult(data), + ) + .expect("A looked-ahead marker always resolves the machine created for it"); + machine + } + + /// Whether the `MarkerRecorded` event for this machine bypasses normal command matching. + /// + /// It does exactly when the machine came from lookahead, because no command was ever issued + /// for it and there is therefore nothing in the outgoing queue to match. A machine that wrote + /// its own marker is in `ResultNotified` and must go through the queue like any other command, + /// or the command it issued would never be consumed. + pub(super) fn marker_should_get_special_handling(&self) -> Result { + match self.state() { + ExternalStreamMachineState::ResultNotified(_) => Ok(false), + ExternalStreamMachineState::ResolvedFromMarkerLookAheadWaitingMarkerEvent(_) => { + Ok(true) + } + _ => Err(WFMachinesError::Fatal(format!( + "Attempted to check for external stream marker handling in invalid state {}", + self.state() + ))), + } + } +} + +fn marker_command(data: &ExternalStreamMarkerData) -> command::Attributes { + command::Attributes::RecordMarkerCommandAttributes(RecordMarkerCommandAttributes { + marker_name: EXTERNAL_STREAM_MARKER_NAME.to_string(), + details: build_external_stream_marker_details(data), + header: None, + failure: None, + }) +} + +impl TryFrom for ExternalStreamMachineEvents { + type Error = (); + + fn try_from(c: CommandType) -> Result { + Ok(match c { + CommandType::RecordMarker => Self::CommandRecordMarker, + _ => return Err(()), + }) + } +} + +impl TryFrom for ExternalStreamMachineEvents { + type Error = WFMachinesError; + + fn try_from(e: HistEventData) -> Result { + let e = e.event; + if e.event_type() != EventType::MarkerRecorded { + return Err(nondeterminism!( + "External stream machine cannot handle this event: {e}" + )); + } + match extract_stream_marker(&e) { + Some(data) => Ok(ExternalStreamMachineEvents::MarkerRecorded(data)), + None => Err(nondeterminism!( + "Marker recorded event {e} is not an external stream marker" + )), + } + } +} + +/// The envelope inside a `MarkerRecorded` event, if it is one of ours. +pub(super) fn extract_stream_marker(e: &HistoryEvent) -> Option { + if e.event_type() != EventType::MarkerRecorded { + return None; + } + match &e.attributes { + Some(history_event::Attributes::MarkerRecordedEventAttributes( + MarkerRecordedEventAttributes { + marker_name, + details, + .. + }, + )) if marker_name == EXTERNAL_STREAM_MARKER_NAME => { + extract_external_stream_marker_data(details) + } + _ => None, + } +} + +impl WFMachinesAdapter for ExternalStreamMachine { + fn adapt_response( + &self, + my_command: Self::Command, + _event_info: Option, + ) -> Result, WFMachinesError> { + Ok(match my_command { + ExternalStreamCommand::RecordMarker => { + vec![MachineResponse::IssueNewCommand(ProtoCommand { + ..marker_command(&self.shared_state.data).into() + })] + } + ExternalStreamCommand::AlreadyRecorded => vec![], + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use temporalio_common::protos::coresdk::external_data::ExternalOutputStreamManifest; + + fn marker_with_stage_token(stage_token: &str) -> ExternalStreamMarkerData { + ExternalStreamMarkerData { + schema_version: 1, + output: Some(ExternalOutputStreamManifest { + schema_version: 1, + fingerprint_version: 1, + stage_token: stage_token.to_string(), + history_floor_event_id: 1, + run_id: "run-id".to_string(), + provider_id: "provider".to_string(), + provider_format_version: 1, + ..Default::default() + }), + ..Default::default() + } + } + + #[test] + fn a_different_output_manifest_is_nondeterministic() { + let expected = marker_with_stage_token("expected"); + let mut state = SharedState { + data: expected.clone(), + }; + let actual = marker_with_stage_token("different"); + + assert!(matches!( + verify_marker_matches(&mut state, &actual), + TransitionResult::Err(WFMachinesError::Nondeterminism(message)) + if message.contains("different external output manifest") + )); + } + + #[test] + fn the_same_output_manifest_matches() { + let expected = marker_with_stage_token("expected"); + let mut state = SharedState { + data: expected.clone(), + }; + + assert!(matches!( + verify_marker_matches(&mut state, &expected), + TransitionResult::Ok { .. } + )); + } +} diff --git a/crates/sdk-core/src/worker/workflow/machines/mod.rs b/crates/sdk-core/src/worker/workflow/machines/mod.rs index bc5e3fcb9..53d44aac0 100644 --- a/crates/sdk-core/src/worker/workflow/machines/mod.rs +++ b/crates/sdk-core/src/worker/workflow/machines/mod.rs @@ -9,6 +9,7 @@ mod cancel_workflow_state_machine; mod child_workflow_state_machine; mod complete_workflow_state_machine; mod continue_as_new_workflow_state_machine; +mod external_stream_state_machine; mod fail_workflow_state_machine; mod local_activity_state_machine; mod modify_workflow_properties_state_machine; @@ -36,6 +37,7 @@ use cancel_workflow_state_machine::CancelWorkflowMachine; use child_workflow_state_machine::ChildWorkflowMachine; use complete_workflow_state_machine::CompleteWorkflowMachine; use continue_as_new_workflow_state_machine::ContinueAsNewWorkflowMachine; +use external_stream_state_machine::ExternalStreamMachine; use fail_workflow_state_machine::FailWorkflowMachine; use local_activity_state_machine::LocalActivityMachine; use modify_workflow_properties_state_machine::ModifyWorkflowPropertiesMachine; @@ -67,6 +69,7 @@ use transition_coverage::add_coverage; #[allow(clippy::enum_variant_names, clippy::large_enum_variant)] enum Machines { ActivityMachine, + ExternalStreamMachine, CancelExternalMachine, CancelWorkflowMachine, ChildWorkflowMachine, diff --git a/crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs b/crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs index 8d4b8f54e..e96e3f380 100644 --- a/crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs +++ b/crates/sdk-core/src/worker/workflow/machines/workflow_machines.rs @@ -29,8 +29,10 @@ use crate::{ WFMachinesError, WorkflowStartedInfo, fatal, history_update::NextWFT, machines::{ - HistEventData, activity_state_machine::ActivityMachine, + HistEventData, + activity_state_machine::ActivityMachine, child_workflow_state_machine::ChildWorkflowMachine, + external_stream_state_machine::{ExternalStreamMachine, extract_stream_marker}, modify_workflow_properties_state_machine::modify_workflow_properties, nexus_operation_state_machine::NexusOperationMachine, update_state_machine::UpdateMachine, @@ -58,10 +60,13 @@ use temporalio_common::{ VERSION_SEARCH_ATTR_KEY, coresdk::{ common::{NamespacedWorkflowExecution, VersioningIntent}, + external_data::{ExternalStreamMarkerData, ParkReason}, + external_stream, workflow_activation::{ - self, NotifyHasPatch, UpdateRandomSeed, WorkflowActivation, workflow_activation_job, + self, NotifyHasPatch, ReplayExternalStreams, UpdateRandomSeed, WorkflowActivation, + workflow_activation_job, }, - workflow_commands::ContinueAsNewWorkflowExecution, + workflow_commands::{ContinueAsNewWorkflowExecution, ExternalStreamWait}, }, temporal::api::{ command::v1::{Command as ProtoCommand, command::Attributes as ProtoCmdAttrs}, @@ -88,8 +93,23 @@ pub(crate) struct WorkflowMachines { last_history_from_server: HistoryUpdate, /// Protocol messages that have yet to be processed for the current WFT. protocol_msgs: Vec, + /// Reserved external stream wake Signals seen in history, decoded and suppressed from user + /// dispatch, waiting to be classified against the run's wait set. + pending_external_stream_wakes: Vec, + /// External stream marker machines whose `MarkerRecorded` event has not been reached yet, in + /// the order the markers appear in History. + /// + /// Matched positionally rather than by id: an external stream marker carries no lang-issued + /// sequence number, and its `quiescence_generation` is not unique either -- a Workflow Task + /// that never became quiescent writes generation 0, and two of those in a row would collide. + /// Markers are written one per Workflow Task and read in History order, so first-in-first-out + /// is exactly the pairing. + external_stream_marker_machines: VecDeque, /// EventId of the last handled WorkflowTaskStarted event current_started_event_id: i64, + /// The exact predecessor of the current WorkflowTaskScheduled event in the ordered History + /// view. Output staging must fail rather than guess when this is unavailable. + current_wft_history_floor_event_id: Option, /// The event id of the next workflow task started event that the machines need to process. /// Eventually, this number should reach the started id in the latest history update, but /// we must incrementally apply the history while communicating with lang. @@ -280,6 +300,7 @@ impl WorkflowMachines { metrics: basics.metrics, // In an ideal world one could say ..Default::default() here and it'd still work. current_started_event_id: 0, + current_wft_history_floor_event_id: None, next_started_event_id: 0, last_processed_event: 0, workflow_start_time: None, @@ -287,6 +308,8 @@ impl WorkflowMachines { wft_start_time: None, current_wf_time: None, observed_internal_flags: Rc::new(RefCell::new(observed_internal_flags)), + pending_external_stream_wakes: vec![], + external_stream_marker_machines: Default::default(), history_size_bytes: 0, continue_as_new_suggested: false, suggest_continue_as_new_reasons: Default::default(), @@ -399,6 +422,10 @@ impl WorkflowMachines { self.current_started_event_id } + pub(crate) fn current_wft_history_floor_event_id(&self) -> Option { + self.current_wft_history_floor_event_id + } + pub(crate) fn history_size_bytes(&self) -> u64 { self.history_size_bytes } @@ -472,9 +499,104 @@ impl WorkflowMachines { .to_owned(), suggest_continue_as_new_reasons: self.suggest_continue_as_new_reasons.clone(), target_worker_deployment_version_changed: self.target_worker_deployment_version_changed, + history_floor_event_id: self.current_wft_history_floor_event_id.unwrap_or_default(), } } + /// Emits **exactly one** marker for this Workflow Task's accumulated annotation. + /// + /// The refusal below belongs to emission itself and needs no finalization job to state it: an + /// annotation without its terminal is durable and wrong, while an abandoned Workflow Task + /// commits no cursor and loses no record. So there is no best-effort path -- if the terminal + /// is missing, nothing is written and the task fails for retry. + pub(crate) fn emit_external_stream_marker( + &mut self, + data: ExternalStreamMarkerData, + ) -> Result<()> { + if data.terminal_boundary() == ParkReason::Unspecified { + return Err(fatal!( + "Refusing to write an external stream marker with no terminal boundary for \ + quiescence generation {}", + data.quiescence_generation + )); + } + if self.replaying { + // The marker for this Workflow Task is already in History and C10's lookahead has + // already handed it back to lang. Issuing a command for it now would be matched + // against the very event it was read from and fail as a command mismatch. + debug!( + quiescence_generation = data.quiescence_generation, + "Not re-writing an external stream marker while replaying" + ); + return Ok(()); + } + let key = self.add_cmd_to_wf_task( + ExternalStreamMachine::record_marker(data), + Default::default(), + CommandIdKind::CoreInternal, + ); + self.external_stream_marker_machines.push_back(key); + // Core generates this command *after* lang's own iteration has already prepared its + // commands, so it has to prepare its own. Without this the marker sits in the current + // task's queue and never reaches the server -- silently, since nothing else looks there. + self.prepare_commands() + } + + /// Hands lang a marker the replay lookahead found, and creates the machine that settles it. + /// + /// Exactly one `ReplayExternalStreams` job per marker. Core is annotation-blind, so it copies + /// the annotation through untouched and supplies only what it can read itself: the quiescent + /// snapshot the marker closed, the waits it covered, and the terminal boundary. Everything the + /// live run needed the backend for -- offsets, integrity, delivery order -- is inside the + /// annotation and is lang's to interpret. + fn replay_external_stream_marker(&mut self, data: ExternalStreamMarkerData) { + let waits = data + .waits + .iter() + .map(|w| ExternalStreamWait { + wait_id: w.wait_id, + generation: w.generation, + // Never set on replay. `immediately_parkable` is a live observation about a write + // fence, and replay reproduces the boundary the marker recorded rather than + // re-deciding it -- the terminal boundary already says how the task ended. + immediately_parkable: false, + }) + .collect(); + self.drive_me.send_job( + workflow_activation_job::Variant::ReplayExternalStreams(ReplayExternalStreams { + quiescence_generation: data.quiescence_generation, + waits, + replay_annotation: data.replay_annotation.clone(), + terminal_boundary: data.terminal_boundary, + output: data.output.clone(), + }) + .into(), + ); + let machine = ExternalStreamMachine::resolved_from_marker_lookahead(data); + let key = self.all_machines.insert(machine.into()); + self.external_stream_marker_machines.push_back(key); + } + + /// Wake Signals decoded from history and not yet validated against the wait set. + /// + /// Validation needs the run's wait set, which lives on `ManagedRun`, so the two steps are + /// split: decode and suppress here, classify there. + pub(crate) fn take_external_stream_wakes(&mut self) -> Vec { + std::mem::take(&mut self.pending_external_stream_wakes) + } + + /// Queue a Core-generated job for lang. + /// + /// Used by the external stream paths, whose activations originate in Core rather than from a + /// history event, so they have no state machine to be driven by. + pub(crate) fn send_core_generated_job(&mut self, variant: workflow_activation_job::Variant) { + self.drive_me.send_job(variant.into()); + } + + pub(crate) fn discard_pending_external_stream_park(&mut self) { + self.drive_me.discard_pending_external_stream_park(); + } + pub(crate) fn has_pending_jobs(&self) -> bool { !self.drive_me.peek_pending_jobs().is_empty() } @@ -513,6 +635,33 @@ impl WorkflowMachines { pub(crate) fn reset_last_started_id(&mut self, id: i64) { debug!("Resetting back to event id {} due to speculative WFT", id); self.current_started_event_id = id; + self.current_wft_history_floor_event_id = None; + // A speculative rejection means the server discarded every command from that task. Most + // command machines are forbidden on a rejected Update task, but Core's external-output + // marker is generated after lang has answered and is therefore the exception. Leaving it + // queued makes the redelivery report both the rejected token and its fresh token. + let discarded_stream_markers: Vec<_> = self + .commands + .iter() + .chain(&self.current_wf_task_commands) + .filter_map(|command| { + matches!( + self.all_machines.get(command.machine), + Some(Machines::ExternalStreamMachine(_)) + ) + .then_some(command.machine) + }) + .collect(); + self.commands + .retain(|command| !discarded_stream_markers.contains(&command.machine)); + self.current_wf_task_commands + .retain(|command| !discarded_stream_markers.contains(&command.machine)); + self.external_stream_marker_machines + .retain(|key| !discarded_stream_markers.contains(key)); + for key in discarded_stream_markers { + self.machine_is_core_created.remove(key); + self.all_machines.remove(key); + } // We must reset the last event we "processed" to be after the last WFT we really completed // + any command events (since the SDK "processed" those when it emitted the commands). This // is also equal to what we just processed in the speculative task, minus two, since we @@ -660,6 +809,9 @@ impl WorkflowMachines { eid )); } + if event.event_type() == EventType::WorkflowTaskScheduled { + self.current_wft_history_floor_event_id = Some(self.last_processed_event); + } let next_event = history.peek(); // This definition of replaying here is that we are no longer replaying as soon as we @@ -749,6 +901,7 @@ impl WorkflowMachines { #[allow(clippy::large_enum_variant)] enum DelayedAction { LocalActivityMarker(Box), + ExternalStreamMarker(Box), ProtocolMessage(IncomingProtocolMessage), } let mut delayed_actions = vec![]; @@ -785,6 +938,12 @@ impl WorkflowMachines { } else { return Err(fatal!("Local activity marker was unparsable: {e:?}")); } + } else if let Some(stream_dat) = extract_stream_marker(e) { + // The marker for a Workflow Task is written by that task's *completion*, so it + // lands in the sequence after it. Finding it here is what lets lang be handed the + // recorded observations in the same activation that consumed them live -- and + // therefore before the wait set they close is resolved. + delayed_actions.push(DelayedAction::ExternalStreamMarker(Box::new(stream_dat))); } else if let Some( history_event::Attributes::WorkflowExecutionUpdateAcceptedEventAttributes(ref atts), ) = e.attributes @@ -818,6 +977,9 @@ impl WorkflowMachines { self.apply_local_activity_peeked_resolutions()?; } } + DelayedAction::ExternalStreamMarker(stream_dat) => { + self.replay_external_stream_marker(*stream_dat); + } DelayedAction::ProtocolMessage(pm) => { self.handle_protocol_message(pm)?; } @@ -949,6 +1111,39 @@ impl WorkflowMachines { } } + if extract_stream_marker(event).is_some() { + // Same shape as the local activity marker above, and for the same reason: a marker the + // lookahead resolved has no command in the outgoing queue to be matched against, + // because replay issues none. A marker no machine is expecting at all is the case + // local activities report as nondeterminism, and it is nondeterminism here too -- + // Core writes exactly one of these per Workflow Task, so an unexpected one means + // history and the machines disagree about what this run did. + let mkey = self + .external_stream_marker_machines + .front() + .copied() + .ok_or_else(|| { + nondeterminism!( + "Encountered an external stream marker with no state machine expecting \ + it: {event:?}" + ) + })?; + let handle_directly = match self.machine(mkey) { + Machines::ExternalStreamMachine(m) => m.marker_should_get_special_handling()?, + _ => { + return Err(fatal!( + "Encountered external stream marker but the associated machine was of \ + the wrong type! {event:?}" + )); + } + }; + self.external_stream_marker_machines.pop_front(); + if handle_directly { + self.submachine_handle_event(mkey, event_dat)?; + return Ok(EventHandlingOutcome::Normal); + } + } + let event_id = event.event_id; let consumed_cmd = loop { @@ -1029,9 +1224,23 @@ impl WorkflowMachines { attrs, )) = event_dat.event.attributes { - self.drive_me.send_job( - workflow_activation::SignalWorkflow::from((attrs, event_id)).into(), - ); + if attrs.signal_name == external_stream::WAKE_SIGNAL_NAME { + // Suppressed from user handlers **whether or not it decodes**, so an + // unknown envelope version or a stale generation can never surface in + // Workflow code as an unhandled Signal. Decoded without a DataConverter + // because Core is the component that has to read it, and a user codec + // that encrypts payloads would make it unreadable to exactly that + // component. + if let Some(wake) = decode_wake_signal(&attrs) { + self.pending_external_stream_wakes.push(wake); + } else { + debug!("Ignoring an unreadable external stream wake Signal"); + } + } else { + self.drive_me.send_job( + workflow_activation::SignalWorkflow::from((attrs, event_id)).into(), + ); + } } else { // err } @@ -1557,6 +1766,22 @@ impl WorkflowMachines { annotations ); } + // External stream commands are consumed above the machine level -- progress + // accumulates into the wait set (C14a), three are answers to + // runtime-internal activations that `ManagedRun` resolves (C6, C8, C15a). None of + // them should reach the machines; the output commit is also intercepted there. + // Reaching any one silently would drop replay-visible state on the floor. + leaked @ (WFCommandVariant::ExternalStreamProgress(_) + | WFCommandVariant::ExternalStreamQuiescent(_) + | WFCommandVariant::ExternalStreamParkResult(_) + | WFCommandVariant::ExternalStreamFinalized(_) + | WFCommandVariant::ExternalOutputStreamCommit(_) + | WFCommandVariant::ExternalOutputStreamBuffered(_)) => { + return Err(fatal!( + "External stream command {leaked} reached the state machines; it should \ + have been consumed by the run's external wait set" + )); + } WFCommandVariant::NoCommandsFromLang => (), } } @@ -1817,3 +2042,25 @@ enum CommandIdKind { /// A command which is fire-and-forget (ex: Upsert search attribs) NeverResolves, } + +/// Decodes a reserved wake Signal's envelope **without** a `DataConverter`. +/// +/// Returns `None` for anything unreadable -- wrong metadata, an unknown envelope version, or +/// bytes that are not a `WakeSignal`. All of those are ignored harmlessly rather than failing the +/// Workflow Task: an old Core must not break because a newer producer learned a new field, and +/// the Signal carries no user data whose loss would matter. +fn decode_wake_signal( + attrs: &temporalio_common::protos::temporal::api::history::v1::WorkflowExecutionSignaledEventAttributes, +) -> Option { + let payload = attrs.input.as_ref()?.payloads.first()?; + let message_type = payload.metadata.get("messageType").map(|v| v.as_slice()); + if message_type != Some(external_stream::WAKE_SIGNAL_MESSAGE_TYPE.as_bytes()) { + return None; + } + let wake = + ::decode(payload.data.as_slice()).ok()?; + if wake.envelope_version != external_stream::WAKE_SIGNAL_ENVELOPE_VERSION { + return None; + } + Some(wake) +} diff --git a/crates/sdk-core/src/worker/workflow/managed_run.rs b/crates/sdk-core/src/worker/workflow/managed_run.rs index e9bcb6272..fa7243fce 100644 --- a/crates/sdk-core/src/worker/workflow/managed_run.rs +++ b/crates/sdk-core/src/worker/workflow/managed_run.rs @@ -2,20 +2,25 @@ use crate::{ MetricsContext, WorkerConfig, abstractions::dbg_panic, internal_flags::CoreInternalFlags, - protosext::{WorkflowActivationExt, protocol_messages::IncomingProtocolMessage}, + protosext::WorkflowActivationExt, worker::{ LEGACY_QUERY_ID, LocalActRequest, WorkflowErrorType, workflow::{ ActivationAction, ActivationCompleteOutcome, ActivationCompleteResult, ActivationOrAuto, BufferedTasks, DrivenWorkflow, EvictionRequestResult, - FailedActivationWFTReport, HeartbeatTimeoutMsg, HistoryUpdate, + ExternalStreamIdleTimeoutMsg, FailedActivationWFTReport, HistoryUpdate, LocalActivityRequestSink, LocalResolution, NextPageReq, OutstandingActivation, - OutstandingTask, PermittedWFT, RequestEvictMsg, RunBasics, + OutstandingTask, PermittedWFT, RequestEvictMsg, RunBasics, RunTimerSink, ServerCommandsWithWorkflowInfo, TaskStorageMetrics, WFCommand, WFCommandVariant, WFMachinesError, WFT_HEARTBEAT_TIMEOUT_FRACTION, WFTReportStatus, WftFailureKind, WorkflowTaskInfo, + external_streams::{ + ExternalStreamReadyResult, ExternalStreamRunStatus, ExternalWaitSet, + ExternalWaitState, ParkResolution, ParkStartOutcome, ParkTrigger, ReadinessOutcome, + }, history_update::HistoryPaginator, machines::{MachinesWFTResponseContent, WorkflowMachines}, + workflow_stream::LocalInputs, }, }, }; @@ -32,11 +37,17 @@ use temporalio_common::protos::{ TaskToken, coresdk::{ common::ExternalStorageMetrics, + external_data::{ExternalStreamMarkerData, ExternalWaitMarker, ParkReason}, workflow_activation::{ + FinalizeExternalStreams, PrepareExternalStreamPark, ResolveExternalStreamWaits, WorkflowActivation, create_evict_activation, query_to_job, remove_from_cache::EvictionReason, workflow_activation_job, }, - workflow_commands::{FailWorkflowExecution, QueryResult}, + workflow_commands::{ + ExternalStreamFinalized, ExternalStreamParkResult, ExternalStreamWait, + FailWorkflowExecution, QueryResult, WorkflowOutputStreamCommit, WorkflowStreamProgress, + external_stream_park_result, + }, workflow_completion, }, temporal::api::{ @@ -72,8 +83,11 @@ pub(super) struct ManagedRun { /// /// This field is `None` when `WorkerTaskTypes.enable_local_activities` is false. local_activity_request_sink: Option>, - /// Set if the run is currently waiting on the execution of some local activities. - waiting_on_la: Option, + /// Schedules this run's timers without going through the local-activity sink, so a + /// workflow-only worker still gets its deadlines. + run_timers: RunTimerSink, + /// Local work that may retain the open workflow task -- today, outstanding local activities. + waiting_on_local_work: WaitingOnLocalWork, /// Is set to true if the machines encounter an error and the only subsequent thing we should /// do is be evicted. am_broken: bool, @@ -107,6 +121,7 @@ impl ManagedRun { basics: RunBasics, wft: PermittedWFT, local_activity_request_sink: Option>, + run_timers: RunTimerSink, ) -> (Self, RunUpdateAct) { let metrics = basics.metrics.clone(); let config = basics.worker_config.clone(); @@ -114,7 +129,8 @@ impl ManagedRun { let mut me = Self { wfm, local_activity_request_sink, - waiting_on_la: None, + run_timers, + waiting_on_local_work: Default::default(), am_broken: false, wft: None, activation: None, @@ -141,7 +157,7 @@ impl ManagedRun { } pub(super) fn waiting_on_local_activities(&self) -> bool { - self.waiting_on_la.is_some() + self.waiting_on_local_work.local_activities.is_some() } pub(super) fn have_seen_terminal_event(&self) -> bool { @@ -179,6 +195,14 @@ impl ManagedRun { ) -> Result, RunUpdateErr> { if self.wft.is_some() { dbg_panic!("Trying to send a new WFT for a run which already has one!"); + // `dbg_panic!` deliberately logs and continues in release builds. + // Continuing into the ordinary admission path would overwrite the + // outstanding task and lose ownership of its task token. Retaining + // the replacement here gives release builds the same safe ordering + // the caller's admission guard normally establishes, while debug + // builds still fail loudly at the violated invariant above. + self.task_buffer.buffer(pwft); + return Ok(None); } let start_time = Instant::now(); @@ -221,13 +245,32 @@ impl ManagedRun { } self.paginator = Some(pwft.paginator); + // A Workflow Task is open from here until it is reported. The unwritten-annotation + // invariant is stated against *that*, not against quiescence -- a task can accumulate a + // delta and complete without ever asking to be retained. + self.waiting_on_local_work + .external_wait_set + .set_wft_open(true); + // A finalization or park handshake still outstanding here belongs to a task that failed + // rather than answering. That boundary is gone with the task, and leaving the expectation + // set would make the next ordinary completion look like a lang protocol violation. + self.waiting_on_local_work.pending_finalization = None; + self.waiting_on_local_work.pending_park = None; + self.waiting_on_local_work.pending_output_commit = None; + self.clear_external_output_buffered(); + self.waiting_on_local_work.output_latency_pending = false; + self.waiting_on_local_work.park_rollback_resolve_pending = false; + // Same reasoning for a query answer held across that finalization: the task it was going + // to be reported on is gone, and the server re-delivers the query on the replacement if it + // is still outstanding. Reporting it on *this* task would answer a query nobody asked. + self.waiting_on_local_work.deferred_query_responses.clear(); self.wft = Some(OutstandingTask { info: wft_info, pending_queries, start_time, permit: pwft.permit, }); - if let Some(waiting) = self.waiting_on_la.as_mut() { + if let Some(waiting) = self.waiting_on_local_work.local_activities.as_mut() { waiting.hb_timeout_handle.abort(); waiting.heartbeat_timeout_pending = false; } @@ -243,36 +286,86 @@ impl ManagedRun { } // The update field is only populated in the event we hit the cache - let activation = if work.update.is_real() { + let update_was_real = work.update.is_real(); + if update_was_real { if is_incremental { self.metrics.sticky_cache_hit(); } - self.wfm.new_work_from_server(work.update, work.messages)? - } else { - let r = self.wfm.get_next_activation()?; - if r.jobs.is_empty() { - return Err(RunUpdateErr { - source: crate::worker::workflow::fatal!( - "Machines created for {} with no jobs", - self.wfm.machines.run_id - ), - complete_resp_chan: None, - }); - } - r - }; + self.wfm + .machines + .new_work_from_server(work.update, work.messages)?; + } + + // A wake Signal reaches Core as a history event, so it can only be classified once that + // history has been applied. The first valid one creates or accompanies this task. + // + // This has to happen *before* the activation is built, not after it. `get_wf_activation` + // derives `is_replaying` from the job list it drains, and an empty list satisfies the + // "every job is a query" test vacuously, so an activation built with no jobs is flagged + // replaying. The reserved wake Signal is suppressed from user handlers and therefore + // produces no job of its own, which is exactly that case: appending the resolve job after + // the build would hand lang a replacement task marked as replay, lang would report + // neither stream progress nor quiescence while replaying, the wait generation would never + // advance, and every later readiness report would be answered `Stale` while the watcher's + // cursor had already moved past those records -- a silent stall. Queueing the job first + // lets the flag be computed over a job list that reflects the work being sent. + if self.apply_external_stream_wakes() { + self.waiting_on_local_work + .external_wait_set + .set_wft_open(true); + self.maybe_issue_external_stream_resolve(); + } + + // Output capacity is backpressure inside the Workflow, rather than an external event the + // server can put on the replacement task. The completion that staged the full batch asked + // the server for this task specifically so lang could enter a new activation and release + // its capacity waiters. Carry that one-shot intent across the report; any real job already + // guarantees the activation, otherwise materialize the existing runtime resume job. An + // empty hint set is intentional and safe -- hints have never been exhaustive, and lang + // probes every active input wait while its ordinary activation turn also releases the + // output waiters. + let output_capacity_activation_pending = mem::take( + &mut self + .waiting_on_local_work + .output_capacity_activation_pending, + ); + if output_capacity_activation_pending && !self.wfm.machines.has_pending_jobs() { + self.wfm.machines.send_core_generated_job( + workflow_activation_job::Variant::ResolveExternalStreamWaits( + ResolveExternalStreamWaits { + quiescence_generation: self + .waiting_on_local_work + .external_wait_set + .quiescence_generation(), + ready_hints: vec![], + }, + ), + ); + } + + let activation = self.wfm.get_next_activation()?; + if !update_was_real && activation.jobs.is_empty() { + return Err(RunUpdateErr { + source: crate::worker::workflow::fatal!( + "Machines created for {} with no jobs", + self.wfm.machines.run_id + ), + complete_resp_chan: None, + }); + } if activation.jobs.is_empty() { if self.wfm.machines.outstanding_local_activity_count() > 0 { // If the activation has no jobs but there are outstanding LAs, we need to restart // the WFT heartbeat. - if let Some(ref mut lawait) = self.waiting_on_la { + if let Some(ref mut lawait) = self.waiting_on_local_work.local_activities { lawait.hb_timeout_handle.abort(); - lawait.hb_timeout_handle = sink_heartbeat_timeout_start( - self.wfm.machines.run_id.clone(), - self.local_activity_request_sink.as_deref(), + let wft_timeout = lawait.wft_timeout; + lawait.hb_timeout_handle = Self::start_la_heartbeat_timeout_with( + &self.run_timers, + &self.wfm.machines.run_id, start_time, - lawait.wft_timeout, + wft_timeout, ); // No activation needs to be sent to lang. We just need to wait for another // heartbeat timeout or LAs to resolve @@ -283,11 +376,22 @@ impl ManagedRun { was no waiting on LA info." ) } - } else { - return Ok(Some(ActivationOrAuto::Autocomplete { - run_id: self.wfm.machines.run_id.clone(), - })); } + if self.waiting_on_local_work.external_wait_set.retains_wft() { + // The replacement task after a rollover. Lang has not been activated, so it + // cannot re-request retention -- Core carries it across instead, along with every + // subscription, cursor, and readiness generation the wait set already holds. + // Autocompleting here would report the replacement task straight back and undo + // the rollover it was created for. + self.waiting_on_local_work + .external_wait_set + .set_wft_open(true); + self.restart_external_stream_deadlines(start_time); + return self._check_more_activations(); + } + return Ok(Some(ActivationOrAuto::Autocomplete { + run_id: self.wfm.machines.run_id.clone(), + })); } Ok(Some(ActivationOrAuto::LangActivation(activation))) @@ -301,6 +405,12 @@ impl ManagedRun { task_storage_metrics: &TaskStorageMetrics, ) -> Option { debug!("Marking WFT completed"); + // No task is open again until a replacement arrives. The wait set itself survives -- the + // subscriptions are still registered and their cursors still hold -- but readiness can no + // longer be delivered locally, which is what `NoOpenWorkflowTask` tells a watcher. + self.waiting_on_local_work + .external_wait_set + .set_wft_open(false); let retme = self.wft.take(); if let Some(ot) = &retme @@ -326,6 +436,10 @@ impl ManagedRun { { if let Some(id) = reset_last_started_to { self.wfm.machines.reset_last_started_id(id); + // The server discarded the speculative task and its capacity marker, so its + // replacement is a redelivery, not the fresh task that may release lang's waiter. + self.waiting_on_local_work + .output_capacity_activation_pending = false; } // Tell the LA manager that we're done with the WFT if let Some(ref local_act_request_sink) = self.local_activity_request_sink { @@ -359,25 +473,60 @@ impl ManagedRun { } if self.wft.is_none() { - // It doesn't make sense to do workflow work unless we have a WFT + // It doesn't make sense to do workflow work unless we have a WFT. + // + // This is also the whole of C15b's second transition, and it is a no-op by + // construction: with no Workflow Task there is nothing accumulated, so no marker is + // written and none is missing, and there is no task token to set `force_new_wft` on + // either. The server-visible replacement is lang's wake sweep, which Core must not + // duplicate (ADR-009). return Ok(None); } + // An eviction tears this Run down, so it closes the stream boundary for exactly the same + // reason Worker shutdown does. Recording the intent *here* rather than in + // `request_eviction` is what orders finalization ahead of eviction: the eviction + // activation is produced in this function's final branch, so a `FinalizeExternalStreams` + // job queued now is always issued -- and answered -- before `RemoveFromCache` is. + if self.trying_to_evict.is_some() { + self.begin_external_stream_teardown(); + } + + // Ready waits become a job here rather than at the notification, so readiness that + // arrived while an activation was outstanding is picked up the moment that activation + // completes -- with no separate path to keep in step. + self.maybe_issue_external_stream_resolve(); + + // The Run is going away while it still holds a Workflow Task. Anything accumulated needs + // lang's terminal before a marker may be written, which is what the finalization job asks + // for; `false` means nothing was accumulated, and the task must then be completed anyway + // rather than abandoned open, because completing is the only way to request the + // replacement task that offers this Run back to the task queue. + let teardown_needs_completion = self.waiting_on_local_work.shutdown_pending + && !self.am_broken + && !self.begin_external_stream_finalization(ParkReason::Shutdown); + if self.wfm.machines.has_pending_jobs() && !self.am_broken { Ok(Some(ActivationOrAuto::LangActivation( self.wfm.get_next_activation()?, ))) - } else if self.waiting_on_la.is_some() - && self.wfm.machines.outstanding_local_activity_count() == 0 + } else if self + .waiting_on_local_work + .finished(self.wfm.machines.outstanding_local_activity_count()) { - self.waiting_on_la + self.waiting_on_local_work + .local_activities .take() - .expect("waiting_on_la was just checked") + .expect("local work was just checked to be present") .hb_timeout_handle .abort(); Ok(Some(ActivationOrAuto::Autocomplete { run_id: self.run_id().to_string(), })) + } else if teardown_needs_completion { + Ok(Some(ActivationOrAuto::Autocomplete { + run_id: self.run_id().to_string(), + })) } else { if !self.am_broken { let has_pending_queries = self @@ -392,7 +541,8 @@ impl ManagedRun { } } if self - .waiting_on_la + .waiting_on_local_work + .local_activities .as_ref() .is_some_and(|waiting| waiting.heartbeat_timeout_pending) { @@ -606,6 +756,9 @@ impl ManagedRun { is_auto_fail: bool, resp_chan: Option>, ) -> RunUpdateAct { + self.clear_external_output_buffered(); + self.waiting_on_local_work.output_latency_pending = false; + self.waiting_on_local_work.park_rollback_resolve_pending = false; let (tt, attempt) = if let Some(t) = self.wft.as_ref() { (t.info.task_token.clone(), t.info.attempt) } else { @@ -744,13 +897,19 @@ impl ManagedRun { ) -> Result, RunUpdateErr> { let completing_heartbeat_autocomplete = matches!(self.activation, Some(OutstandingActivation::Autocomplete)) - && self.waiting_on_la.is_some(); + && self.waiting_on_local_work.local_activities.is_some(); let completing_la_heartbeat = completing_heartbeat_autocomplete || self - .waiting_on_la + .waiting_on_local_work + .local_activities .as_ref() .is_some_and(|waiting| waiting.heartbeat_timeout_pending); - let data = CompletionDataForWFT { + // A run-level rollover deadline forces a replacement task on its own, with no local + // activity involved -- which is the whole point of the deadline being the run's rather + // than the local-activity subsystem's. + let completing_deadline_rollover = + mem::take(&mut self.waiting_on_local_work.deadline_rollover_pending); + let mut data = CompletionDataForWFT { task_token: completion.task_token, query_responses: completion.query_responses, has_pending_query: completion.has_pending_query, @@ -771,10 +930,493 @@ impl ManagedRun { ))); } + // A query answer held across a `FinalizeExternalStreams` round trip rides back onto the + // completion that finally reports the task. The round trip keeps the Workflow Task open + // and runs no user Workflow code, so lang has no way to resend the answer; without this it + // would be answered to nobody and the server would wait its query out. + if !self + .waiting_on_local_work + .deferred_query_responses + .is_empty() + { + data.query_responses + .append(&mut self.waiting_on_local_work.deferred_query_responses); + } + + // External stream commands are consumed here rather than by the machines. Taking them + // first is also what makes "no server-bound command accompanies the completion" checkable: + // whatever is left after this *is* the server-bound set. + let mut lang_commands = completion.commands; + let mut stream_commands = match take_external_stream_commands(&mut lang_commands) { + Ok(taken) => taken, + Err(source) => { + return Err(RunUpdateErr { + source, + complete_resp_chan: completion.resp_chan, + }); + } + }; + let output_was_buffered = self.waiting_on_local_work.output_buffered; + if let Some(commit) = stream_commands.output_commit.take() { + if self.waiting_on_local_work.pending_output_commit.is_some() { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "Lang sent more than one WorkflowOutputStreamCommit for one Workflow Task" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + let Some(manifest) = commit.manifest.as_ref() else { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "WorkflowOutputStreamCommit carried no staged output manifest".to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + }; + let Some(history_floor_event_id) = self + .wfm + .machines + .current_wft_history_floor_event_id() + .filter(|id| *id > 0) + else { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "Refusing an external output commit because Core could not identify the \ + exact event preceding this Workflow Task's scheduled event" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + }; + if manifest.history_floor_event_id != history_floor_event_id { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal(format!( + "External output manifest history floor {} does not match this Workflow \ + Task's exact floor {history_floor_event_id}", + manifest.history_floor_event_id + )), + complete_resp_chan: completion.resp_chan, + }); + } + if manifest.run_id != self.wfm.machines.run_id { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal(format!( + "External output manifest names Run {} but this Workflow Task belongs to {}", + manifest.run_id, self.wfm.machines.run_id + )), + complete_resp_chan: completion.resp_chan, + }); + } + if manifest.stage_token.is_empty() { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "External output manifest carried an empty stage token".to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + if manifest.schema_version != 1 + || manifest.fingerprint_version != 1 + || manifest.provider_id.is_empty() + || manifest.provider_format_version == 0 + || manifest.topics.is_empty() + || manifest.segments.is_empty() + || <_ as prost::Message>::encoded_len(manifest) > 64 * 1024 + { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "External output manifest has an unsupported version, missing provider, \ + empty schedule, or exceeds the 64 KiB marker budget" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + let mut topic_names = HashSet::new(); + if manifest.topics.iter().any(|topic| { + topic.topic.is_empty() + || !topic_names.insert(topic.topic.as_str()) + || topic.record_count == 0 + || topic.logical_byte_count == 0 + || topic.logical_fingerprint.len() != 32 + }) || manifest + .segments + .iter() + .any(|segment| segment.record_counts_by_topic.len() != manifest.topics.len()) + || manifest + .topics + .iter() + .enumerate() + .any(|(topic_index, topic)| { + manifest + .segments + .iter() + .map(|segment| u64::from(segment.record_counts_by_topic[topic_index])) + .sum::() + != u64::from(topic.record_count) + }) + { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "External output manifest has duplicate or malformed topics, \ + fingerprints, or activation segment counts" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + self.waiting_on_local_work.pending_output_commit = Some(commit); + self.clear_external_output_buffered(); + } + if let Some(max_publish_latency) = stream_commands.output_buffered_latency.take() + && !self.wfm.machines.replaying + { + self.note_external_output_buffered(max_publish_latency); + } + let output_commit_pending = self.waiting_on_local_work.pending_output_commit.is_some(); + let completing_output_capacity = self + .waiting_on_local_work + .pending_output_commit + .as_ref() + .is_some_and(|commit| commit.request_rollover); + let has_server_bound_commands = !lang_commands.is_empty(); + + // Lang answered `PrepareExternalStreamPark`. Paired against the *issued* job for the same + // reason finalization is: a park job answered without a result would leave Core holding a + // boundary it decided with no terminal to write, and an unprompted result would let lang + // close an annotation Core never asked it to close. + let output_latency_preempts_park = self.waiting_on_local_work.output_latency_pending + && self.waiting_on_local_work.pending_park.is_some(); + let park_outcome = match ( + self.waiting_on_local_work.pending_park, + stream_commands.park_result, + ) { + (Some(PendingPark::Queued(_)), None) if output_latency_preempts_park => { + self.waiting_on_local_work.pending_park = None; + self.wfm.machines.discard_pending_external_stream_park(); + self.abort_external_stream_park_for_boundary(); + Some(ParkApplication::Aborted) + } + (Some(PendingPark::Issued(_)), Some(_)) if output_latency_preempts_park => { + self.waiting_on_local_work.pending_park = None; + self.abort_external_stream_park_for_boundary(); + Some(ParkApplication::Aborted) + } + (Some(PendingPark::Issued(reason)), Some(result)) => { + self.waiting_on_local_work.pending_park = None; + Some(self.apply_park_result(reason, result)) + } + (Some(PendingPark::Issued(reason)), None) => { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal(format!( + "Lang answered PrepareExternalStreamPark({reason:?}) without an \ + ExternalStreamParkResult. Core never manufactures a terminal, so no \ + marker is written and the Workflow Task is retried." + )), + complete_resp_chan: completion.resp_chan, + }); + } + (_, Some(_)) => { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "Lang sent ExternalStreamParkResult with no park handshake outstanding" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + (Some(PendingPark::Queued(_)) | None, None) => None, + }; + // A handshake in flight retains the Workflow Task in every state but one. Queued or + // issued, the set is in `Parking` and the job still has to be delivered and answered; + // aborted or stale, the resolve activation Core owes lang needs a task to arrive on. + // Only a *confirmed* park ends the task, and it ends it by writing the marker whose + // terminal the confirmation just supplied. + let park_retains = self.waiting_on_local_work.pending_park.is_some() + || matches!( + park_outcome, + Some(ParkApplication::Aborted | ParkApplication::Stale) + ); + let staged_output_from_losing_park = output_commit_pending + && matches!( + park_outcome, + Some(ParkApplication::Aborted | ParkApplication::Stale) + ); + if staged_output_from_losing_park { + // Lang may already have staged output before Core discovers that readiness beat its + // confirmation. The park terminal cannot be used, so finish rolling the park back and + // obtain a valid terminal through finalization before recording the staged batch. + self.abort_external_stream_park_for_boundary(); + } + + // A `FinalizeExternalStreams` job's only legal responses are `ExternalStreamFinalized` or + // an activation failure. Anything else means Core asked for a terminal and did not get + // one, and there is no best-effort path from there: writing a marker anyway would commit + // a truncated annotation, which is durable and wrong, so the Workflow Task is failed and + // retried instead. An abandoned task commits no cursor and loses no record. + let finalized_boundary = match ( + self.waiting_on_local_work.pending_finalization.take(), + &stream_commands.finalized, + ) { + (Some(reason), Some(finalized)) => { + if (reason == ParkReason::OutputLatency || output_was_buffered) + && !output_commit_pending + { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal(format!( + "Lang answered FinalizeExternalStreams({reason:?}) while external \ + output was buffered but supplied no WorkflowOutputStreamCommit" + )), + complete_resp_chan: completion.resp_chan, + }); + } + self.waiting_on_local_work + .external_wait_set + .accumulate_annotation(&finalized.final_observation_delta); + Some(reason) + } + (Some(reason), None) => { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal(format!( + "Lang answered FinalizeExternalStreams({reason:?}) without an \ + ExternalStreamFinalized command. Core never manufactures a terminal, so \ + no marker is written and the Workflow Task is retried." + )), + complete_resp_chan: completion.resp_chan, + }); + } + (None, Some(_)) => { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "Lang sent ExternalStreamFinalized with no finalization job outstanding" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + (None, None) => None, + }; + + // Accumulate on *every* completion path, retained or not: consuming a record and + // committing that consumption are separate steps, and the second is not conditional on + // why the Workflow Task ended. An empty delta accumulates like any other -- it is how a + // subscription that observed nothing still records that it observed. + for progress in &stream_commands.progress { + self.waiting_on_local_work + .external_wait_set + .accumulate_annotation(&progress.observation_delta); + if progress.request_rollover { + // Lang decided this boundary, so the rollover needs no finalization round trip -- + // this very command already carried the terminal. + self.waiting_on_local_work.budget_rollover_pending = true; + } + } + let completing_budget_rollover = + mem::take(&mut self.waiting_on_local_work.budget_rollover_pending); + // This Run is being torn down -- Worker shutdown, or an eviction -- while it still holds + // the Workflow Task (C15b). Like the rollover deadline it is a boundary *Core* decided, so + // the terminal is owed by a finalization round trip; and like the rollover intent it has + // to ride across that round trip, because the completion that *asked* consumed the flag. + let shutdown_was_pending = mem::take(&mut self.waiting_on_local_work.shutdown_pending); + let completing_shutdown = + shutdown_was_pending || finalized_boundary == Some(ParkReason::Shutdown); + // A finalization response carries the rollover intent across its round trip: the deadline + // flag was consumed by the completion that *asked*, so without this the completion that + // finally writes the marker would forget to request a replacement task. + let completing_rollover = completing_deadline_rollover + || completing_budget_rollover + || finalized_boundary == Some(ParkReason::Rollover); + let output_latency_was_pending = + mem::take(&mut self.waiting_on_local_work.output_latency_pending); + let completing_output_latency = + output_latency_was_pending || finalized_boundary == Some(ParkReason::OutputLatency); + let completing_park_rollback = finalized_boundary.is_some() + && mem::take(&mut self.waiting_on_local_work.park_rollback_resolve_pending); + if completing_park_rollback { + // The mixed resolve/finalize activation removes the losing park's backend intents, + // but finalization deliberately does not run Workflow code. Re-publishing readiness + // makes the forced replacement task perform the drain the losing park interrupted. + self.waiting_on_local_work + .external_wait_set + .mark_all_ready_after_aborted_park(); + } + + // Registering the wait set and retaining the Workflow Task are two separate questions, and + // they are decided separately here. Both are decided before anything is pushed into the + // machines so the marker can be ordered ahead of lang's own commands. + // + // These are the boundaries that answer *both* questions "no". Each of them either tears + // the Run down or hands it to a finalization round trip that has already named the current + // quiescence generation, so re-recording lang's snapshot would bump that generation + // underneath a job already in flight for it. + let boundary_closes_the_run = data.activation_was_eviction + || completing_rollover + || completing_shutdown + || completing_output_latency + || staged_output_from_losing_park + || finalized_boundary.is_some(); + // A pending rollover overrides a retention request. Lang asked to be held open without + // knowing the deadline had already expired, and honouring that would restart the deadline + // and hold the task past the timeout it exists to stay inside -- rollover is + // authoritative, so it wins. + // + // A teardown overrides it for a harder reason still: lang asked to be held open on a + // Worker that is going away, and honouring that would leave a Workflow Task retained with + // nothing left to release it and no replacement task ever coming. + let retention_requested = (stream_commands.quiescence.is_some() + || park_retains + || self.waiting_on_local_work.output_buffered) + && !boundary_closes_the_run; + // Replay runs no timers. A replayed Run still has to *register* its wait set -- that set + // is per-Worker runtime state, not History, so nothing else rebuilds it -- but retaining a + // replayed task would arm the idle and rollover deadlines against wall-clock time that has + // nothing to do with the recorded boundary, and a replay slower than the idle timeout + // would queue a park handshake in between replay activations. The marker reproduces the + // boundary instead; there is nothing here for a timer to decide. + let replaying = self.wfm.machines.replaying; + let answering_a_query = !data.query_responses.is_empty(); + let will_retain = retention_requested + && !has_server_bound_commands + && !answering_a_query + && !replaying + && !output_commit_pending; + + // Registering is not retaining. A completion that also produced a timer, activity, child + // workflow or signal must be reported so the server can act on it -- but the subscriptions + // that same completion described are still what the Workflow is blocked on, and Core is + // the only place they are recorded. Dropping them along with the retention leaves the Run + // permanently unresumable: nothing is registered, so a wake Signal marks nothing ready, + // readiness has no wait to resolve against, and every Workflow Task the wake produces is + // empty. + let registers_without_retaining = !will_retain && !boundary_closes_the_run; + // The one refusal lang's own commands do not account for. Lang asked to be held open, so + // its `WorkflowStreamProgress` deliberately carried no terminal -- the terminal was going + // to arrive on a park result or a finalization. Core is refusing because a query answer + // has to be reported, and that makes this a boundary *Core* decided, owed a finalization + // round trip exactly as the rollover deadline and a teardown are. Writing `TaskCompleted` + // here instead would commit an annotation with no terminal frame, which ADR-008 forbids + // without exception. + // + // "Still blocked" covers two shapes and has to cover both. Lang may have reported a fresh + // snapshot on this very completion, or the snapshot it reported earlier may still be + // registered and still retaining -- which is what a query answered on its own activation + // looks like, since the completion carrying the answer produces no stream command at all. + // Lang's terminal is no more available in the second shape than in the first, and the + // annotation at risk was accumulated before either. + let stream_waits_still_pending = stream_commands.quiescence.is_some() + || park_retains + || self.waiting_on_local_work.external_wait_set.retains_wft() + || self.waiting_on_local_work.output_buffered; + let query_refused_retention = stream_waits_still_pending + && !boundary_closes_the_run + && !has_server_bound_commands + && !replaying + && answering_a_query; + + // `WorkflowStreamProgress` precedes every command whose value could depend on the + // consumed data, and so must the marker recording it. Emitting after lang's commands were + // pushed would put the marker *after* the terminal command in History, and on replay the + // command would then be matched before the record it came from was validated. + let terminal = if will_retain { + None + } else if let Some(ParkApplication::Confirmed(reason)) = park_outcome { + // Park owns its own marker path. The terminal arrived on the park result itself, so + // there is no finalization round trip to wait for and nothing is owed -- which is why + // idle park is C8's and not the finalization protocol's. + Some(reason) + } else if let Some(reason) = finalized_boundary { + // Core decided this boundary and lang has now supplied its terminal. + Some(reason) + } else if lang_commands.iter().any(|c| c.variant.is_terminal()) { + Some(ParkReason::WorkflowCompleted) + } else if completing_output_capacity { + Some(ParkReason::OutputCapacity) + } else if completing_budget_rollover { + Some(ParkReason::BudgetRollover) + } else if has_server_bound_commands { + Some(ParkReason::CommandsProduced) + } else if completing_deadline_rollover + || completing_shutdown + || completing_output_latency + || staged_output_from_losing_park + || query_refused_retention + { + // Core decided this boundary and lang was never asked for a terminal. Nothing may be + // written until the finalization round trip below supplies one. + None + } else { + Some(ParkReason::TaskCompleted) + }; + + if terminal.is_some() + && self.waiting_on_local_work.output_buffered + && !output_commit_pending + { + return Err(RunUpdateErr { + source: WFMachinesError::Fatal( + "A Workflow Task reached a durable boundary while external output was still \ + buffered; lang must stage it and send WorkflowOutputStreamCommit first" + .to_string(), + ), + complete_resp_chan: completion.resp_chan, + }); + } + + if let Some(terminal) = terminal + && let Err(source) = self.emit_external_stream_marker(terminal) + { + return Err(RunUpdateErr { + source, + complete_resp_chan: completion.resp_chan, + }); + } + if terminal == Some(ParkReason::OutputCapacity) { + // `force_new_wft` obtains the task, and this one-shot flag makes that otherwise-empty + // task observable to lang. It is consumed only when the replacement is admitted. + self.waiting_on_local_work + .output_capacity_activation_pending = true; + } + + // Recorded *after* the marker, so a marker still closes the snapshot that was in effect + // while the records it carries were consumed, and *before* the finalization request below, + // so a job asking lang to finalize names the snapshot lang has just reported rather than + // the one it superseded. + if registers_without_retaining && let Some(request) = stream_commands.quiescence.take() { + self.register_external_stream_quiescence(request); + } + + // The rollover deadline, a teardown, and a query answer are the three boundaries that + // reach here still owing a terminal. `false` means there was no annotation to finalize, so + // the task simply completes with no marker -- nothing is owed and nothing is missing. + // Teardown outranks the deadline when both are pending: the replacement task the deadline + // wanted is the same one the teardown asks for, and one boundary gets one marker. + let awaiting_finalization = terminal.is_none() + && !will_retain + && (completing_deadline_rollover + || completing_shutdown + || completing_output_latency + || staged_output_from_losing_park + || query_refused_retention) + && self.begin_external_stream_finalization(if completing_shutdown { + ParkReason::Shutdown + } else if completing_deadline_rollover { + ParkReason::Rollover + } else if completing_output_latency { + ParkReason::OutputLatency + } else { + ParkReason::TaskCompleted + }); + + if awaiting_finalization { + self.cancel_external_output_flush_timer(); + } + let outcome = (|| { // Send commands from lang into the machines then check if the workflow run needs // another activation and mark it if so - self.wfm.push_commands_and_iterate(completion.commands)?; + self.wfm.push_commands_and_iterate(lang_commands)?; if let Some(update) = update_from_new_page { self.wfm.feed_history_from_new_page(update)?; } @@ -806,53 +1448,122 @@ impl ManagedRun { match outcome { Ok(None) => { - if let Some(waiting) = self.waiting_on_la.take() { + if let Some(waiting) = self.waiting_on_local_work.local_activities.take() { waiting.hb_timeout_handle.abort(); } + + // The finalization job is queued but not yet shipped. Reporting the task now would + // complete it before its terminal exists, so the task stays open until lang + // answers -- which is the whole point of Core never writing a marker for a + // boundary it decided without one. + if awaiting_finalization { + // A query answer that reached here must survive the round trip. The task is + // still open, so nothing has been reported to the server yet, and the + // finalization activation runs no user Workflow code -- lang cannot resend + // what it already answered. + self.waiting_on_local_work.deferred_query_responses = + mem::take(&mut data.query_responses); + return Ok(Some(FulfillableActivationComplete { + result: ActivationCompleteResult { + outcome: ActivationCompleteOutcome::DoNothing, + replaying: self.wfm.machines.replaying, + }, + resp_chan: completion.resp_chan, + })); + } + + // Retention applies only when nothing server-bound rides along. A completion that + // also produced a timer, activity, child workflow, or signal must be reported so + // the server can act on it; the subscriptions stay registered and are woken by + // the wake Signal instead. + if will_retain { + // An abandoned park retains without a new snapshot: the quiescent generation + // it failed to close is still the current one, so re-recording it would bump + // the generation and strand any readiness already accepted against the old. + if let Some(request) = stream_commands.quiescence { + self.begin_external_stream_quiescence(request); + } + return Ok(Some(FulfillableActivationComplete { + result: ActivationCompleteResult { + outcome: ActivationCompleteOutcome::DoNothing, + replaying: self.wfm.machines.replaying, + }, + resp_chan: completion.resp_chan, + })); + } + + // Not retaining: whatever quiescent snapshot was recorded no longer holds a task + // open, so its timers must not outlive it. + self.cancel_external_stream_idle_timer(); + self.cancel_wft_rollover_timer(); + Ok(Some(self.prepare_complete_resp( completion.resp_chan, data, - completing_heartbeat_autocomplete, + completing_heartbeat_autocomplete + || completing_rollover + || completing_shutdown + || completing_output_capacity + || completing_output_latency + || (output_commit_pending && stream_waits_still_pending), ))) } Ok(Some((start_t, wft_timeout))) => { - if let Some(wola) = self.waiting_on_la.as_mut() { + if let Some(wola) = self.waiting_on_local_work.local_activities.as_mut() { wola.hb_timeout_handle.abort(); } if completing_la_heartbeat || !data.query_responses.is_empty() { // Reporting a query while an LA is still running must request another WFT; // otherwise the LA could resolve without a task on which to deliver its job. - let hb_timeout_handle = sink_heartbeat_timeout_start( - self.run_id().to_string(), - self.local_activity_request_sink.as_deref(), + let hb_timeout_handle = Self::start_la_heartbeat_timeout_with( + &self.run_timers, + self.run_id(), start_t, wft_timeout, ); hb_timeout_handle.abort(); - self.waiting_on_la = Some(WaitingOnLAs { - wft_timeout, - hb_timeout_handle, - // Keep this set until the replacement WFT arrives. If pending workflow - // jobs prevent this completion from being reported, the heartbeat still - // needs to be honored after those jobs are processed. - heartbeat_timeout_pending: completing_la_heartbeat, - }); + self.waiting_on_local_work.local_activities = + Some(LocalActivityHeartbeatState { + wft_timeout, + hb_timeout_handle, + // Keep this set until the replacement WFT arrives. If pending workflow + // jobs prevent this completion from being reported, the heartbeat still + // needs to be honored after those jobs are processed. + heartbeat_timeout_pending: completing_la_heartbeat, + }); Ok(Some(self.prepare_complete_resp( completion.resp_chan, data, true, ))) } else { - self.waiting_on_la = Some(WaitingOnLAs { - wft_timeout, - hb_timeout_handle: sink_heartbeat_timeout_start( - self.run_id().to_string(), - self.local_activity_request_sink.as_deref(), - start_t, + self.waiting_on_local_work.local_activities = + Some(LocalActivityHeartbeatState { wft_timeout, - ), - heartbeat_timeout_pending: false, - }); + hb_timeout_handle: Self::start_la_heartbeat_timeout_with( + &self.run_timers, + self.run_id(), + start_t, + wft_timeout, + ), + heartbeat_timeout_pending: false, + }); + // Retaining the task reports nothing to the server, so a rollover that was + // pending has not been acted on and must stay pending -- otherwise the + // deadline would be silently swallowed by the very completion that keeps the + // task open past it. Unless a finalization is already in flight for it, in + // which case that round trip carries the intent and re-arming here would ask + // for the same boundary twice. + self.waiting_on_local_work.deadline_rollover_pending |= + completing_deadline_rollover && !awaiting_finalization; + self.waiting_on_local_work.budget_rollover_pending |= + completing_budget_rollover; + // Same for the teardown intent, and for the same reason: the task is still + // open, so nothing has been handed back to the task queue yet. Only the raw + // flag is restored -- a boundary already carried by a finalization in flight + // would otherwise be asked for twice. + self.waiting_on_local_work.shutdown_pending |= + shutdown_was_pending && !awaiting_finalization; Ok(Some(FulfillableActivationComplete { result: ActivationCompleteResult { outcome: ActivationCompleteOutcome::DoNothing, @@ -882,6 +1593,690 @@ impl ManagedRun { } } + // --- External stream retention (C6) ------------------------------------ + + /// This run's workflow task timeout, which both run-level deadlines derive from. + fn wft_timeout(&self) -> Option { + self.wfm + .machines + .get_started_info() + .and_then(|attrs| attrs.workflow_task_timeout) + } + + /// When the workflow task this run currently holds started. + /// + /// The rollover deadline is anchored here rather than at the moment it is (re)armed, because + /// what it bounds is the *task*, not the wait it was armed for. + fn current_wft_start_time(&self) -> Instant { + self.wft + .as_ref() + .map(|wft| wft.start_time) + .unwrap_or_else(Instant::now) + } + + /// Writes the marker for a Workflow Task lang itself closed. + /// + /// These are the three paths where lang's own `WorkflowStreamProgress` carried the terminal, + /// so no finalization round trip is needed -- Core already has everything it needs. The + /// Core-decided boundaries (park, rollover deadline, shutdown) each integrate against this + /// same primitive but obtain their terminal first. + fn emit_external_stream_marker( + &mut self, + terminal_boundary: ParkReason, + ) -> Result<(), WFMachinesError> { + let output = self + .waiting_on_local_work + .pending_output_commit + .take() + .and_then(|commit| commit.manifest); + let set = &mut self.waiting_on_local_work.external_wait_set; + if set.replay_annotation().is_empty() && output.is_none() { + // Nothing was observed, so there is nothing to record. Not an error: a Workflow Task + // that touched no stream is the ordinary case. + return Ok(()); + } + let quiescence_generation = set.quiescence_generation(); + let waits = set + .marker_waits() + .into_iter() + .map(|(wait_id, generation)| ExternalWaitMarker { + wait_id, + generation, + }) + .collect(); + let replay_annotation = set + .take_annotation() + .map_err(|err| WFMachinesError::Fatal(err.to_string()))?; + + self.wfm + .machines + .emit_external_stream_marker(ExternalStreamMarkerData { + schema_version: EXTERNAL_STREAM_MARKER_SCHEMA_VERSION, + quiescence_generation, + waits, + replay_annotation, + terminal_boundary: terminal_boundary as i32, + output, + }) + } + + /// Asks lang for the terminal of a boundary **Core** decided (C15a). + /// + /// The annotation ends with a blocked cursor snapshot and only lang can encode it, so Core + /// cannot close a boundary it decided without asking. This is the protocol primitive, and it + /// is deliberately independent of which boundary triggered it -- rollover and shutdown both + /// come through here. Park does **not**: it obtains its terminal from + /// `ExternalStreamParkResult`, which is a different round trip. + /// + /// Returns `true` when a job was issued. `false` means there is nothing to finalize, because + /// nothing was accumulated -- so no marker is owed and none is missing. + fn begin_external_stream_finalization(&mut self, reason: ParkReason) -> bool { + if self.waiting_on_local_work.pending_finalization.is_some() { + // Already asked. A second job for one boundary would put two runtime-internal + // activations in flight, and there is never more than one outstanding per run. + return true; + } + let set = &self.waiting_on_local_work.external_wait_set; + if set.replay_annotation().is_empty() + && !self.waiting_on_local_work.output_buffered + && self.waiting_on_local_work.pending_output_commit.is_none() + { + return false; + } + let quiescence_generation = set.quiescence_generation(); + let waits = set + .wait_snapshot() + .into_iter() + .map( + |(wait_id, generation, immediately_parkable)| ExternalStreamWait { + wait_id, + generation, + immediately_parkable, + }, + ) + .collect(); + + self.waiting_on_local_work.pending_finalization = Some(reason); + self.wfm.machines.send_core_generated_job( + workflow_activation_job::Variant::FinalizeExternalStreams(FinalizeExternalStreams { + quiescence_generation, + waits, + reason: reason as i32, + }), + ); + true + } + + /// The Worker is shutting down while this Run may still be holding a Workflow Task (C15b). + /// + /// Nothing else will close that boundary: the pollers are stopped so no replacement task is + /// coming, lang is not being activated, and `shutdown_done` counts an open Workflow Task as + /// pending work -- so a Run retained by an external stream wait set would keep the whole + /// Worker from finishing. + /// + /// Runs with no open Workflow Task are left untouched, deliberately. That is not an omission: + /// nothing is accumulated there, so no marker is missing, and `force_new_wft` needs a task + /// token the Run does not have. Lang's wake sweep is the server-visible replacement, and Core + /// reimplementing it here would send a Signal for a Run that is about to be finalized anyway. + pub(super) fn external_stream_shutdown(&mut self) -> RunUpdateAct { + if !self.begin_external_stream_teardown() { + return None; + } + let res = self._check_more_activations(); + self.update_to_acts(res.map(Into::into)) + } + + /// Records that this Run's open Workflow Task must be closed because the Run is going away. + /// + /// Returns whether this Run is in the state ADR-009's first row is about. The classification + /// is the same one lang's shutdown sweep gets from `external_stream_run_status`, on purpose: + /// the two halves must agree about which Run is in which state, or a Run would be swept by + /// both mechanisms or by neither. + fn begin_external_stream_teardown(&mut self) -> bool { + // `wft` is Core's own record of holding the task; the probe additionally distinguishes a + // parked set, which holds no task open even though the Run may still be cached. + if self.wft.is_none() + || (!self.waiting_on_local_work.output_buffered + && !matches!( + self.external_stream_run_status(), + ExternalStreamRunStatus::WftOpen + )) + { + return false; + } + self.cancel_external_output_flush_timer(); + self.waiting_on_local_work.shutdown_pending = true; + true + } + + /// Moves the complete set into `Parking` and asks lang to run the backend handshake (C8). + /// + /// Runtime-internal: no user Workflow code runs for this job. Lang installs one park intent + /// per subscription, rechecks every stream, and answers `ExternalStreamParkResult` -- + /// `ParkSetConfirmed` carrying the terminal only Core cannot encode, or `StreamSetBecameReady` + /// abandoning this parking generation. + /// + /// Returns `true` when a job was issued. `false` means the set refused to park: the generation + /// named is no longer current, or readiness Core already accepted is sitting in it, and + /// parking a set with a ready wait in it would strand that wait's record until a producer + /// happened to signal. + fn begin_external_stream_park( + &mut self, + quiescence_generation: u64, + trigger: ParkTrigger, + ) -> bool { + if self.waiting_on_local_work.pending_park.is_some() { + // One handshake at a time. A second job for one set would put two runtime-internal + // activations in flight, and there is never more than one outstanding per run. + return false; + } + let set = &mut self.waiting_on_local_work.external_wait_set; + let reason = match set.start_parking(quiescence_generation, trigger) { + ParkStartOutcome::Started(ParkTrigger::IdleTimeout) => ParkReason::Idle, + ParkStartOutcome::Started(ParkTrigger::AllWriteFenced) => ParkReason::AllWriteFenced, + ParkStartOutcome::StaleGeneration | ParkStartOutcome::AlreadyReady => return false, + }; + let waits = set + .wait_snapshot() + .into_iter() + .map( + |(wait_id, generation, immediately_parkable)| ExternalStreamWait { + wait_id, + generation, + immediately_parkable, + }, + ) + .collect(); + + // The quiescent snapshot this park closes is over. Its idle timer must not survive the + // handshake: firing again would start a second park for a set already in one. + self.cancel_external_stream_idle_timer(); + self.waiting_on_local_work.pending_park = Some(PendingPark::Queued(reason)); + self.wfm.machines.send_core_generated_job( + workflow_activation_job::Variant::PrepareExternalStreamPark( + PrepareExternalStreamPark { + quiescence_generation, + waits, + reason: reason as i32, + }, + ), + ); + true + } + + /// Applies lang's answer to `PrepareExternalStreamPark`. + /// + /// Both orderings of the readiness/park race resolve here, through the one pure function that + /// decides them: readiness accepted while the handshake was in flight has already moved a wait + /// out of `Parking`, and that is exactly what makes the confirmation which follows stale. + fn apply_park_result( + &mut self, + reason: ParkReason, + result: ExternalStreamParkResult, + ) -> ParkApplication { + let confirmed = matches!( + result.outcome, + Some(external_stream_park_result::Outcome::Confirmed(_)) + ); + let set = &mut self.waiting_on_local_work.external_wait_set; + match set.resolve_park(result.quiescence_generation, confirmed) { + ParkResolution::Confirmed => { + // The terminal arrives *here*, on the park result -- park issues no finalization + // job, so this is the only chance Core gets to obtain it before writing. + set.accumulate_annotation(&result.final_observation_delta); + ParkApplication::Confirmed(reason) + } + ParkResolution::Aborted => { + // The recheck found records. No boundary was reached, so nothing is written; lang + // is resumed by a *normal* resolve activation rather than by user code running + // from inside the park path. + set.mark_all_ready_after_aborted_park(); + ParkApplication::Aborted + } + // Readiness beat the confirmation, or a later snapshot replaced the one being parked. + // Discarded with no effect and, above all, no marker: the boundary it claims to close + // was never reached. + ParkResolution::StaleGeneration => ParkApplication::Stale, + } + } + + fn abort_external_stream_park_for_boundary(&mut self) { + self.waiting_on_local_work + .external_wait_set + .abort_park_for_boundary(); + self.waiting_on_local_work.park_rollback_resolve_pending = true; + } + + /// Records a quiescent snapshot, and nothing else. + /// + /// This is the *registration* half of quiescence, and it exists apart from the retaining half + /// because they answer different questions. What lang reports here is the set of subscriptions + /// the Workflow is blocked on, and Core is the only place that set is recorded -- a completion + /// that also produced a timer must be reported to the server, but the Workflow is no less + /// blocked on those streams for it, and a Run whose waits were never registered can never be + /// woken: readiness has no wait to resolve against and a wake Signal marks nothing ready. + /// + /// Nothing is armed here. No idle timer, no rollover deadline, and no all-fenced immediate + /// park -- each of those exists to end a *retained* Workflow Task, and a task that is about to + /// be reported, or that is being replayed, has no retention for them to end. + /// + /// Returns the new quiescence generation and the clamped idle timeout the snapshot was + /// recorded with, so the retaining path can arm its deadlines against the same values. + fn register_external_stream_quiescence( + &mut self, + request: QuiescenceRequest, + ) -> (u64, Duration) { + let idle_timeout = clamp_idle_below_rollover(request.idle_timeout, self.wft_timeout()); + let generation = self + .waiting_on_local_work + .external_wait_set + .become_quiescent(request.waits, idle_timeout); + (generation, idle_timeout) + } + + /// Records a quiescent snapshot and starts the timers that bound it. + /// + /// **One** idle timer for the whole wait set, not one per subscription: the timeout measures + /// *global* quiescence, so an idle stream cannot park a workflow task another stream is still + /// driving. + fn begin_external_stream_quiescence(&mut self, request: QuiescenceRequest) { + // The rollover deadline is what stops a continuously fed stream -- one whose gaps never + // reach the idle timeout -- from holding the task until it *fails*. + // + // It is anchored at the Workflow Task's start, not at this snapshot. Becoming quiescent + // again is what a delivered record *does*, so re-anchoring here would push the deadline + // out for as long as records keep arriving -- and that is exactly the workload rollover + // exists for. The deadline would then never fire, the idle timer is clamped below it and + // cannot fire either, and the retained task would run until the server timed it out. + if let Some(wft_timeout) = self.wft_timeout() { + let started_at = self.current_wft_start_time(); + self.start_wft_rollover_timer(started_at, wft_timeout); + } + + let (generation, idle_timeout) = self.register_external_stream_quiescence(request); + + self.cancel_external_stream_idle_timer(); + + // Every wait reached a write fence with no later record immediately available, so there is + // nothing left for the idle delay to wait *for*: park now instead of holding the task open + // for a timeout that can only expire. One fenced stream does not qualify -- parking is + // all-or-nothing across the set -- and a set holding accepted readiness refuses, which is + // what leaves the resolve below reachable. + if self + .waiting_on_local_work + .external_wait_set + .all_immediately_parkable() + && self.begin_external_stream_park(generation, ParkTrigger::AllWriteFenced) + { + return; + } + + self.waiting_on_local_work.idle_timer = Some(self.run_timers.start( + Instant::now().add(idle_timeout), + LocalInputs::ExternalStreamIdleTimeout(ExternalStreamIdleTimeoutMsg { + run_id: self.wfm.machines.run_id.clone(), + quiescence_generation: generation, + }), + )); + + // Readiness that arrived while lang was computing this snapshot survives it when the + // generations match, and must still reach lang. + self.maybe_issue_external_stream_resolve(); + } + + /// Re-arms the idle and rollover deadlines for a wait set carried onto a replacement task. + /// + /// The quiescent snapshot is *not* renewed: the same generation continues, so a readiness + /// notification in flight across the rollover still matches and is not lost. + fn restart_external_stream_deadlines(&mut self, start_time: Instant) { + let wft_timeout = self.wft_timeout(); + if let Some(wft_timeout) = wft_timeout { + self.start_wft_rollover_timer(start_time, wft_timeout); + } + let idle_timeout = clamp_idle_below_rollover( + self.waiting_on_local_work + .external_wait_set + .idle_timeout() + .unwrap_or(Duration::from_secs(1)), + wft_timeout, + ); + let generation = self + .waiting_on_local_work + .external_wait_set + .quiescence_generation(); + self.cancel_external_stream_idle_timer(); + self.waiting_on_local_work.idle_timer = Some(self.run_timers.start( + start_time.add(idle_timeout), + LocalInputs::ExternalStreamIdleTimeout(ExternalStreamIdleTimeoutMsg { + run_id: self.wfm.machines.run_id.clone(), + quiescence_generation: generation, + }), + )); + } + + fn cancel_external_stream_idle_timer(&mut self) { + if let Some(handle) = self.waiting_on_local_work.idle_timer.take() { + handle.abort(); + } + } + + fn note_external_output_buffered(&mut self, max_publish_latency: Duration) { + self.waiting_on_local_work.output_buffered = true; + + if let Some(wft_timeout) = self.wft_timeout() { + self.start_wft_rollover_timer(self.current_wft_start_time(), wft_timeout); + } + + let deadline = Instant::now().add(max_publish_latency); + if self + .waiting_on_local_work + .output_flush_timer + .as_ref() + .is_some_and(|timer| timer.deadline <= deadline) + { + return; + } + self.cancel_external_output_flush_timer(); + self.waiting_on_local_work.output_flush_timer = Some(OutputFlushTimer { + deadline, + handle: self.run_timers.start( + deadline, + LocalInputs::ExternalOutputFlushDeadline(self.wfm.machines.run_id.clone()), + ), + }); + } + + fn cancel_external_output_flush_timer(&mut self) { + if let Some(timer) = self.waiting_on_local_work.output_flush_timer.take() { + timer.handle.abort(); + } + } + + fn clear_external_output_buffered(&mut self) { + self.cancel_external_output_flush_timer(); + self.waiting_on_local_work.output_buffered = false; + } + + // --- Run-level timers -------------------------------------------------- + + /// The local-activity heartbeat deadline, now scheduled through the run's own timer facility. + /// + /// A static so it can be called while `self.waiting_on_local_work` is mutably borrowed. The + /// deadline itself is unchanged: 80% of the workflow task timeout. + fn start_la_heartbeat_timeout_with( + timers: &RunTimerSink, + run_id: &str, + wft_start_time: Instant, + wft_timeout: Duration, + ) -> AbortHandle { + let deadline = wft_start_time.add(wft_timeout.mul_f32(WFT_HEARTBEAT_TIMEOUT_FRACTION)); + timers.start(deadline, LocalInputs::HeartbeatTimeout(run_id.to_string())) + } + + /// Starts (or restarts) this run's workflow task rollover deadline. + /// + /// A retained workflow task is bounded by the server's workflow task timeout, so a + /// continuously fed stream whose gaps stay below the idle timeout would otherwise hold the + /// task until it *fails* rather than merely being held too long. + #[allow(dead_code, reason = "started by the retention path in C6")] + pub(super) fn start_wft_rollover_timer( + &mut self, + wft_start_time: Instant, + wft_timeout: Duration, + ) { + self.cancel_wft_rollover_timer(); + let deadline = wft_start_time.add(wft_timeout.mul_f32(WFT_HEARTBEAT_TIMEOUT_FRACTION)); + self.waiting_on_local_work.wft_rollover_timer = Some(self.run_timers.start( + deadline, + LocalInputs::WftRolloverDeadline(self.wfm.machines.run_id.clone()), + )); + } + + #[allow(dead_code, reason = "cancelled by the retention path in C6")] + pub(super) fn cancel_wft_rollover_timer(&mut self) { + if let Some(handle) = self.waiting_on_local_work.wft_rollover_timer.take() { + handle.abort(); + } + } + + /// The rollover deadline expired. + /// + /// Records that the next completion must request a replacement task, and autocompletes if + /// there is no activation outstanding to carry it. Preserving every subscription, cursor, and + /// readiness generation across that replacement is C12a's. + pub(super) fn wft_rollover_deadline(&mut self) -> RunUpdateAct { + self.waiting_on_local_work.wft_rollover_timer = None; + self.waiting_on_local_work.deadline_rollover_pending = true; + let maybe_act = if self.activation.is_none() && self.wft.is_some() { + Some(ActivationOrAuto::Autocomplete { + run_id: self.wfm.machines.run_id.clone(), + }) + } else { + None + }; + self.update_to_acts(Ok(maybe_act.into())) + } + + /// The earliest max-publish-latency deadline for buffered output expired. + pub(super) fn external_output_flush_deadline(&mut self) -> RunUpdateAct { + self.waiting_on_local_work.output_flush_timer = None; + if !self.waiting_on_local_work.output_buffered || self.wft.is_none() { + return None; + } + + if self.activation.is_some() { + self.waiting_on_local_work.output_latency_pending = true; + return None; + } + + if !self.begin_external_stream_finalization(ParkReason::OutputLatency) { + return None; + } + let res = self._check_more_activations(); + self.update_to_acts(res.map(Into::into)) + } + + // --- External Workflow Streams ----------------------------------------- + + /// A watcher reports a record is buffered for one wait. + /// + /// Returns the acknowledgement alongside any activation, because the watcher's next action + /// depends on which of the five results this was and only the wait set can say. + pub(super) fn external_stream_ready( + &mut self, + wait_id: u32, + wait_generation: u64, + ) -> (ExternalStreamReadyResult, RunUpdateAct) { + let outcome = self + .waiting_on_local_work + .external_wait_set + .notify_ready(wait_id, wait_generation); + + if outcome != ReadinessOutcome::Accepted { + return (outcome.into(), None); + } + + // Readiness ends this quiescent snapshot, so the timer measuring it must not survive it. + // The rollover deadline is *not* cancelled: it bounds the workflow task itself, which is + // still open and still being held. + self.cancel_external_stream_idle_timer(); + + // `_check_more_activations` is what turns pending readiness into an activation, and it + // does so whether readiness arrived now or while an earlier activation was outstanding -- + // so both orderings coalesce through one path. + let act = match self._check_more_activations() { + Ok(act) => self.update_to_acts(Ok(act.into())), + Err(err) => self.update_to_acts(Err(err)), + }; + (outcome.into(), act) + } + + /// Classifies the wake Signals the machines decoded out of this task's history. + /// + /// Returns `true` if any of them should wake the Run. Every one is suppressed from user + /// handlers regardless -- that already happened in the machines -- so what is decided here is + /// only whether the Run resumes. + fn apply_external_stream_wakes(&mut self) -> bool { + let wakes = self.wfm.machines.take_external_stream_wakes(); + if wakes.is_empty() { + return false; + } + let chain = self + .wfm + .machines + .get_started_info() + .map(|info| info.first_execution_run_id.clone()) + .unwrap_or_default(); + + let mut resume = false; + for wake in wakes { + // Chain identity, not Run identity. The Signal is addressed to the Workflow ID + // without a Run ID, so it always lands on the current Run of the chain -- and a + // Signal from a *different* chain is a mis-addressed message, not a stale one. + // + // Compared strictly, including when this run's chain id is unknown: a Signal naming + // a chain we cannot confirm is ours is exactly the case that must not be honoured. + if wake.first_execution_run_id != chain { + debug!( + signalled_chain = %wake.first_execution_run_id, + "Rejecting an external stream wake Signal for a different chain" + ); + continue; + } + if !self + .waiting_on_local_work + .external_wait_set + .accepts_wake_generation(wake.park_generation) + { + // A *non-zero* generation this Run does not recognise is a claim that turned out + // to be wrong. Generation 0 is never rejected here: it is the unparked wake, and + // an unnecessary one costs at most one empty Workflow Task. + debug!( + park_generation = wake.park_generation, + "Ignoring a stale external stream wake Signal" + ); + continue; + } + resume = true; + } + + if resume { + // The Signal names one stream, but it is only a hint: every active wait is marked so + // lang rechecks all of them on wakeup. A wake for a stream that turns out to have + // nothing costs one empty drain, and missing one costs a stalled Workflow. + self.waiting_on_local_work + .external_wait_set + .mark_all_ready_for_wake(); + } + resume + } + + /// Queues one `ResolveExternalStreamWaits` if readiness is pending and a task can carry it. + /// + /// Coalescing lives here rather than at the notification: every wait known ready ships in one + /// activation, and notifications arriving while an activation is outstanding accumulate for + /// the next one. There is never more than one outstanding activation per run. + fn maybe_issue_external_stream_resolve(&mut self) { + if self.activation.is_some() || self.wft.is_none() || self.am_broken { + return; + } + let set = &mut self.waiting_on_local_work.external_wait_set; + if !set.has_pending_readiness() { + return; + } + let quiescence_generation = set.quiescence_generation(); + let ready_hints = set + .take_ready_wait_ids() + .into_iter() + .filter_map(|wait_id| { + set.wait(wait_id).map(|w| ExternalStreamWait { + wait_id: w.wait_id, + generation: w.wait_generation, + immediately_parkable: w.immediately_parkable, + }) + }) + .collect(); + self.wfm.machines.send_core_generated_job( + workflow_activation_job::Variant::ResolveExternalStreamWaits( + ResolveExternalStreamWaits { + quiescence_generation, + ready_hints, + }, + ), + ); + } + + /// Test scaffolding -- see [`ExternalStreamSeedWaitsMsg`]. + #[cfg(test)] + pub(super) fn seed_external_wait_set(&mut self, msg: &super::ExternalStreamSeedWaitsMsg) { + let set = &mut self.waiting_on_local_work.external_wait_set; + set.become_quiescent( + msg.wait_ids + .iter() + .map(|id| super::external_streams::ExternalWaitState::new(*id, 0, false)), + msg.idle_timeout, + ); + if let Some(generation) = msg.parked_at { + set.force_parked(generation); + } else { + set.set_wft_open(msg.wft_open); + } + } + + /// Test scaffolding -- see [`super::EmitTerminalLessMarkerMsg`]. + #[cfg(test)] + pub(super) fn emit_terminal_less_marker(&mut self) -> bool { + self.wfm + .machines + .emit_external_stream_marker(ExternalStreamMarkerData { + schema_version: EXTERNAL_STREAM_MARKER_SCHEMA_VERSION, + quiescence_generation: 1, + waits: vec![], + replay_annotation: b"no terminal here".to_vec(), + terminal_boundary: ParkReason::Unspecified as i32, + output: None, + }) + .is_err() + } + + /// The accumulated, unwritten replay annotation. Core never parses it. + #[cfg(test)] + pub(super) fn external_stream_annotation(&self) -> &[u8] { + self.waiting_on_local_work + .external_wait_set + .replay_annotation() + } + + /// The read-only status probe. Must leave the run exactly as it was. + pub(super) fn external_stream_run_status(&self) -> ExternalStreamRunStatus { + if self.waiting_on_local_work.external_wait_set.is_empty() { + // No waits registered at all. The run is cached but holds nothing this probe is + // about, which for the sweep is the same instruction as "no open Workflow Task". + return ExternalStreamRunStatus::NoOpenWorkflowTask; + } + self.waiting_on_local_work + .external_wait_set + .run_status() + .into() + } + + /// The global quiescence timer for `quiescence_generation` expired. + /// + /// Nothing was delivered on any active wait for the whole timeout, so the complete set is + /// asked to park. A timer for a snapshot the Workflow has already run past changes nothing. + pub(super) fn external_stream_idle_timeout( + &mut self, + quiescence_generation: u64, + ) -> RunUpdateAct { + // The handle is spent: this *is* its expiry. + self.waiting_on_local_work.idle_timer = None; + if !self.begin_external_stream_park(quiescence_generation, ParkTrigger::IdleTimeout) { + return None; + } + let res = self._check_more_activations(); + self.update_to_acts(res.map(Into::into)) + } + pub(super) fn heartbeat_timeout(&mut self) -> RunUpdateAct { let maybe_act = if self._heartbeat_timeout() { Some(ActivationOrAuto::Autocomplete { @@ -894,7 +2289,7 @@ impl ManagedRun { } /// Returns `true` if autocompletion should be issued to report the heartbeat WFT completion. fn _heartbeat_timeout(&mut self) -> bool { - if let Some(ref mut wait_dat) = self.waiting_on_la { + if let Some(ref mut wait_dat) = self.waiting_on_local_work.local_activities { wait_dat.hb_timeout_handle.abort(); wait_dat.heartbeat_timeout_pending = true; return self.activation.is_none(); @@ -936,7 +2331,12 @@ impl ManagedRun { ) -> Option { let about_to_issue_evict = self.trying_to_evict.is_some(); let has_activation = self.activation().is_some(); - if has_activation || about_to_issue_evict || self.more_pending_work() { + if must_buffer_wft( + self.wft.is_some(), + has_activation, + about_to_issue_evict, + self.more_pending_work(), + ) { debug!(run_id = %self.run_id(), "Got new WFT for a run with outstanding work, buffering it act: {:?} wft: {:?} about to evict: {:?}", &self.activation(), &self.wft, about_to_issue_evict); self.task_buffer.buffer(work); @@ -1146,6 +2546,15 @@ impl ManagedRun { one outstanding: {old_act:?}" ); } + // A normal activation drains every queued job, so a park handshake that was waiting for + // one is now in lang's hands -- and the completion answering *this* activation is the one + // that owes Core the park result. An eviction or autocomplete drains nothing, so a job + // queued behind it is still only queued. + if matches!(act_type, OutstandingActivation::Normal) + && let Some(PendingPark::Queued(reason)) = self.waiting_on_local_work.pending_park + { + self.waiting_on_local_work.pending_park = Some(PendingPark::Issued(reason)); + } self.activation = Some(act_type); } @@ -1190,6 +2599,28 @@ impl ManagedRun { force_new_wft = true; } + // Reporting the task ends local delivery, and it ends it *here* rather than at + // `mark_wft_complete`, which runs only once the server has answered. `Accepted` promises + // the watcher that Core will activate, and a task on its way to the server can no longer + // carry that activation: from this instant readiness is answered `NoOpenWorkflowTask` + // instead, which is the one answer that tells the watcher to send the wake Signal itself. + // + // Readiness already accepted against this task has no such fallback -- its watcher was + // told to do nothing and will not report again -- and the completion paths that reach + // here never turn it into a job: a snapshot that registers without retaining (a replaying + // completion, one carrying server-bound commands, or one answering a query) records the + // wait set and reports the task, and `_check_more_activations` is not on that path. So the + // replacement task is asked for explicitly, and `apply_new_wft` re-opens the wait set on + // it and issues the resolve job the pending readiness was promised. Without it the record + // stays buffered behind a Run that Core believes it has already told. + if should_respond || has_query_responses { + let waits = &mut self.waiting_on_local_work.external_wait_set; + waits.set_wft_open(false); + if waits.has_pending_readiness() { + force_new_wft = true; + } + } + let outcome = if should_respond || has_query_responses { // If we broke there could be commands or messages in the pipe that we didn't // get a chance to handle properly during replay. Don't send them. @@ -1375,30 +2806,164 @@ fn put_queries_in_act(act: &mut WorkflowActivation, wft: &mut OutstandingTask) { .map(|q| workflow_activation_job::Variant::QueryWorkflow(q).into()); act.jobs.extend(query_jobs); } -fn sink_heartbeat_timeout_start( - run_id: String, - sink: Option<&dyn LocalActivityRequestSink>, - wft_start_time: Instant, - wft_timeout: Duration, -) -> AbortHandle { - // The heartbeat deadline is 80% of the WFT timeout - let deadline = wft_start_time.add(wft_timeout.mul_f32(WFT_HEARTBEAT_TIMEOUT_FRACTION)); - let (abort_handle, abort_reg) = AbortHandle::new_pair(); - if let Some(la_sink) = sink { - la_sink.sink_reqs(vec![LocalActRequest::StartHeartbeatTimeout { - send_on_elapse: HeartbeatTimeoutMsg { - run_id, - span: Span::current(), - }, - deadline, - abort_reg, - }]); + +/// The annotation format version this Core writes. +/// +/// Leads the marker envelope so a marker written by an older SDK stays readable. +const EXTERNAL_STREAM_MARKER_SCHEMA_VERSION: u32 = 1; + +/// Where a park handshake is between Core deciding to park and lang answering. +/// +/// The two states are not decoration: the completion that must carry an `ExternalStreamParkResult` +/// is the one that answers the activation *carrying* the job, and an activation can already be +/// outstanding when the set begins parking -- the all-fenced trigger fires from inside a +/// completion, and a timer can expire while lang is still working. Enforcing the pairing against +/// `Queued` would fail a completion for a job lang has not been handed yet. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PendingPark { + /// The job is queued; lang has not been activated with it. + Queued(ParkReason), + /// Lang holds the job. The completion that answers it must carry the result. + Issued(ParkReason), +} + +/// What lang's `ExternalStreamParkResult` did to the wait set. +/// +/// Three outcomes, not two, because a confirmation that lost the race with readiness is neither a +/// park nor an abort: nothing was parked and nothing was rechecked, so the completion must leave +/// the task exactly as it found it -- above all writing no marker for a boundary never reached. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ParkApplication { + /// The set is parked. Carries the reason, which becomes the marker's terminal boundary. + Confirmed(ParkReason), + /// The final recheck found records; every wait is ready again. + Aborted, + /// The result named a generation that is no longer parking. Discarded. + Stale, +} + +/// Lang's `WorkflowStreamQuiescent`, validated. +struct QuiescenceRequest { + waits: Vec, + idle_timeout: Duration, +} + +/// The external stream commands pulled out of a completion. +/// +/// They are consumed above the state machines: progress accumulates into the wait set, and the +/// other three answer runtime-internal activations. Letting them reach the machines would drop a +/// replay-visible observation delta on the floor. +#[derive(Default)] +struct ExternalStreamCommands { + quiescence: Option, + progress: Vec, + park_result: Option, + finalized: Option, + output_commit: Option, + output_buffered_latency: Option, +} + +/// Splits lang's commands, leaving everything else in `commands`. +fn take_external_stream_commands( + commands: &mut Vec, +) -> Result { + let mut taken = ExternalStreamCommands::default(); + let mut remaining = Vec::with_capacity(commands.len()); + let mut seen_other_command = false; + for command in commands.drain(..) { + match command.variant { + WFCommandVariant::ExternalStreamQuiescent(q) => { + let idle_timeout = q + .idle_timeout + .map(|d| d.try_into().unwrap_or(Duration::ZERO)) + .unwrap_or(Duration::ZERO); + if idle_timeout.is_zero() { + // Rejected rather than coerced: a zero or absent timeout would either park + // instantly or hold the task until it timed out, and neither is something a + // caller can have meant. + return Err(WFMachinesError::Fatal( + "WorkflowStreamQuiescent carried a non-positive idle timeout".to_string(), + )); + } + taken.quiescence = Some(QuiescenceRequest { + waits: q + .waits + .into_iter() + .map(|w| { + ExternalWaitState::new(w.wait_id, w.generation, w.immediately_parkable) + }) + .collect(), + idle_timeout, + }); + } + WFCommandVariant::ExternalStreamProgress(p) => { + // Ordering is normative, not stylistic. On replay this is what guarantees a + // record's integrity is validated *before* the command derived from it is + // matched; the other way round, a damaged stream would be discovered only after + // its consequences had already been accepted as durable. + if seen_other_command { + return Err(WFMachinesError::Fatal( + "WorkflowStreamProgress must precede every command whose value could \ + depend on the consumed data, but one followed another command" + .to_string(), + )); + } + taken.progress.push(p); + } + WFCommandVariant::ExternalStreamParkResult(p) => taken.park_result = Some(p), + WFCommandVariant::ExternalStreamFinalized(f) => taken.finalized = Some(f), + WFCommandVariant::ExternalOutputStreamCommit(commit) => { + if taken.output_commit.replace(commit).is_some() { + return Err(WFMachinesError::Fatal( + "Lang sent more than one WorkflowOutputStreamCommit in one completion" + .to_string(), + )); + } + } + WFCommandVariant::ExternalOutputStreamBuffered(buffered) => { + let max_publish_latency = buffered + .max_publish_latency + .map(|duration| duration.try_into().unwrap_or(Duration::ZERO)) + .unwrap_or(Duration::ZERO); + if max_publish_latency.is_zero() { + return Err(WFMachinesError::Fatal( + "WorkflowOutputStreamBuffered carried a non-positive max publish latency" + .to_string(), + )); + } + taken.output_buffered_latency = Some( + taken + .output_buffered_latency + .map_or(max_publish_latency, |current| { + current.min(max_publish_latency) + }), + ); + } + _ => { + seen_other_command = true; + remaining.push(command); + } + } + } + *commands = remaining; + Ok(taken) +} + +/// Keeps the idle deadline strictly inside the rollover deadline. +/// +/// Rollover is authoritative: a retained task is bounded by the server's workflow task timeout +/// whatever the idle timeout says, so an idle timer allowed to outlast it would simply never fire. +fn clamp_idle_below_rollover(idle: Duration, wft_timeout: Option) -> Duration { + match wft_timeout { + // The same fraction the rollover deadline uses, pulled in far enough that the idle timer + // still gets a chance to fire first when it was configured to. + Some(wft) => idle.min(wft.mul_f32(WFT_HEARTBEAT_TIMEOUT_FRACTION * 0.9)), + None => idle, } - abort_handle } /// Tracks the heartbeat while a workflow task has outstanding local activities. -struct WaitingOnLAs { +struct LocalActivityHeartbeatState { wft_timeout: Duration, /// Can be used to abort heartbeat timeouts hb_timeout_handle: AbortHandle, @@ -1406,6 +2971,126 @@ struct WaitingOnLAs { /// complete the workflow task. heartbeat_timeout_pending: bool, } + +struct OutputFlushTimer { + deadline: Instant, + handle: AbortHandle, +} + +/// Local work that may retain the open workflow task. +/// +/// Retention used to be a local-activity concept, expressed as `Option` and keyed +/// off `outstanding_local_activity_count()`. The broader per-run concept is *local work that may +/// retain the workflow task*, of which outstanding local activities are one kind and an external +/// stream wait set is another, so the two can be asked the same question without either knowing +/// about the other. +#[derive(Default)] +struct WaitingOnLocalWork { + /// Present while local activities are outstanding, or while a heartbeat is deferred waiting + /// for lang to finish an activation. + local_activities: Option, + /// This run's external stream waits. Empty until lang reports quiescence. + external_wait_set: ExternalWaitSet, + /// Cancels the run-level workflow task rollover deadline, when one is running. + /// + /// Separate from the local-activity heartbeat handle: a retained task needs a rollover + /// deadline whether or not any local activity is outstanding. + wft_rollover_timer: Option, + /// Set when the *deadline* expired. Forces a replacement task -- but writes no marker, because + /// Core decided this boundary and has no terminal for it until finalization supplies one. + deadline_rollover_pending: bool, + /// Set when *lang* asked for a rollover at the annotation byte budget. Also forces a + /// replacement task, and unlike the deadline it may write its marker immediately: the very + /// command that asked for it already carried the terminal. + budget_rollover_pending: bool, + /// Cancels the *global* quiescence timer for the external wait set. + /// + /// One timer for the whole set. Readiness for any member cancels it, which is what makes an + /// idle stream unable to park a workflow task another stream is still driving. + idle_timer: Option, + /// Earliest deadline reported for output buffered in lang but not yet staged. + output_flush_timer: Option, + /// Whether lang still owns unstaged external output for this Workflow Task. + output_buffered: bool, + /// The flush timer expired while a lang activation was outstanding. The completion of that + /// activation queues the finalization job rather than expecting an answer to an unseen job. + output_latency_pending: bool, + /// A losing park's resolve job removed backend intents but did not run Workflow code. Once + /// finalization is durable, its forced replacement must issue the resolve again. + park_rollback_resolve_pending: bool, + /// The boundary a `FinalizeExternalStreams` job is outstanding for. + /// + /// Core is annotation-blind, so it cannot manufacture a terminal. Between issuing the job and + /// receiving `ExternalStreamFinalized` the accumulated annotation is held and **no marker may + /// be written** -- a truncated annotation is durable and wrong, while an abandoned Workflow + /// Task commits no cursor and loses no record. + pending_finalization: Option, + /// The trigger a `PrepareExternalStreamPark` job is outstanding for, as the terminal boundary + /// it will become. + /// + /// Held rather than recomputed, because the reason belongs to the moment parking *started*: an + /// idle expiry and an all-fenced snapshot are indistinguishable by the time the answer comes + /// back, and the marker must say which one it actually was. Its presence is also what stops a + /// second handshake being started for a set already in one. + pending_park: Option, + /// The compact manifest for output already staged by lang but not yet recorded in History. + pending_output_commit: Option, + /// The accepted output-capacity boundary requested an otherwise-empty replacement Workflow + /// Task. That task must enter lang once so it can release the publisher's capacity waiter. + output_capacity_activation_pending: bool, + /// Set when the Run is being torn down -- Worker shutdown or an eviction -- while it still + /// holds a Workflow Task. + /// + /// Like the rollover deadline this forces a replacement task, and for a related reason: the + /// completion is an *offer* of this Run back to the task queue, so that any eligible Worker + /// can pick it up and reconstruct the subscriptions from the marker. Unlike the deadline, the + /// Run does not expect to serve that replacement itself. + shutdown_pending: bool, + /// A query answer held while a `FinalizeExternalStreams` round trip runs. + /// + /// A query response is what refuses retention on an otherwise ordinary quiescent completion, + /// and that refusal makes the boundary Core's rather than lang's -- so lang's report carries + /// no terminal and one has to be asked for. The round trip keeps the Workflow Task open, so + /// nothing has been reported yet, and it runs no user Workflow code, so lang cannot resend the + /// answer. Held here, it rides onto the completion that finally reports the task. + deferred_query_responses: Vec, +} + +impl Drop for WaitingOnLocalWork { + /// A run's external stream deadlines do not outlive the run. + /// + /// Each timer task holds a clone of the local-input sender, so one left running keeps the + /// workflow stream alive for its whole duration -- a Worker shutting down with a retained wait + /// set would sit out the full idle deadline before it could finish, and would then deliver a + /// park handshake to a run that no longer exists. The local-activity heartbeat handle is + /// deliberately *not* aborted here: its callers abort it explicitly and have always relied on + /// dropping the handle being a no-op. + fn drop(&mut self) { + if let Some(handle) = self.idle_timer.take() { + handle.abort(); + } + if let Some(handle) = self.wft_rollover_timer.take() { + handle.abort(); + } + if let Some(timer) = self.output_flush_timer.take() { + timer.handle.abort(); + } + } +} + +impl WaitingOnLocalWork { + /// Whether local work was retaining the task and has now finished. + /// + /// The outstanding local activity count lives on the machines rather than here, so it is + /// passed in. Asking through this method is what keeps the retention decision in one place as + /// more kinds of local work are added, instead of each caller keying off the count directly. + /// + /// Note this is not the negation of "retains": a run that never had local work at all has not + /// "finished", and autocompleting it would report a workflow task nothing was waiting on. + fn finished(&self, outstanding_local_activities: usize) -> bool { + self.local_activities.is_some() && outstanding_local_activities == 0 + } +} #[derive(Debug)] struct CompletionDataForWFT { task_token: TaskToken, @@ -1437,22 +3122,8 @@ impl WorkflowManager { } } - /// Given info that was just obtained from a new WFT from server, pipe it into this workflow's - /// machines. - /// - /// Should only be called when a workflow has caught up on replay (or is just beginning). It - /// will return a workflow activation if one is needed. - fn new_work_from_server( - &mut self, - update: HistoryUpdate, - messages: Vec, - ) -> Result { - self.machines.new_work_from_server(update, messages)?; - self.get_next_activation() - } - /// Update the machines with some events from fetching another page of history. Does *not* - /// attempt to pull the next activation, unlike [Self::new_work_from_server]. + /// attempt to pull the next activation, unlike [Self::get_next_activation]. fn feed_history_from_new_page(&mut self, update: HistoryUpdate) -> Result<()> { self.machines.new_history_from_server(update) } @@ -1648,19 +3319,60 @@ fn sorted(names: &[String]) -> Vec { v } +/// Whether a newly polled WFT for an existing run must be buffered rather than applied. +/// +/// Pulled out of [`ManagedRun::buffer_wft_if_outstanding_work`] because the condition *is* the +/// thing that has to be right: `_incoming_wft` treats a second WFT for one run as a bug and +/// `dbg_panic!`s on it, so every state that means "this run already has a WFT" has to be kept out +/// here. +/// +/// `has_wft` is listed first because it was the one missing. `more_pending_work` is not a stand-in +/// for it -- that is `wft.is_some() && machines.has_pending_jobs()`, which answers `false` for an +/// outstanding WFT whose machines have nothing queued, and that is exactly the state a freshly +/// polled task used to sail through. Buffering instead is safe in a way admitting is not: the +/// buffer is drained on the next run update that finds no pending work, and `has_any_pending_work` +/// counts an outstanding WFT, so the task waits precisely until the one in flight is cleared. +fn must_buffer_wft( + has_wft: bool, + has_activation: bool, + about_to_issue_evict: bool, + more_pending_work: bool, +) -> bool { + has_wft || has_activation || about_to_issue_evict || more_pending_work +} + #[cfg(test)] mod tests { use super::{ - TaskStorageMetrics, log_workflow_task_duration, parse_wft_duration_warn_threshold, + ManagedRun, TaskStorageMetrics, log_workflow_task_duration, must_buffer_wft, + parse_wft_duration_warn_threshold, + }; + use crate::{ + MetricsContext, + abstractions::tests::fixed_size_permit_dealer, + protosext::ValidPollWFTQResponse, + replay::TestHistoryBuilder, + test_help::{ResponseType, hist_to_poll_resp, mock_worker_client, test_worker_cfg}, + worker::{ + WorkflowSlotKind, + client::mocks::DEFAULT_TEST_CAPABILITIES, + workflow::{ + ActivationOrAuto, HistoryPaginator, HistoryUpdate, PermittedWFT, RunBasics, + RunTimerSink, WFCommand, WFCommandVariant, WFTReportStatus, + }, + }, }; - use crate::worker::workflow::{WFCommand, WFCommandVariant}; use std::{ fmt::Write, mem::{Discriminant, discriminant}, sync::{Arc, Mutex}, - time::Duration, + time::{Duration, Instant}, }; - use temporalio_common::protos::coresdk::common::ExternalStorageMetrics; + use temporalio_common::protos::{ + coresdk::{WorkflowSlotInfo, common::ExternalStorageMetrics}, + temporal::api::enums::v1::EventType, + }; + use tokio::sync::mpsc::unbounded_channel; use tracing::{ Event, Level, Metadata, Subscriber, field::{Field, Visit}, @@ -1722,6 +3434,164 @@ mod tests { sub.events.lock().unwrap().drain(..).next() } + async fn permitted_wft( + history: &TestHistoryBuilder, + response_type: ResponseType, + ) -> PermittedWFT { + let valid = ValidPollWFTQResponse::try_from( + hist_to_poll_resp(history, "workflow", response_type).resp, + ) + .unwrap(); + let (paginator, work) = HistoryPaginator::from_poll(valid, Arc::new(mock_worker_client())) + .await + .unwrap(); + let permit = fixed_size_permit_dealer::(1) + .try_acquire_owned() + .unwrap() + .into_used(WorkflowSlotInfo { + workflow_type: work.workflow_type.clone(), + is_sticky: work.is_incremental(), + }); + PermittedWFT { + work, + permit, + paginator, + } + } + + async fn run_with_outstanding_wft_and_replacement() -> (ManagedRun, PermittedWFT) { + let mut history = TestHistoryBuilder::default(); + history.add_by_type(EventType::WorkflowExecutionStarted); + history.add_workflow_task_scheduled_and_started(); + let first = permitted_wft(&history, ResponseType::AllHistory).await; + + history.add_workflow_task_completed(); + history.add_workflow_task_scheduled_and_started(); + let replacement = permitted_wft(&history, ResponseType::OneTask(2)).await; + + let (timer_tx, _timer_rx) = unbounded_channel(); + let (mut run, first_activation) = ManagedRun::new( + RunBasics { + worker_config: Arc::new(test_worker_cfg().build().unwrap()), + workflow_id: "workflow".to_string(), + workflow_type: first.work.workflow_type.clone(), + run_id: history.get_orig_run_id().to_string(), + history: HistoryUpdate::dummy(), + metrics: MetricsContext::no_op(), + capabilities: DEFAULT_TEST_CAPABILITIES, + sdk_name: "test-core", + sdk_version: "0.0.0", + }, + first, + None, + RunTimerSink { tx: timer_tx }, + ); + assert!(matches!( + first_activation, + Some(ActivationOrAuto::LangActivation(_)) + )); + + run.finish_activation(|_| true); + assert!(run.activation().is_none()); + assert!(run.wft().is_some()); + assert!(!run.more_pending_work()); + (run, replacement) + } + + #[test] + fn an_outstanding_wft_alone_buffers_a_newly_polled_task() { + // The regression. No activation, nothing about to evict, and machines with nothing + // queued -- so `more_pending_work` is false -- but a WFT is still outstanding. Admitting + // here reaches `_incoming_wft`, which treats two WFTs for one run as a bug and + // `dbg_panic!`s: the workflow-processing thread dies on a debug build, and a release build + // logs and carries on into code that was never written to hold two. + assert!( + must_buffer_wft(true, false, false, false), + "a run with an outstanding WFT admitted another one" + ); + } + + #[test] + fn a_run_with_nothing_outstanding_admits_the_task() { + // The other half: buffering unconditionally would stall every run, since the buffer is + // only drained by a later run update. + assert!(!must_buffer_wft(false, false, false, false)); + } + + #[test] + fn every_other_kind_of_outstanding_work_still_buffers() { + // These three were already handled and must stay handled -- the fix adds a reason to + // buffer, it does not replace the existing ones. + assert!(must_buffer_wft(false, true, false, false), "activation"); + assert!(must_buffer_wft(false, false, true, false), "pending evict"); + assert!(must_buffer_wft(false, false, false, true), "pending jobs"); + } + + #[tokio::test] + async fn an_outstanding_managed_run_wft_buffers_then_drains_a_polled_task() { + let (mut run, replacement) = run_with_outstanding_wft_and_replacement().await; + + assert!(run.buffer_wft_if_outstanding_work(replacement).is_none()); + assert!(run.has_buffered_wft()); + assert!(run.wft().is_some()); + + assert!( + run.mark_wft_complete( + WFTReportStatus::Reported { + reset_last_started_to: None, + completion_time: Instant::now(), + }, + &TaskStorageMetrics::default(), + ) + .is_some() + ); + assert!(run.wft().is_none()); + + assert!(matches!( + run.check_more_activations(), + Some(ActivationOrAuto::Autocomplete { .. }) + )); + assert!(run.wft().is_some()); + assert!(!run.has_buffered_wft()); + } + + #[cfg(debug_assertions)] + #[tokio::test] + #[should_panic(expected = "Trying to send a new WFT for a run which already has one!")] + async fn bypassing_wft_admission_guard_trips_the_debug_invariant() { + let (mut run, replacement) = run_with_outstanding_wft_and_replacement().await; + + run.incoming_wft(replacement); + } + + #[cfg(not(debug_assertions))] + #[tokio::test] + async fn bypassing_wft_admission_guard_buffers_in_release() { + let (mut run, replacement) = run_with_outstanding_wft_and_replacement().await; + let original_task_token = run.wft().unwrap().info.task_token.clone(); + + assert!(run.incoming_wft(replacement).is_none()); + assert!(run.has_buffered_wft()); + assert_eq!(run.wft().unwrap().info.task_token, original_task_token); + + assert!( + run.mark_wft_complete( + WFTReportStatus::Reported { + reset_last_started_to: None, + completion_time: Instant::now(), + }, + &TaskStorageMetrics::default(), + ) + .is_some() + ); + assert!(matches!( + run.check_more_activations(), + Some(ActivationOrAuto::Autocomplete { .. }) + )); + assert!(run.wft().is_some()); + assert!(!run.has_buffered_wft()); + } + #[test] fn tmprl1104_warns_only_over_threshold() { let none = TaskStorageMetrics::default(); diff --git a/crates/sdk-core/src/worker/workflow/mod.rs b/crates/sdk-core/src/worker/workflow/mod.rs index c0a8d74b9..fe9c96f5b 100644 --- a/crates/sdk-core/src/worker/workflow/mod.rs +++ b/crates/sdk-core/src/worker/workflow/mod.rs @@ -3,6 +3,7 @@ //! a diagram of the internals. mod driven_workflow; +mod external_streams; mod history_update; mod machines; mod managed_run; @@ -11,6 +12,8 @@ mod wft_extraction; pub(crate) mod wft_poller; mod workflow_stream; +pub use external_streams::{ExternalStreamReadyResult, ExternalStreamRunStatus}; + pub(crate) use driven_workflow::DrivenWorkflow; pub(crate) use history_update::HistoryUpdate; @@ -44,7 +47,12 @@ use crate::{ }, }; use anyhow::anyhow; -use futures_util::{Stream, StreamExt, future::abortable, stream, stream::BoxStream}; +use futures_util::{ + Stream, StreamExt, + future::{AbortHandle, abortable}, + stream, + stream::BoxStream, +}; use itertools::Itertools; use prost_types::TimestampError; use std::{ @@ -147,6 +155,12 @@ pub(crate) struct Workflows { pub(crate) struct WorkflowBasics { pub(crate) worker_config: Arc, pub(crate) shutdown_token: CancellationToken, + /// The Worker's *own* shutdown token, of which `shutdown_token` is a child. + /// + /// Cancelled only when shutdown was actually requested, whereas the child is also cancelled + /// by the workflow stream itself when the workflow task poller dies. Both mean "stop", but + /// only this one means "the Worker is going away", which is the boundary C15b closes. + pub(crate) worker_shutdown_token: CancellationToken, pub(crate) metrics: MetricsContext, pub(crate) server_capabilities: get_system_info_response::Capabilities, pub(crate) namespace_capabilities: Arc, @@ -182,6 +196,9 @@ impl Workflows { tracing_sub: Option>, ) -> Self { let (local_tx, local_rx) = unbounded_channel(); + let run_timers = RunTimerSink { + tx: local_tx.clone(), + }; let (fetch_tx, fetch_rx) = unbounded_channel(); let shutdown_tok = basics.shutdown_token.clone(); let task_queue = basics.worker_config.task_queue.clone(); @@ -240,6 +257,7 @@ impl Workflows { extracted_wft_stream, locals_stream, local_activity_request_sink, + run_timers, ); while let Some(output) = stream.next().await { @@ -698,6 +716,141 @@ impl Workflows { }); } + /// Tell a workflow that a record is buffered for one of its external stream waits. + /// + /// Routed like `notify_of_local_result` but **acknowledged**: the caller is a watcher whose + /// next action depends on which result comes back, so a fire-and-forget send would leave it + /// unable to tell "Core will activate" from "send a wake Signal". + pub(super) fn notify_external_stream_ready( + &self, + run_id: impl Into, + wait_id: u32, + wait_generation: u64, + ) -> impl Future { + let (tx, rx) = oneshot::channel(); + self.send_local(ExternalStreamReadyMsg { + run_id: run_id.into(), + wait_id, + wait_generation, + response_tx: tx, + }); + // A dropped sender means the workflow stream is gone, which for the watcher is + // indistinguishable from the run having been evicted -- and calls for the same action. + async move { rx.await.unwrap_or(ExternalStreamReadyResult::RunNotFound) } + } + + /// Ask what state a run's external stream wait set is in, changing nothing. + /// + /// Answered on the same serialized lane as readiness, so the answer is as authoritative as a + /// readiness acknowledgement. Deliberately not the readiness call: readiness means "a record + /// is buffered", and probing with it would assert something false and manufacture a spurious + /// activation during shutdown. + pub(super) fn external_stream_run_status( + &self, + run_id: impl Into, + ) -> impl Future { + let (tx, rx) = oneshot::channel(); + self.send_local(ExternalStreamRunStatusMsg { + run_id: run_id.into(), + response_tx: tx, + }); + async move { rx.await.unwrap_or(ExternalStreamRunStatus::RunNotFound) } + } + + /// Tell a workflow its external stream idle timer expired. + /// + /// Core's own quiescence timer starts calling this once retention exists (C6); until then + /// only tests drive it. + #[allow(dead_code, reason = "driven by the quiescence timer in C6")] + pub(super) fn notify_external_stream_idle_timeout( + &self, + run_id: impl Into, + quiescence_generation: u64, + ) { + self.send_local(ExternalStreamIdleTimeoutMsg { + run_id: run_id.into(), + quiescence_generation, + }); + } + + /// Test-only reach into the output deadline's serialized local input. + #[cfg(test)] + pub(super) fn notify_external_output_flush_deadline(&self, run_id: impl Into) { + self.send_local(LocalInputs::ExternalOutputFlushDeadline(run_id.into())); + } + + /// Test scaffolding -- see [`EmitTerminalLessMarkerMsg`]. + #[cfg(test)] + pub(super) fn emit_terminal_less_external_stream_marker( + &self, + run_id: impl Into, + ) -> impl Future { + let (tx, rx) = oneshot::channel(); + self.send_local(EmitTerminalLessMarkerMsg { + run_id: run_id.into(), + response_tx: tx, + }); + async move { rx.await.unwrap_or(false) } + } + + /// Test scaffolding -- see [`ExternalStreamAnnotationMsg`]. + #[cfg(test)] + pub(super) fn external_stream_annotation( + &self, + run_id: impl Into, + ) -> impl Future> { + let (tx, rx) = oneshot::channel(); + self.send_local(ExternalStreamAnnotationMsg { + run_id: run_id.into(), + response_tx: tx, + }); + async move { rx.await.unwrap_or_default() } + } + + /// Test scaffolding -- see [`StartRolloverTimerMsg`]. + #[cfg(test)] + pub(super) fn start_wft_rollover_timer( + &self, + run_id: impl Into, + wft_timeout: Duration, + ) -> impl Future { + let (tx, rx) = oneshot::channel(); + self.send_local(StartRolloverTimerMsg { + run_id: run_id.into(), + wft_timeout, + response_tx: tx, + }); + async move { + let _ = rx.await; + } + } + + /// Test scaffolding -- see [`ExternalStreamSeedWaitsMsg`]. + #[cfg(test)] + pub(super) fn seed_external_stream_waits( + &self, + run_id: impl Into, + wait_ids: Vec, + parked_at: Option, + wft_open: bool, + ) -> impl Future { + let (tx, rx) = oneshot::channel(); + self.send_local(ExternalStreamSeedWaitsMsg { + run_id: run_id.into(), + wait_ids, + // Long enough that no seeded set parks itself out from under a test. Seeding starts no + // timer, but a wait set carried onto a replacement Workflow Task re-arms one from this + // value, and a set that parked mid-test would be answering a question nobody asked. + idle_timeout: Duration::from_secs(300), + parked_at, + wft_open, + response_tx: tx, + }); + async move { + let _ = rx.await; + } + } + /// Send a `GetStateInfoMsg` to the workflow stream. Can be used to bump the stream if there /// would otherwise be no new inputs. pub(super) fn send_get_state_info_msg(&self) -> oneshot::Receiver { @@ -1277,11 +1430,131 @@ pub(crate) struct HeartbeatTimeoutMsg { pub(crate) run_id: String, pub(crate) span: Span, } + +/// Schedules run-scoped timers **without** going through the local-activity request sink. +/// +/// The sink is `Some` only when `WorkerTaskTypes.enable_local_activities` is set, and lang sets +/// that from whether any activities are registered. A worker registering workflows and no +/// activities therefore has no sink -- and `sink_heartbeat_timeout_start` silently returned a +/// handle to a timer that was never started. That is fine when the only thing a timer bounds is a +/// local activity, and not fine at all once a workflow task can be retained for a reason that has +/// nothing to do with local activities. +/// +/// So the timer facility becomes the run's, and the local-activity heartbeat becomes one caller of +/// it rather than its owner. +#[derive(Clone)] +pub(in crate::worker::workflow) struct RunTimerSink { + tx: UnboundedSender, +} + +impl RunTimerSink { + /// Sends `on_elapse` onto the run's local-input lane at `deadline`. + /// + /// Returns a handle that cancels the timer. Dropping the handle does **not** cancel it -- + /// same as the local-activity heartbeat handle it replaces, whose callers all abort + /// explicitly. + pub(in crate::worker::workflow) fn start( + &self, + deadline: Instant, + on_elapse: LocalInputs, + ) -> AbortHandle { + let (abort_handle, abort_reg) = AbortHandle::new_pair(); + let tx = self.tx.clone(); + let span = Span::current(); + tokio::spawn(futures_util::future::Abortable::new( + async move { + tokio::time::sleep_until(deadline.into()).await; + let _ = tx.send(LocalInput { + input: on_elapse, + span, + }); + }, + abort_reg, + )); + abort_handle + } +} #[derive(Debug)] struct GetStateInfoMsg { response_tx: oneshot::Sender, } +/// A watcher reporting that a record is buffered for one external stream wait. +/// +/// Unlike a local-activity resolution this is **acknowledged**, not fire-and-forget: what the +/// watcher does next -- keep waiting, re-probe, send a wake Signal, or tear itself down -- +/// depends entirely on which of the five results comes back. +#[derive(Debug)] +pub(crate) struct ExternalStreamReadyMsg { + pub(crate) run_id: String, + pub(crate) wait_id: u32, + pub(crate) wait_generation: u64, + pub(crate) response_tx: oneshot::Sender, +} + +/// The global quiescence timer for one run's wait set expired. +#[derive(Debug)] +pub(crate) struct ExternalStreamIdleTimeoutMsg { + pub(crate) run_id: String, + /// The snapshot the timer was started for. A timer that fires for a superseded snapshot is + /// discarded rather than parking a set the Workflow has since run past. + pub(crate) quiescence_generation: u64, +} + +/// The read-only status probe lang's shutdown sweep uses. +#[derive(Debug)] +pub(crate) struct ExternalStreamRunStatusMsg { + pub(crate) run_id: String, + pub(crate) response_tx: oneshot::Sender, +} + +/// Test scaffolding: put a run's wait set into a given state. +/// +/// Registering waits for real is C6's `WorkflowStreamQuiescent` handling and confirming a park is +/// C8's handshake, neither of which exists yet. This rides the same serialized local-input lane +/// as everything else, so a test that uses it still exercises the real routing rather than +/// reaching into `ManagedRun`'s state. +/// Test scaffolding: ask a run to emit a marker with no terminal boundary. +/// +/// No completion path can produce this, which is exactly why the guard needs driving directly. +#[cfg(test)] +#[derive(Debug)] +pub(crate) struct EmitTerminalLessMarkerMsg { + pub(crate) run_id: String, + /// Whether emission refused. `false` would mean a terminal-less annotation was written. + pub(crate) response_tx: oneshot::Sender, +} + +/// Test scaffolding: read a run's accumulated, unwritten replay annotation. +#[cfg(test)] +#[derive(Debug)] +pub(crate) struct ExternalStreamAnnotationMsg { + pub(crate) run_id: String, + pub(crate) response_tx: oneshot::Sender>, +} + +/// Test scaffolding: ask a run to start its workflow task rollover deadline. +#[cfg(test)] +#[derive(Debug)] +pub(crate) struct StartRolloverTimerMsg { + pub(crate) run_id: String, + pub(crate) wft_timeout: Duration, + pub(crate) response_tx: oneshot::Sender<()>, +} + +#[cfg(test)] +#[derive(Debug)] +pub(crate) struct ExternalStreamSeedWaitsMsg { + pub(crate) run_id: String, + pub(crate) wait_ids: Vec, + pub(crate) idle_timeout: Duration, + /// When set, the seeded set is left confirmed-parked at this generation. + pub(crate) parked_at: Option, + /// When false, the set is left with no Workflow Task open. + pub(crate) wft_open: bool, + pub(crate) response_tx: oneshot::Sender<()>, +} + /// Each activation completion produces one of these #[derive(Debug)] struct ActivationCompleteResult { @@ -1573,6 +1846,18 @@ enum WFCommandVariant { UpdateResponse(UpdateResponse), ScheduleNexusOperation(ScheduleNexusOperation), RequestCancelNexusOperation(RequestCancelNexusOperation), + /// Commits an external stream observation delta. Never implies WFT retention. + ExternalStreamProgress(WorkflowStreamProgress), + /// Asks Core to retain the open WFT. Carries no annotation data. + ExternalStreamQuiescent(WorkflowStreamQuiescent), + /// Lang's answer to `PrepareExternalStreamPark`. + ExternalStreamParkResult(ExternalStreamParkResult), + /// Lang's answer to `FinalizeExternalStreams`. + ExternalStreamFinalized(ExternalStreamFinalized), + /// Compact manifest for an external output batch already staged by lang. + ExternalOutputStreamCommit(WorkflowOutputStreamCommit), + /// External output is buffered in lang and needs a run-scoped flush deadline. + ExternalOutputStreamBuffered(WorkflowOutputStreamBuffered), } impl TryFrom for WFCommand { @@ -1634,6 +1919,24 @@ impl TryFrom for WFCommand { workflow_command::Variant::RequestCancelNexusOperation(s) => { WFCommandVariant::RequestCancelNexusOperation(s) } + workflow_command::Variant::WorkflowStreamProgress(s) => { + WFCommandVariant::ExternalStreamProgress(s) + } + workflow_command::Variant::WorkflowStreamQuiescent(s) => { + WFCommandVariant::ExternalStreamQuiescent(s) + } + workflow_command::Variant::ExternalStreamParkResult(s) => { + WFCommandVariant::ExternalStreamParkResult(s) + } + workflow_command::Variant::ExternalStreamFinalized(s) => { + WFCommandVariant::ExternalStreamFinalized(s) + } + workflow_command::Variant::WorkflowOutputStreamCommit(commit) => { + WFCommandVariant::ExternalOutputStreamCommit(commit) + } + workflow_command::Variant::WorkflowOutputStreamBuffered(buffered) => { + WFCommandVariant::ExternalOutputStreamBuffered(buffered) + } }; Ok(Self { variant, @@ -1678,6 +1981,9 @@ enum CommandID { #[derive(Debug, Clone)] struct WorkflowStartedInfo { workflow_task_timeout: Option, + /// Chain identity, not this Run's id. A wake Signal names it so Core can tell "same chain, + /// stale generation" -- harmless -- from "different chain" -- rejected. + first_execution_run_id: String, memo: Option, search_attrs: Option, retry_policy: Option, diff --git a/crates/sdk-core/src/worker/workflow/run_cache.rs b/crates/sdk-core/src/worker/workflow/run_cache.rs index 698bd6a55..6ce31282c 100644 --- a/crates/sdk-core/src/worker/workflow/run_cache.rs +++ b/crates/sdk-core/src/worker/workflow/run_cache.rs @@ -3,6 +3,7 @@ use crate::{ telemetry::metrics::workflow_type, worker::workflow::{ HistoryUpdate, LocalActivityRequestSink, PermittedWFT, RequestEvictMsg, RunBasics, + RunTimerSink, managed_run::{ManagedRun, RunUpdateAct}, }, }; @@ -20,6 +21,7 @@ pub(super) struct RunCache { /// Run id -> Data runs: LruCache, local_activity_request_sink: Option>, + run_timers: RunTimerSink, metrics: MetricsContext, } @@ -30,6 +32,7 @@ impl RunCache { sdk_name_and_version: (String, String), server_capabilities: get_system_info_response::Capabilities, local_activity_request_sink: Option, + run_timers: RunTimerSink, metrics: MetricsContext, ) -> Self { // The cache needs room for at least one run, otherwise we couldn't do anything. In @@ -48,6 +51,7 @@ impl RunCache { ), local_activity_request_sink: local_activity_request_sink .map(|s| Rc::new(s) as Rc), + run_timers, metrics, } } @@ -81,6 +85,7 @@ impl RunCache { }, pwft, self.local_activity_request_sink.clone(), + self.run_timers.clone(), ); if self.runs.push(run_id, mrh).is_some() { panic!("Overflowed run cache! Cache owner is expected to avoid this!"); @@ -137,6 +142,14 @@ impl RunCache { self.runs.iter().map(|(_, v)| v) } + /// Every cached run, mutably, without disturbing the LRU order. + /// + /// Used by the shutdown sweep, which is about reaching *every* run rather than a particular + /// one, and which must not make a run look recently used on the way out. + pub(super) fn handles_mut(&mut self) -> impl Iterator { + self.runs.iter_mut().map(|(_, v)| v) + } + pub(super) fn is_full(&self) -> bool { self.runs.cap().get() == self.runs.len() } diff --git a/crates/sdk-core/src/worker/workflow/workflow_stream.rs b/crates/sdk-core/src/worker/workflow/workflow_stream.rs index 07031e522..3eb91c5f4 100644 --- a/crates/sdk-core/src/worker/workflow/workflow_stream.rs +++ b/crates/sdk-core/src/worker/workflow/workflow_stream.rs @@ -1,3 +1,4 @@ +use super::external_streams::{ExternalStreamReadyResult, ExternalStreamRunStatus}; use crate::{ MetricsContext, abstractions::dbg_panic, @@ -35,6 +36,11 @@ pub(super) struct WFStream { history_fetch_refcounter: Arc, shutdown_token: CancellationToken, + /// Cancelled only when the Worker was actually asked to shut down. `shutdown_token` is a + /// child of this one and is *also* cancelled by this stream when the poller dies, which is + /// why the C15b sweep keys off this token instead: a dead poller is not a Worker going away, + /// and closing every Run's stream boundary on one would hand Runs back that nothing asked for. + worker_shutdown_token: CancellationToken, ignore_evicts_on_shutdown: bool, metrics: MetricsContext, @@ -67,6 +73,7 @@ impl WFStream { wft_stream: impl Stream> + Send + 'static, local_rx: impl Stream + Send + 'static, local_activity_request_sink: Option, + run_timers: RunTimerSink, ) -> impl Stream> { let all_inputs = stream::select_with_strategy( local_rx.map(Into::into), @@ -78,13 +85,14 @@ impl WFStream { // Priority always goes to the local stream |_: &mut ()| PollNext::Left, ); - Self::build_internal(all_inputs, basics, local_activity_request_sink) + Self::build_internal(all_inputs, basics, local_activity_request_sink, run_timers) } fn build_internal( all_inputs: impl Stream, basics: WorkflowBasics, local_activity_request_sink: Option, + run_timers: RunTimerSink, ) -> impl Stream> { let mut state = WFStream { buffered_polls_need_cache_slot: Default::default(), @@ -93,9 +101,11 @@ impl WFStream { (basics.sdk_name.clone(), basics.sdk_version.clone()), basics.server_capabilities, local_activity_request_sink, + run_timers, basics.metrics.clone(), ), shutdown_token: basics.shutdown_token, + worker_shutdown_token: basics.worker_shutdown_token, ignore_evicts_on_shutdown: basics.worker_config.ignore_evicts_on_shutdown, metrics: basics.metrics, runs_needing_fetching: Default::default(), @@ -140,6 +150,12 @@ impl WFStream { LocalInputs::HeartbeatTimeout(hbt) => { state.process_heartbeat_timeout(hbt) } + LocalInputs::WftRolloverDeadline(run_id) => { + state.process_wft_rollover_deadline(run_id) + } + LocalInputs::ExternalOutputFlushDeadline(run_id) => { + state.process_external_output_flush_deadline(run_id) + } LocalInputs::RequestEviction(evict) => { state.request_eviction(evict).into_run_update_resp() } @@ -150,6 +166,63 @@ impl WFStream { }); None } + LocalInputs::ExternalStreamReady(msg) => { + state.external_stream_ready(msg) + } + LocalInputs::ExternalStreamIdleTimeout(msg) => { + state.external_stream_idle_timeout(msg) + } + LocalInputs::ExternalStreamRunStatus(msg) => { + // Answered on this lane precisely so its answer is as + // authoritative as a readiness acknowledgement -- and through + // `peek`, which leaves even the cache's LRU order untouched. + let status = state + .runs + .peek(&msg.run_id) + .map(|rh| rh.external_stream_run_status()) + .unwrap_or(ExternalStreamRunStatus::RunNotFound); + let _ = msg.response_tx.send(status); + None + } + #[cfg(test)] + LocalInputs::EmitTerminalLessMarker(msg) => { + let refused = state + .runs + .get_mut(&msg.run_id) + .map(|rh| rh.emit_terminal_less_marker()) + .unwrap_or(false); + let _ = msg.response_tx.send(refused); + None + } + #[cfg(test)] + LocalInputs::ExternalStreamAnnotation(msg) => { + let annotation = state + .runs + .peek(&msg.run_id) + .map(|rh| rh.external_stream_annotation().to_vec()) + .unwrap_or_default(); + let _ = msg.response_tx.send(annotation); + None + } + #[cfg(test)] + LocalInputs::StartRolloverTimer(msg) => { + if let Some(rh) = state.runs.get_mut(&msg.run_id) { + rh.start_wft_rollover_timer( + std::time::Instant::now(), + msg.wft_timeout, + ); + } + let _ = msg.response_tx.send(()); + None + } + #[cfg(test)] + LocalInputs::ExternalStreamSeedWaits(msg) => { + if let Some(rh) = state.runs.get_mut(&msg.run_id) { + rh.seed_external_wait_set(&msg); + } + let _ = msg.response_tx.send(()); + None + } LocalInputs::BumpStream => { // Ensures pending workflow activation polls resolve for // scenarios like during shutdown @@ -202,6 +275,14 @@ impl WFStream { }; activations.extend(maybe_act); + // C15b -- a Run holding a Workflow Task open on external stream waits is not + // released by anything else: no replacement task is coming once the pollers are + // stopped, and `shutdown_done` counts that open task as pending work. Swept on + // every input rather than once, because a buffered task can still be applied to a + // Run after shutdown began. + if state.worker_shutdown_token.is_cancelled() { + activations.extend(state.external_stream_shutdown_sweep()); + } activations.extend(state.reconcile_buffered()); actions.extend( activations @@ -478,6 +559,48 @@ impl WFStream { } } + fn process_wft_rollover_deadline(&mut self, run_id: String) -> RunUpdateAct { + if let Some(rh) = self.runs.get_mut(&run_id) { + rh.wft_rollover_deadline() + } else { + None + } + } + + fn process_external_output_flush_deadline(&mut self, run_id: String) -> RunUpdateAct { + if let Some(rh) = self.runs.get_mut(&run_id) { + rh.external_output_flush_deadline() + } else { + None + } + } + + /// A watcher reports that a record is buffered for one external stream wait. + /// + /// The acknowledgement is sent from here rather than from the caller because only this lane + /// knows whether the run is cached at all, and `RunNotFound` is a different instruction to the + /// watcher than any of the states a cached run can be in. + fn external_stream_ready(&mut self, msg: ExternalStreamReadyMsg) -> RunUpdateAct { + let Some(rh) = self.runs.get_mut(&msg.run_id) else { + let _ = msg.response_tx.send(ExternalStreamReadyResult::RunNotFound); + return None; + }; + let (result, act) = rh.external_stream_ready(msg.wait_id, msg.wait_generation); + let _ = msg.response_tx.send(result); + act + } + + fn external_stream_idle_timeout(&mut self, msg: ExternalStreamIdleTimeoutMsg) -> RunUpdateAct { + if let Some(rh) = self.runs.get_mut(&msg.run_id) { + rh.external_stream_idle_timeout(msg.quiescence_generation) + } else { + // The run went away while its idle timer was pending. Nothing is retained, so there + // is nothing to park. + debug!(run_id = %msg.run_id, "External stream idle timeout for an untracked run"); + None + } + } + /// Request a workflow eviction. This will (eventually, after replay is done) queue up an /// activation to evict the workflow from the lang side. Workflow will not *actually* be evicted /// until lang replies to that activation @@ -594,6 +717,21 @@ impl WFStream { acts } + /// Closes the external stream boundary of every Run still holding a Workflow Task (C15b). + /// + /// Two transitions, and only one of them exists in any given Run state (ADR-009). This is the + /// first: with a Workflow Task open, Core asks lang for the terminal, writes the marker, and + /// completes requesting a replacement task. The second -- no open Workflow Task -- is + /// deliberately *not* here; nothing is accumulated there, so no marker is missing, and the + /// server-visible replacement is lang's wake sweep, which needs a Signal rather than a + /// completion this Run has no task token for. + fn external_stream_shutdown_sweep(&mut self) -> Vec { + self.runs + .handles_mut() + .filter_map(|rh| rh.external_stream_shutdown()) + .collect() + } + fn shutdown_done(&self) -> bool { if self.shutdown_token.is_cancelled() { if Arc::strong_count(&self.history_fetch_refcounter) > 1 { @@ -679,6 +817,31 @@ pub(super) enum LocalInputs { RequestEviction(RequestEvictMsg), HeartbeatTimeout(String), GetStateInfo(GetStateInfoMsg), + // External Workflow Stream inputs. These ride the same prioritized local-input lane as + // local-activity completions, which is what makes stream readiness serialize with every other + // input to a run rather than racing them. + ExternalStreamReady(ExternalStreamReadyMsg), + ExternalStreamIdleTimeout(ExternalStreamIdleTimeoutMsg), + ExternalStreamRunStatus(ExternalStreamRunStatusMsg), + /// The run-level workflow task rollover deadline expired. + /// + /// Deliberately a separate input from `HeartbeatTimeout` rather than a reuse of it: the + /// heartbeat is meaningful only while local activities are outstanding, and folding the two + /// together would change what a late heartbeat does to a run whose local activities have + /// already finished. + #[from(ignore)] + WftRolloverDeadline(String), + /// The earliest max-publish-latency deadline for buffered external output expired. + #[from(ignore)] + ExternalOutputFlushDeadline(String), + #[cfg(test)] + ExternalStreamSeedWaits(ExternalStreamSeedWaitsMsg), + #[cfg(test)] + StartRolloverTimer(StartRolloverTimerMsg), + #[cfg(test)] + ExternalStreamAnnotation(ExternalStreamAnnotationMsg), + #[cfg(test)] + EmitTerminalLessMarker(EmitTerminalLessMarkerMsg), BumpStream, } impl LocalInputs { @@ -690,6 +853,19 @@ impl LocalInputs { LocalInputs::PostActivation(pa) => &pa.run_id, LocalInputs::RequestEviction(re) => &re.run_id, LocalInputs::HeartbeatTimeout(hb) => hb, + LocalInputs::ExternalStreamReady(r) => &r.run_id, + LocalInputs::ExternalStreamIdleTimeout(t) => &t.run_id, + LocalInputs::ExternalStreamRunStatus(s) => &s.run_id, + LocalInputs::WftRolloverDeadline(run_id) => run_id, + LocalInputs::ExternalOutputFlushDeadline(run_id) => run_id, + #[cfg(test)] + LocalInputs::ExternalStreamSeedWaits(sw) => &sw.run_id, + #[cfg(test)] + LocalInputs::StartRolloverTimer(sr) => &sr.run_id, + #[cfg(test)] + LocalInputs::ExternalStreamAnnotation(a) => &a.run_id, + #[cfg(test)] + LocalInputs::EmitTerminalLessMarker(m) => &m.run_id, LocalInputs::GetStateInfo(_) | LocalInputs::BumpStream => return None, }) } diff --git a/crates/sdk/src/workflow_future.rs b/crates/sdk/src/workflow_future.rs index f18604ce0..e6623868a 100644 --- a/crates/sdk/src/workflow_future.rs +++ b/crates/sdk/src/workflow_future.rs @@ -339,6 +339,15 @@ impl WorkflowFuture { Variant::RemoveFromCache(_) => { unreachable!("Cache removal should happen higher up"); } + // External Workflow Streams are a Python-SDK feature. This SDK never emits + // `WorkflowStreamQuiescent`, so Core has nothing to retain a Workflow Task for + // and cannot produce any of these jobs for it. + Variant::ResolveExternalStreamWaits(_) + | Variant::PrepareExternalStreamPark(_) + | Variant::ReplayExternalStreams(_) + | Variant::FinalizeExternalStreams(_) => { + bail!("External stream activation jobs are not supported by this SDK"); + } } } diff --git a/crates/workflow/src/runtime/instance.rs b/crates/workflow/src/runtime/instance.rs index 7c6b3a033..86caddf0f 100644 --- a/crates/workflow/src/runtime/instance.rs +++ b/crates/workflow/src/runtime/instance.rs @@ -1119,6 +1119,23 @@ where ActivationJobResult::None } Some(ActivationVariant::RemoveFromCache(_)) => ActivationJobResult::None, + // External Workflow Streams are a Python-SDK feature; this runtime never asks + // Core to retain a Workflow Task for a stream wait, so it can only ever see one + // of these by mistake. Failing loudly beats a wildcard that would silently drop + // a job carrying replay-visible state. + Some( + job @ (ActivationVariant::ResolveExternalStreamWaits(_) + | ActivationVariant::PrepareExternalStreamPark(_) + | ActivationVariant::ReplayExternalStreams(_) + | ActivationVariant::FinalizeExternalStreams(_)), + ) => { + return Err(Box::new(Failure { + message: format!( + "External stream activation job {job} is not supported by this SDK" + ), + ..Default::default() + })); + } None => { return Err(Box::new(Failure { message: "Activation job missing variant".to_string(),