Replayed Max's external streams prototype onto the envelope base.

CHANGELOG.md

@@ -344,6 +344,9 @@
 * Rust SDK workers now warn when autoscaling task polling encounters errors continuously for one
   minute. Repeated warnings use exponential backoff up to 15-minute intervals and stop after
   polling recovers.
+* Workflow workers now preserve the outstanding workflow-task token if an internal admission
+  invariant is violated, buffering the replacement task instead of overwriting the task in flight
+  in release builds.
 * Unhandled workflow payload conversion errors now fail the Workflow Task so it can retry instead
   of failing the Workflow Execution. Workflows may still explicitly handle these errors.
 * Workers no longer send worker heartbeats or appear in centralized heartbeat reports before

arch_docs/streaming-poc-docs/README.md

@@ -0,0 +1,94 @@
+---
+doc_id: EWS-DOCS-INDEX
+status: pre-production
+audience: [all]
+---
+
+# External Workflow Streams — documentation map
+
+External Workflow Streams keep high-volume input and output payloads in a pluggable backend rather
+than Temporal History while compact markers preserve deterministic replay. No Temporal Server
+changes are required.
+
+The documentation is deliberately split by reading mode:
+
+| You need | Start here | Authority |
+|---|---|---|
+| A mental model, diagrams, guarantees, and operational tradeoffs | [`guide/`](guide/README.md) | Explanatory; intentionally simplified |
+| Exact implementation behavior and every edge case | [`spec/`](spec/README.md) | Normative for accepted behavior |
+| Why a non-obvious design was chosen | [`decisions/`](decisions/README.md) | Accepted rationale |
+| Candidate or not-yet-accepted behavior | [`proposals/`](proposals/README.md) | Non-normative until promoted |
+| Required observable coverage | [`required-tests/`](required-tests/) | Executable validation contract |
+
+The old [`overview.md`](overview.md) URL remains as a short compatibility landing page.
+
+## Human guide
+
+Read these in order when learning or reviewing the feature:
+
+| Guide | Answers |
+|---|---|
+| [`guide/architecture.md`](guide/architecture.md) | Which components own data, coordination, History, serialization, and provider state? |
+| [`guide/input-lifecycle.md`](guide/input-lifecycle.md) | How does a Workflow Task move through active, quiescent, parking, parked, and no-open-task states? |
+| [`guide/output-commit.md`](guide/output-commit.md) | How does output move from logical buffering through staging and History proof to visibility? |
+| [`guide/replay-and-recovery.md`](guide/replay-and-recovery.md) | What is recorded, how exact-range replay works, and what survives retry, eviction, and Continue-As-New? |
+| [`guide/operations.md`](guide/operations.md) | Which failures retry, which require intervention, and what must a production deployment operate? |
+
+Diagrams explain the shape of the system but never define an edge case by themselves. Each guide
+links to the specification sections that own its exact behavior.
+
+## Normative implementation reference
+
+Use the structured [`spec/` index](spec/README.md) to route by subsystem or stable document ID.
+
+| If you need to… | Read |
+|---|---|
+| Change public names, handle roles, Worker configuration, or feature boundaries | [`spec/public-surface.md`](spec/public-surface.md) |
+| Implement or evaluate a backend provider | [`spec/backend-contract.md`](spec/backend-contract.md) |
+| Change Workflow Task retention, parking, rollover, shutdown, or output arbitration | [`spec/wft-lifecycle.md`](spec/wft-lifecycle.md) |
+| Change replay markers, segmentation, budgets, or continuation state | [`spec/annotation-format.md`](spec/annotation-format.md) |
+| Change proto messages, activation jobs, readiness calls, or Core state | [`spec/core-lang-protocol.md`](spec/core-lang-protocol.md) |
+| Change Worker-side delivery, conversion, merge, close, or staging | [`spec/python-runtime.md`](spec/python-runtime.md) |
+| Change wake encoding, deduplication, acknowledgement, or producer recovery | [`spec/wake-signal.md`](spec/wake-signal.md) |
+| Classify errors, metrics, retries, or operator response | [`spec/failure-taxonomy.md`](spec/failure-taxonomy.md) |
+
+Every normative spec carries machine-readable front matter with a stable `doc_id`, status, audience,
+canonical scope, and related ADRs.
+
+## Proposals and promotion records
+
+Each proposal has two entry points:
+
+- a short overview for deciding whether the direction is desirable; and
+- a detailed design for API, protocol, failure, compatibility, and validation review.
+
+The implemented output direction has been promoted into `spec/`; its detailed proposal remains a
+non-normative rationale record. Workflow-to-Workflow subscriptions remain a future enhancement and
+must not be inferred as available. See [`proposals/README.md`](proposals/README.md) for current status.
+
+## Supporting sets
+
+- [`decisions/`](decisions/README.md) contains 47 current ADRs. A specification says what is true; an
+  ADR says why the rejected alternatives were not chosen.
+- [`required-tests/`](required-tests/) contains the parsed Milestone 1 and Milestone 2 case lists.
+  Editing their counted headings or bullets changes what the validation gate demands.
+- [`verification-hazards.md`](verification-hazards.md) records seven constraints on trustworthy test
+  evidence, including native-extension freshness, repository alignment, sandbox isolation, and
+  controlled fault injection.
+
+Status reports, review rounds, implementation plans, and investigation handoffs do not belong in the
+design set. Their durable conclusions must be promoted into a specification, ADR, required test, or
+proposal.
+
+## Documentation rules
+
+- **One normative fact, one home.** Other specifications link to the owning document. Human guides
+  may summarize it but are explicitly non-normative.
+- **Exact details stay textual.** Every diagram has a nearby explanation and specification link, so
+  no transition exists only as an image.
+- **No line-number references or restatement of local code.** Specifications name stable symbols and
+  files; repository search resolves their current locations.
+- **Every rule explains its failure consequence.** That consequence is what distinguishes a
+  load-bearing invariant from an implementation preference.
+- **Rejected alternatives live in ADRs.** Candidate behavior lives in proposals. Accepted current
+  truth lives in specifications.

arch_docs/streaming-poc-docs/decisions/ADR-001-coexist-with-contrib-workflow-streams.md

@@ -0,0 +1,50 @@
+# ADR-001 — Coexist with `contrib.workflow_streams` rather than replacing it
+
+**Status:** Accepted · **Affects:** P9, P17 · **Spec:** `spec/public-surface.md`
+
+## Context
+
+The Python SDK already ships an experimental `temporalio.contrib.workflow_streams` package
+(`WorkflowStream`, `WorkflowStreamClient`, `TopicHandle`, `WorkflowTopicHandle`), reserving the
+Signal/Update/Query names `__temporal_workflow_stream_publish`, `__temporal_workflow_stream_poll`,
+and `__temporal_workflow_stream_offset`. Shipping a second streaming feature alongside it risks
+two overlapping APIs users must choose between.
+
+## Options
+
+**A. Replace it behind a compatible API.** One public surface; the external-backend
+implementation slots underneath.
+
+**B. Coexist as a separate module.** Two features, two module names, users choose.
+
+**C. Add an external-backend storage mode to the existing package** without changing its public
+surface.
+
+## Decision
+
+**B.** The two features are mirror images, not two implementations of one idea:
+
+| | `contrib.workflow_streams` | External Workflow Streams |
+|---|---|---|
+| Direction | Workflow **produces**; external clients consume | Both external-input and external-output directions |
+| Workflow-side API | `topic().publish()`; deliberately no workflow-side `subscribe()` | Input topics `subscribe()`; output topics `publish()` and `finish()` on distinct handles |
+| Storage | Append-only log in Workflow state, i.e. in History | Pluggable external backend; payloads never enter History |
+| Offsets | Dense `int`, exposed in `subscribe(from_offset: int)` and the offset Query | Opaque, provider-defined, totally ordered tokens (Redis IDs are `<ms>-<seq>`) |
+| Transport | Signals, long-poll Updates, SSE bridge | Direct backend reads; Signals carry no payload and supply only wakeups |
+
+A is not feasible: there is no consumption API in the contrib feature to be compatible *with*, and
+its `int` offset surface is incompatible with opaque provider offsets even for the directions that
+overlap.
+
+C stays open as future work but is explicitly not a dependency of this design — it requires solving
+the dense-integer-offset mapping. External Workflow Streams instead include a distinct output
+direction with opaque client cursors; its separation of handle types preserves this decision.
+
+## Consequences
+
+- Both features ship; neither is deprecated by this work.
+- This feature takes distinct names: module `temporalio.contrib.external_workflow_streams`, entry
+  points `external_stream` and `external_output_stream`, reserved Signal
+  `__temporal_external_stream_wake`.
+- **No name in this feature may begin with `__temporal_workflow_stream`.** P9's completion criteria
+  assert this.

arch_docs/streaming-poc-docs/decisions/ADR-002-cursor-is-a-position-boundary.md

@@ -0,0 +1,47 @@
+# ADR-002 — A cursor is a position boundary, not a record identity
+
+**Status:** Accepted · **Affects:** P2, P3, P5 · **Spec:** `spec/backend-contract.md`
+
+## Context
+
+A subscription must persist "where I got to" so a later Run, or a replay, resumes correctly. The
+representation has to work for a consumer sitting at the current tail, where the next record does
+not exist yet.
+
+## Options
+
+**A. The cursor names the next record to deliver.** Intuitive; resume is "read from the cursor".
+
+**B. The cursor is a position boundary** — `BEGINNING | AFTER(last_consumed_offset)`.
+
+## Decision
+
+**B.**
+
+A is not implementable. After a consumer reads the current tail, the next record does not exist and
+therefore has no offset. Redis offers no way to name the ID the next `XADD` will produce, so a
+consumer parked at the tail has nothing to persist as a next-to-deliver cursor.
+
+```text
+cursor := BEGINNING
+        | AFTER(last_consumed_offset)
+```
+
+`BEGINNING` is the provider's beginning-of-stream boundary and need not be the offset of any real
+record. `AFTER(x)` names the boundary immediately following the record at offset `x`, whether or not
+a record after `x` exists yet.
+
+A provider may represent the boundary as `(offset, inclusive | exclusive)` instead. What it may not
+do is require the cursor token to be the offset of a record that does not exist yet.
+
+## Consequences
+
+- Two distinct provider primitives, not one: an **exclusive** watch strictly after a boundary
+  (Redis `XREAD BLOCK`, or from sentinel `0-0` for `BEGINNING`), and an **inclusive** range read over
+  an explicit `[first, last]` pair (Redis `XRANGE`).
+- **Replay never reads "from the cursor".** It reads explicit recorded ranges; the answer is already
+  in the marker.
+- Offsets are compared by the provider's ordering rule, not lexically. Redis IDs compare as numeric
+  `(ms, seq)` tuples — string comparison breaks as soon as the millisecond component changes width.
+- The conformance suite must contain a case that parks a consumer at the tail, appends a record whose
+  ID could not have been predicted, and resumes; a backend requiring a nameable next ID fails it.

arch_docs/streaming-poc-docs/decisions/ADR-003-structural-immutability-is-required.md

@@ -0,0 +1,50 @@
+# ADR-003 — Structural immutability is required of every provider
+
+**Status:** Accepted · **Affects:** P2, P17, P5, P13 · **Spec:** `spec/backend-contract.md`
+
+## Context
+
+Replay re-reads records from the backend rather than from History. It must detect damage to those
+records — otherwise a Workflow silently observes different data on replay than it did live. The
+question is what class of damage the design must detect, and what that detection costs per record.
+
+## Options
+
+**A. Require every provider to guarantee stored bytes cannot change.** Replay then validates
+presence, count, order, and control positions only.
+
+**B. Make it optional, with a per-record content-hash mode** for providers that cannot make the
+guarantee. Replay validates hashes for those providers.
+
+**C. Detect nothing**; trust the backend entirely.
+
+## Decision
+
+**A.** Structural immutability is a **registration-time requirement**, checked when a backend is
+registered on the Worker (P17), not a per-provider runtime mode. A backend that cannot make the
+guarantee does not satisfy the contract and is rejected before any Workflow can name it.
+
+Redis Streams is the worked example and satisfies it: an entry can be deleted by `XDEL` or removed by
+trimming, but its fields cannot be rewritten in place.
+
+B costs one hash per record in every marker — the single encoding in the whole design that would grow
+with item count — to defend against a failure mode no supported backend exhibits. Requiring the
+guarantee up front is both cheaper and stricter than making it optional and paying to detect its
+absence. C gives up the feature's determinism claim.
+
+## Consequences
+
+- Given the guarantee, replay needs to detect exactly one class of damage — **a record that is no
+  longer there** — and offsets suffice. Deletion, trimming, and retention expiry are the realistic
+  operational failure modes, and all are caught by the four range checks in
+  `spec/annotation-format.md`.
+- **Nothing in the annotation encoding is per-record.** A run costs two offsets, a count, and a sparse
+  control list whether it covers ten records or a hundred thousand. This is what makes the
+  single-stream marker-size claim unqualified.
+- **Accepted cost, stated plainly:** if a provider silently violates immutability — a buggy custom
+  backend, or out-of-band surgery on the stream — replay delivers the altered bytes as though they
+  were original, and no error is raised. The risk is bounded by the registration requirement and the
+  conformance suite.
+- `schema_version` leads the annotation encoding, so a content-hash mode can be introduced later
+  without a format break if a provider ever needs one.
+- A decode failure is therefore never an integrity failure — see ADR-015.

arch_docs/streaming-poc-docs/decisions/ADR-004-progress-and-quiescence-are-separate-commands.md

@@ -0,0 +1,41 @@
+# ADR-004 — Progress and quiescence are separate commands
+
+**Status:** Accepted · **Affects:** C1, C6, C14a, P10a, P10b · **Spec:** `spec/wft-lifecycle.md`
+
+## Context
+
+At every activation return the runtime may need to do two things: commit what the Workflow observed,
+and ask Core to hold the Workflow Task open. A single command carrying both is the obvious encoding.
+
+## Options
+
+**A. One command** that reports quiescence and carries the consumed records.
+
+**B. Two commands** — `WorkflowStreamProgress` (commits an observation delta) and
+`WorkflowStreamQuiescent` (asks for retention).
+
+## Decision
+
+**B.** The two questions are independent, and conflating them loses progress on every path where the
+answers differ:
+
+1. *Did replay-visible stream state change?* → `WorkflowStreamProgress`, on **every** completion
+   path.
+2. *Should the Workflow Task be retained?* → `WorkflowStreamQuiescent`, which starts the idle timer.
+
+Under A, an activation that both consumed records **and** produced a server-bound or terminal command
+is not quiescent, so it emits nothing — and the consumption is never committed. See ADR-005 for why
+that is silently fatal.
+
+A completion may carry either command, both, or neither. `WorkflowStreamProgress` never implies
+retention, and `WorkflowStreamQuiescent` carries no annotation data.
+
+## Consequences
+
+- Two command tags (23 and 24) rather than one.
+- Retention applies only when external stream waits are the sole reason the activation cannot
+  progress. If the completion also carries server-bound commands, Python emits progress but not
+  quiescence, and the wakeup comes from a watcher instead — see `spec/wft-lifecycle.md`.
+- **Command ordering is normative:** `WorkflowStreamProgress` precedes every command whose value could
+  depend on consumed data, so replay validates a record before matching the command derived from it.
+- Core accumulates successive deltas for one Workflow Task and writes them as exactly one marker.

arch_docs/streaming-poc-docs/decisions/ADR-005-progress-is-an-observation-delta.md

@@ -0,0 +1,48 @@
+# ADR-005 — Progress is an observation delta, emitted on every completion path
+
+**Status:** Accepted · **Affects:** C14a, P5, P10b · **Spec:** `spec/annotation-format.md`
+
+## Context
+
+`WorkflowStreamProgress` commits what replay must reproduce. The emission rule decides which
+activations produce one.
+
+## Options
+
+**A. Emit when records were consumed.** The natural reading of "progress".
+
+**B. Emit on quiescence only.** Fewer commands; one delta per park.
+
+**C. Emit whenever replay-visible state changed** — an observation delta.
+
+## Decision
+
+**C.**
+
+A loses the empty-stream case entirely. A Workflow that subscribes to an empty stream, becomes
+quiescent, and parks has consumed nothing, so it emits no delta, so its marker carries no annotation
+— and replay then has no provider identity, no stream key, no starting cursor boundary, and no
+recorded no-data boundary, leaving it nothing to work from but live backend state, which this design
+forbids.
+
+B loses progress on every non-quiescent completion path, which is the failure ADR-004 exists to
+prevent.
+
+> Emit a `WorkflowStreamProgress` delta whenever the activation changed anything replay must
+> reproduce. That includes deliveries, but also the first observation of a subscription, an
+> activation that observed no records, and the boundary on which the activation returned.
+
+Progress must reach the marker on **every** Workflow Task completion path that consumed external
+data: normal completion; a completion carrying server-bound commands; a terminal command; parking
+after quiescence; and rollover.
+
+## Consequences
+
+- The **first** delta for any subscription always carries provider identity, provider format version,
+  stream key, and explicit starting cursor boundary, whether or not a record has ever been delivered.
+- Every activation that ran a drain contributes a segment, **even an empty one**. The annotation
+  grammar is `segment*` and `run*`, not `+`, and both empty cases must round-trip through the codec.
+- The failure this rules out is specific and silent: if a consumed record influences a command that
+  lands in History while the consumption is never marked, replay re-reads that record from the last
+  committed cursor and delivers it again, while the command it produced is already durable. The
+  divergence surfaces as an unrelated nondeterminism error much later.