Repository navigation
Conversation
Design documentation for adding External Workflow Streams to the Temporal
Python SDK: high-volume stream payloads live in a pluggable external backend
(Redis Streams first), never in Temporal History, while deterministic replay
is preserved with compact per-Workflow-Task markers. No Temporal Server
changes are required.
Organized by subsystem so a single deliverable needs one or two files rather
than the whole corpus:
overview.md what the feature is, API sketch, naming, non-goals
spec/ backend contract, Workflow Task lifecycle, annotation format,
Core/lang protocol, Python runtime, wake Signal, failure
taxonomy, and a single code-anchors table holding every
file-and-line reference
plan/ 47 deliverables across three milestones, a 12-stage schedule,
and the 55 + 12 required-test lists
decisions/ 25 records, each stating the options considered and why the
choice was made
Conventions the tree holds to: one fact has one home; line numbers live only
in spec/code-anchors.md so a Core rebase updates one table; specs state
current truth and rejected alternatives live in decisions/.
The plan's dependency graph is machine-checked. Deliverable dependencies,
milestone membership, and the stage schedule are declared in a parseable form
and validated by tools/check_plan_graph.py in the task directory, which fails
on seven structural conditions: duplicate IDs, unknown dependencies, undeclared
milestone members, deliverables in no or two milestones, cycles, milestones not
closed over their dependencies, and a schedule that omits a deliverable or
places one beside or before a dependency.
This commit carries documentation only; no Core source is modified.
Stage 2 of the External Workflow Streams plan: C1, C2, and C5. C1 -- protos only, zero behavior. Adds the four completion commands (WorkflowStreamProgress, WorkflowStreamQuiescent, ExternalStreamParkResult, ExternalStreamFinalized) on tags 23-26 plus the shared ExternalStreamWait; the four activation jobs on tags 17-20; ExternalStreamMarkerData, ExternalWaitMarker, and ParkReason in external_data; and the WakeSignal envelope in a new coresdk.external_stream package. ParkReason lives in exactly one place -- the Core-readable marker envelope -- so it cannot disagree with a second copy inside the opaque annotation. The new oneof variants make several matches non-exhaustive. Each is filled in explicitly rather than with a wildcard: the machines reject an external stream command that reaches them, because reaching them silently would drop a replay-visible observation delta, and the two Rust SDKs reject the activation jobs, which they can only ever see by mistake. C2 -- ExternalWaitSet/ExternalWaitState as plain types with the transitions as pure functions, so both orderings of the park/readiness race are testable with no worker running. Parking is all-or-nothing across the complete set, so a confirmation is stale if *any* wait left Parking, not only if all of them did. C5 -- pure refactor. WaitingOnLAs becomes LocalActivityHeartbeatState nested inside WaitingOnLocalWork, and the retention predicate moves off outstanding_local_activity_count() into a named method, so an external wait set can answer the same question later without either kind of local work knowing about the other. All 46 local-activity and heartbeat tests stay green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… timers Stage 3's Core half: C3, C4, and C13. C3 -- three external stream inputs on the prioritized local-input lane local activity completions already use, so stream readiness serializes with every other input to a run rather than racing them, plus the read-only status probe's own input. C4 -- notify_external_stream_ready and external_stream_run_status on Worker. The first is acknowledged rather than fire-and-forget because the caller is a watcher whose next action depends entirely on which of five results comes back; in particular NoOpenWorkflowTask and RunNotFound stay separate, since a run cached between Workflow Tasks is healthy and reporting it as missing would tell the watcher to tear itself down while it is still needed. The status probe is answered through the cache's `peek`, so asking leaves even the LRU order alone. C13 -- the gap ADR-017 names. The rollover deadline was scheduled by pushing into the local-activity request sink, which exists only when enable_local_activities is set, so `sink_heartbeat_timeout_start` silently returned a handle to a timer that was never started on a workflow-only worker -- exactly the worker external streams must support. The timer facility becomes the run's, and the local-activity heartbeat becomes one caller of it. The rollover deadline gets its own local input rather than reusing HeartbeatTimeout, so a late heartbeat still does nothing to a run whose local activities have finished. All 46 local-activity and heartbeat tests stay green. Registering waits for real is C6's and confirming a park is C8's, so the tests seed those states through a cfg(test) input that rides the same serialized lane -- the routing under test is real even where the state is planted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lang's WorkflowStreamQuiescent is consumed above the state machines, which is also what makes "no server-bound command accompanies the completion" checkable: whatever is left after the split *is* the server-bound set. When nothing is left, the task is retained with ActivationCompleteOutcome::DoNothing and both run-level deadlines start. One idle timer for the whole wait set, not one per subscription: the timeout measures global quiescence, so an idle stream cannot park a task another stream is still driving. It is clamped strictly inside the rollover deadline, since rollover is authoritative and a longer idle timer would simply never fire. A non-positive idle timeout is rejected as a malformed completion rather than coerced -- zero would park instantly and absent would hold the task until it timed out, and neither is something a caller can have meant. Not retaining cancels both timers, so a snapshot that no longer holds a task open cannot leave a timer behind to fire against a later one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ries state Stage 5's Core half: C7, C14a, C12a. C7 -- readiness cancels the idle timer and queues one ResolveExternalStreamWaits. Coalescing lives in _check_more_activations rather than at the notification, so readiness that arrived while an activation was outstanding is picked up the moment that activation completes, with no second path to keep in step. The rollover deadline is deliberately *not* cancelled: it bounds the task itself, which is still open. Writing the tests turned up a real bug. become_quiescent cleared every pending readiness, so a notification accepted between lang computing its snapshot and Core processing the completion was dropped -- stranding a buffered record until the idle timer or a producer's Signal. Readiness now carries the wait_generation it was accepted at and survives a snapshot reporting that same generation: same generation means lang has not re-blocked and never saw the record, a bumped one means it did. That is precisely what the generation is for. C14a -- deltas accumulate on every completion path, retained or not, because consuming a record and committing that consumption are separate steps and the second is not conditional on why the task ended. An empty delta accumulates like any other. Progress ordered after another command is rejected as malformed: on replay that ordering is what guarantees a record's integrity is validated before the command derived from it is matched. C12a -- rollover transport. A replacement task arrives with lang never having been activated, so it cannot re-request retention; Core carries it across instead along with every subscription, cursor, and readiness generation. Autocompleting the replacement would report it straight back and undo the rollover it was created for. No marker is written -- that is C12b's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Signal is intercepted by name before user dispatch and its envelope decoded without a DataConverter -- Core is the component that has to read it, and a user codec that encrypts payloads would make it unreadable to exactly that component. Suppressed from user handlers **whether or not it validates**, so an unknown envelope version or a stale generation can never surface in Workflow code as an unhandled Signal. Decoding and classification are split: the machines decode and suppress, `ManagedRun` classifies against the wait set, because only the run has one. park_generation 0 is the unparked wake and is accepted as a recheck request after chain validation -- an unnecessary one costs at most one empty Workflow Task. A *non-zero* generation the Run does not recognise is a claim that turned out to be wrong, so it is ignored. Chain identity is compared strictly, including when this Run's chain id is unknown: a Signal naming a chain we cannot confirm is ours is precisely the case that must not be honoured. A validated wake marks *every* active wait ready, not only the one the Signal names. The Signal's stream is a hint; a wake for a stream that turns out to have nothing costs one empty drain, and missing one costs a stalled Workflow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…C14b) C9 -- the marker state machine, modeled on the local-activity one and for the same reason: a marker written live must be *matched* by its MarkerRecorded event on replay. Exactly one machine per Workflow Task, however many progress reports that task carried; accumulating the annotation stays the wait set's job. Core cannot check the annotation -- it is opaque by design -- so the quiescence generation is the only thing it *can* check against history. Checking it is what turns a reordered marker into a nondeterminism error rather than a silently different stream result, and writing the tests proved it works: a canned history with the wrong generation was caught immediately. C14b -- the emission primitive. It refuses to write an annotation with no terminal, a refusal that belongs to emission itself and needs no finalization job to state it. Integrated here only on the three paths where lang's own progress command carried the terminal; the Core-decided boundaries obtain one first. Two ordering facts the tests forced out: - The marker is emitted *before* lang's commands enter the machines, so it precedes the terminal command in History. Emitting after put it second, and on replay the command would then be matched before the record it came from was validated. - Core generates the marker after lang's own iteration has already prepared its commands, so emission prepares its own. Without that the marker sat in the task's queue and never reached the server -- silently, since nothing else looks there. Deadline-driven and budget-driven rollover are now separate flags. Both force a replacement task; only the budget one may write a marker, because only it has a terminal. The deadline's marker waits for the finalization round trip (C15a) and its integration (C12b), which is exactly what makes C12a the half that can ship without markers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…12b) C15a -- the protocol primitive, independent of which boundary triggers it. When Core decides a boundary with no lang activation outstanding it issues FinalizeExternalStreams, accepts ExternalStreamFinalized, and hands the finalized annotation to the emitter. It never manufactures a terminal: if one cannot be obtained, no marker is written and the Workflow Task fails for retry. An abandoned task commits no cursor and loses no record, while a truncated annotation is durable and wrong. The task is held open (DoNothing) while a finalization is outstanding, so it cannot complete before its terminal exists. A finalization still pending when a new task arrives belongs to a task that failed, so it is cleared -- otherwise the next ordinary completion would look like a lang protocol violation. C12b -- the durable half of rollover. A deadline that expires with no activation outstanding finalizes first, writes exactly one marker carrying the returned terminal, and only then completes with force_new_wft. Budget rollover keeps its shortcut and is now asserted to issue no finalization job at all. Two real bugs the work surfaced: - A pending rollover was swallowed by a retention request. A completion that both asked to be retained and had a deadline pending would retain *and restart the deadline*, holding the task past the very timeout the deadline exists to stay inside. Rollover is authoritative and now overrides retention. - The rollover intent was lost across the finalization round trip: the flag is consumed by the completion that asks, so the completion that finally writes the marker would forget to request a replacement. Where the spec was silent -- what happens when the deadline fires while a lang activation is outstanding -- finalization is keyed off the terminal being unobtainable rather than off timer state. If lang's own completion closed the boundary its progress already carried the terminal; only when nothing supplied one does Core ask. That keeps "Core never writes a marker for a boundary it decided without a terminal" exact without adding a second rule. Verified independently: bypassing the finalization round trip makes three of the new tests fail, so they assert the mechanism rather than the outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…(C8, C10) C8 — the complete-set park handshake. The idle timer expiring, or every wait reporting immediately parkable, moves the set to Parking and asks lang for a park result; a confirmation accumulates the final observation delta, writes exactly one marker, and completes the task, while an abort marks every wait ready and retains the task so the resolve activation has somewhere to land. Parking is all-or-nothing: a set holding an accepted readiness refuses, which is what keeps the abort path reachable at all. The park job is paired against an outstanding-job state rather than a strict Some/None, because the all-fenced trigger fires from inside a completion and the idle timer can expire while lang is working -- a strict pairing would fail a completion for a job lang had not yet been handed. A park job answered without a result is refused, applying to PrepareExternalStreamPark the strict-response rule the protocol states for FinalizeExternalStreams: ADR-008 requires it either way, since a park answered without a result leaves Core with a boundary it decided and no terminal. resolve_park no longer clears wft_open on confirmation. The task *is* still open at that instant -- Core writes the park's marker on it and only then completes it -- and clearing it there made take_annotation refuse the very annotation the confirmation had just terminalised. C10 — the replay marker lookahead. External-stream markers are now scanned ahead of the Workflow Task like local-activity markers, each producing one ReplayExternalStreams activation carrying the generation, waits, opaque annotation, and terminal boundary. Machines are matched FIFO because an external-stream marker has no lang sequence number and its quiescence generation is not unique -- a task that never quiesced writes 0. A marker with no machine expecting it is nondeterminism, reported as such rather than as the generic "no command scheduled for event". The machine constructor is split in two: recording a marker live issues a command, resolving one from lookahead does not. The old single constructor pushed a command on the replay path too, so prepare_commands would have driven an invalid transition out of WaitingResolveFromMarkerLookAhead -- unreachable until this deliverable, and removed by construction rather than guarded. Emission now refuses to write while replaying: the marker is already in History, and a second command would mismatch the event it was read from. Also fixes a timer lifetime bug found on the way. RunTimerSink::start spawns a task holding a clone of the local-input sender, and nothing aborted those handles when a run was evicted, so a live timer kept the workflow stream open for its full duration. A Worker shutting down with a retained wait set sat out the entire idle deadline and then delivered a park handshake to a dead run. WaitingOnLocalWork now aborts its idle and rollover timers on drop. Removes the out-of-band park-result input lane C3 stubbed. Lang answers with the ExternalStreamParkResult completion command, which is what the protocol specifies and what the Python side already sends. 80 external-stream tests pass, 471 in sdk-core; clippy is clean apart from two pre-existing warnings in temporalio-client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two teardown transitions, kept apart because they use different mechanisms and only one of them exists in each Run state (ADR-009). With a Workflow Task open -- retained by the wait set, or open with an unfinished activation -- the Run cannot simply disappear: anything accumulated needs lang's terminal before a marker may be written. Teardown therefore asks for it through C15a's finalization protocol, writes the marker through C14b's emitter, and completes with a replacement task requested. When nothing was accumulated there is no marker to write and none is missing, but the task is still completed rather than abandoned open, because completing is the only way to offer this Run back to the task queue. With no open Workflow Task nothing is accumulated, so no marker is written and no completion is made. The server-visible replacement is P20's Python-side wake sweep, which is already in place: `force_new_wft` needs a task token this Run does not have. The shutdown boundary rides across the finalization round trip the same way the rollover intent does, because the completion that *asked* consumed the flag -- without that the completion which finally writes the marker would forget to request a replacement task. Teardown outranks a pending rollover deadline when both are due: the replacement task the deadline wanted is the same one teardown asks for, and one boundary gets one marker. 85 external-stream tests and 476 in sdk-core; clippy clean apart from the two pre-existing warnings in temporalio-client. Verified by mutation: disabling the finalization primitive fails exactly the teardown tests -- shutdown with a task open, and eviction finalizing before the eviction activation -- along with the three earlier tests that depend on the same primitive. An earlier mutation of one call site alone changed nothing, because the transition has two entry points and the tests reach it through the other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects were reported during implementation against code that turned out to be correct. Both had the same shape -- the apparatus was broken, not the system -- and both are cheap to check for once named, so they are written down rather than remembered. A stale native extension. `maturin develop` run from `temporalio/bridge/` builds that crate's own package into the virtualenv, while the SDK imports `temporalio.bridge.temporal_sdk_bridge` from the package directory. The command succeeds and prints Installed; nothing imports the result. A stale extension does not error, it is quietly correct for the old code, so the suite stays green and only the newest tests disagree -- which reads as those tests having found a bug. Here it produced a nondeterminism error naming the replay marker lookahead, in a Core built before the lookahead existed. A test helper that restarts a producer sequence. `(session_id, sequence)` is the append idempotency key and it is idempotent on identity, so a helper that restarts its count re-uses a key with different content and the backend rejects it -- correctly, and that rejection is a required conformance case. The record is never appended, the Workflow waits for data that will never arrive, and the exception surfaces in the producer several lines away from the Workflow that is now stuck. It presented as a livelock between retention and quiescence. Two test modules grew the same helper independently and both were wrong the same way. The document ends with what to confirm before reporting a defect a test found: that the binary under test is the one you built, that the test's own setup succeeded, that it reproduces at the smallest layer showing it, and that mutating the suspected code path actually fails the test -- if nothing fails, the test was not covering what you thought. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An activation runs under a 2-second deadlock timeout. ADR-011 keeps I/O off the Workflow thread, which bounds how long any single step of a drain takes and says nothing about how many steps there are. Delivery volume was the unbounded direction. The Workflow thread drains the subscription's buffer while the watcher refills it concurrently from the Worker's loop, so a producer that never lets the buffer run dry produces an iterator that never blocks and an activation that never ends. Measured against the real iterator: 316,086 records consumed in two seconds, blocking zero times. The Workflow Task then fails on the deadlock timeout and every retry does the same -- a healthy producer, a healthy backend, and a Workflow that never progresses. Bounding by elapsed time would be nondeterministic: how many records fit in a time slice depends on machine speed, record size, converter cost, and load, and because segment boundaries are recorded in the annotation, replay would divide the same records differently and fire `wait_condition` predicates a different number of times. Bounding by the buffer size does not bound an activation at all -- the buffer bounds memory held ahead of delivery, not the records that pass through it, and "drain until empty" is a condition a keeping-pace producer never reaches. Workflow Task rollover is no help either: it bounds the Task, is decided between activations, and cannot interrupt one in progress. So delivery is bounded by a fixed record count, and the subscription blocks when the budget is exhausted even though records are still buffered, because that block is what ends the activation. Re-arming readiness for those buffered records is part of the decision rather than an optimization: the watcher has already advanced past them and no further notification is coming, so a budget that blocked without re-arming would trade a deadlock timeout for a silent stall -- which is worse, because the stall reports nothing. The segment that exhausts the budget ends with `BATCH_LIMIT`, which is what that segment_end_reason has always meant and had no producer until now. Also corrects two statements this makes wrong: that draining continues while records are available, which is true only across activations rather than within one; and the backpressure rule, which bounds memory held ahead of delivery rather than how much one activation delivers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…red cases A continuously fed stream never rolled over. `begin_external_stream_quiescence` re-armed the rollover deadline from `Instant::now()`, and becoming quiescent again is precisely what a delivered record does -- so every record pushed the deadline out and it never fired. The idle timer is clamped below it and could not fire either, so the retained task ran until the *server* timed it out. That is the exact workload rollover exists for. The deadline is now anchored at the Workflow Task's start, the way `restart_external_stream_deadlines` already anchors it: what it bounds is the task, not the wait it was armed for. The dbg_panic seen after that timeout is a consequence rather than a separate defect -- it needs a second Workflow Task to arrive for a run whose previous task Core still holds, which only happens once the server has timed the retained task out. With rollover working, Core completes at 80% of the timeout and the sequence does not arise. A residual that rollover cannot close remains, since rollover bounds the task rather than an activation; making it non-fatal is a design decision, not a bug fix, and is left alone. Eight cases of the Milestone 1 required list close. Worth naming individually because each covers a completion path that had no test at all: a bare `WorkflowStreamProgress` completion; an Activity command with the marker ordered before it and its record delivered exactly once on replay; the fail and continue-as-new terminals, where a missing marker hands the next Run a stale cursor; the finalization-ownership table asserted as a table rather than as ten separate paths; a budget-driven split asserted in bytes; a second Worker reconstructing a subscription from the bytes and wait list the first one wrote on its way out; the invariant that an unwritten annotation exists only while a Workflow Task is open; and a pending timer suppressing retention without taking the wait set with it. Fifteen mutations were run against the source, each killing the intended test. Three were caught by nothing but the new tests -- a `TaskCompleted` terminal turned into `None`, a marker's wait list emptied, and a wait set cleared only on completions carrying server-bound commands -- which is the direct evidence that these gaps were real rather than bookkeeping. 96 external-stream tests and 487 in sdk-core; clippy clean apart from the two pre-existing warnings in temporalio-client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wake-Signal spec says an unparked wake's request ID includes the sender's identity and a per-sender counter precisely so that two Workers shutting down at different times derive different IDs and both wakes are delivered. The implementation used the *client* identity, and two Workers in one process share a Client while each one's counter restarts at 1 -- so their first unparked wakes derived byte-identical request IDs, the server deduplicated the second, and the Run the surviving Worker had picked up never got a Workflow Task. The mechanism built to prevent silent loss was producing it. The sender identity is now drawn once per manager and held for its lifetime. Both halves matter and both are tested: two Workers must differ, or a wake is lost; and one sender's retry must not, or the shutdown sweep's in-grace retry becomes a second wake rather than a resolution of the first. The value is random, which is safe here because the wake Signal is sent from the Worker's own event loop and never from Workflow code, so it is not replay-visible. The client identity rides along as a prefix so a request ID stays traceable to a client in server-side logs. Parked wakes are untouched and must stay that way: they are identified by their generation, so every producer racing to wake one generation still derives the same ID and the server keeps one. Verified by mutation in both directions. Sharing the identity between Workers fails three tests including the two-Workers case; redrawing it per wake at the call site fails the retry case. An earlier mutation of the generator function itself changed nothing and proved nothing -- it is called once at construction, so the property under test is that the manager holds the value, not what the function returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`get_wf_activation` derives `is_replaying` from the job list it drains, and an empty list satisfies "every job is a query" vacuously, so an activation built with no jobs is flagged replaying. The reserved wake Signal is suppressed from user handlers and produces no job of its own, and the resolve job was appended after the build -- so a live replacement Workflow Task reached lang marked as replay. Every link after that behaved exactly as designed, which is what made it hard to see: lang correctly reports neither stream progress nor quiescence while replaying, so the wait generation never advanced, so every later readiness report was answered Stale, and the watcher owes a Signal only on the answers meaning local readiness could not be delivered. The Run went quiet with records buffered in the Worker. The first link was a lie and the rest were faithful to it. Fixed by applying the history, classifying wakes, and only then building the activation, so the flag is computed over a job list that reflects the work being sent. With no wakes the sequence is byte-for-byte what it was. The tempting fix -- guarding the vacuous case with `!jobs.is_empty()` -- is wrong, and this is worth recording. The vacuous `true` is load-bearing upstream: the ReadyForQueries path builds an empty-job activation and stuffs the queries in afterwards, relying on exactly that to keep query-only activations marked replaying, as the comment two lines above the computation says it intends. Under that change a bare query on a caught-up run would be shipped to lang as live work *and* would take the branch that re-stamps the run's deployment version. A speculative build in `get_next_activation`, whose result is discarded, would mutate the same state before the next Workflow Task is applied. So this is a latent trap upstream rather than a live defect: no upstream path sends an empty activation to lang, and any future caller that queues a Core-generated job after the build inherits a silent replay flag. Ours was the first to do it. Two tests, over identical history: an intercepted wake Signal's activation is not flagged replaying, and an ordinary user Signal on the same history is not either. The second is the control -- if it ever fails, the history stopped being live work and the first is measuring nothing. 98 external-stream tests, 489 in sdk-core, no regressions on the run that would have exposed the rejected fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Workflow that started a timer and first blocked on a stream in the same activation was permanently unresumable. One activation reached lang, then Workflow Tasks started and completed with no activation at all while wake Signals sat in History doing nothing. Registering and retaining were the same act. `become_quiescent` -- the only writer of the wait set -- sat inside `if will_retain`, and `will_retain` is false whenever the completion carries server-bound commands. So a completion carrying a timer registered nothing, `mark_all_ready_for_wake` had nothing to mark, and every wake produced an empty activation. The spec has always said these are separate: a completion carrying server-bound commands must not request retention *while the subscriptions stay registered*, with the wake Signal covering the window that leaves. Suppressing retention was right; losing the registration was not. Sending the quiescent command anyway does not help -- Core accepts it and ignores the registration -- which was established by trying it rather than by reading, and is why this needed a Core change at all. Two consequences of the separation are handled here too. Lang must now report its quiescent snapshot while replaying, because the wait set is per-Worker runtime state a replayed Run rebuilds from nothing; Core registers those waits and arms no timer, since a replay slower than the idle timeout would otherwise queue a park handshake between replay activations. And a query response riding along no longer commits an annotation with no terminal frame: query responses are deferred so the terminal is asked for first, which ADR-008 requires and which lang cannot do for itself, since it never learns a query was there. Three tests, each dying under mutation and only the intended one: welding registration back to retention kills the server-bound-command and replaying- quiescence tests; treating a query response as absent kills the query test. 101 external-stream tests, 492 in sdk-core; clippy and fmt clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both documents said the manager probes each Run's state after Core stops polling. Core has no lane left to answer on by then. `initiate_shutdown()` itself ends the workflow-state lane -- `bump_stream()` pushes an input, `shutdown_done()` sees the shutdown token cancelled and an idle cached Run with no pending work, and the stream returns `PollError::ShutDown` -- so `external_stream_run_status` falls through to RunNotFound. The same Run answers NoOpenWorkflowTask one line before that call and RunNotFound one line after. What made this survive review is that the resulting sweep still looks correct: RunNotFound owes a wake exactly as NoOpenWorkflowTask does, so the wake still goes out. It is the two answers meaning *do not send one* that stop being reachable -- a parked Run gets a wake it does not need, and a Run holding a Workflow Task gets one that races Core's own shutdown transition. So the sweep is in two halves at two points, and the documents now say why neither can move. Probing happens before Core's shutdown is initiated, the last moment an answer exists. Acting on those answers happens after the pollers have stopped and every activation has been answered, because per-Run teardown is driven by RemoveFromCache and a finalization in flight must be answered before the manager's state disappears -- and because a wake sent while this Worker is still polling offers the Run to a queue this Worker will answer itself, which is not a hand-off. The trade is stated rather than hidden: the probe no longer carries the guarantee that no Run can acquire a Workflow Task after it, so a Run recorded NoOpenWorkflowTask may pick one up before the pollers stop and receive both C15b's replacement task and the sweep's wake. That costs one extra empty Workflow Task, which this design permits, and the ordering that would prevent it buys the guarantee with an answer that describes nothing. P20's done-when gains the criterion that already guards it: the sweep's probe must answer NoOpenWorkflowTask rather than RunNotFound, which is what shows it ran early enough to distinguish anything at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The implementation is complete and its comments carry the reasoning, so most of what these documents held is now said twice. They keep only what no file can own: invariants spanning Core, lang and the provider, rules a reader needs before opening any one file, and the shape of the lifecycle as a whole. The plan is gone -- deliverables, milestones, schedule, dependency graph. It described work that is finished. The two required-test lists survive it because they are not prose: the gate parses their headings and bullets at test time, so they move to `required-tests/` where that is what they look like. `code-anchors.md` is gone too. Line numbers cannot be maintained and grep is never wrong about them. `python-runtime.md` drops the register/prefetch/drain protocol table, the rules list, and the finalization section, each of which sat verbatim on the constant or method it described. `wft-lifecycle.md` drops the rollover narrative and six failure-semantics restatements. The cut line throughout: if one function's own comment already says it, it goes. Three things the implementation does that no document stated: Registering a wait set is separate from retaining the Workflow Task, and only the second is refusable. A completion carrying server-bound commands cannot ask for retention but is no less blocked, and Core is the only place that set is recorded -- so it registers the waits and arms nothing. Welding the two together leaves a Workflow that starts a timer and first blocks on a stream in the same activation registered nowhere, unresumable by any wake: a permanent deadlock reachable from ordinary user code. The shutdown sweep is two halves at two points, and `python-runtime.md` now states the three obligations that are the runtime's own rather than repeating the sequence `wft-lifecycle.md` owns. A park intent exists only while its park is outstanding. Both halves are obligations: the consumer removes the intent when the park ends, and the provider reports no generation once it is removed. A provider answering from a remembered last generation breaks both readers identically and invisibly -- producers name a generation Core discards as stale, and the shutdown sweep sends a parked wake the server deduplicates against the one that ended that park. Also corrects three claims the implementation contradicted: there are four cursor positions rather than three, and the row describing delivery described consumption; the finalization-ownership table was missing the query-refused- retention boundary, which owes a terminal under ADR-008; and the parking operations were listed as five where the interface requires seven, without `parked_wait_ids`, which a producer cannot address a wake without. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The one thing a reviewer needs that no other document here provides: which commits are in scope, across two repositories, and in what order to read them. It is marked as a review artifact rather than design, because it is the only file in this set that carries history. Everything else states what is true now; this one is a snapshot stamped with the two commits it was written against, and stale as soon as anything else lands. It also says where to look hardest, which is the part worth arguing with. The four areas named are the ones where two concepts the specification treats as separate were fused in the implementation, each half correct alone -- registering a wait set versus retaining the Workflow Task, the annotation's terminal versus its header, a parked wake's request ID versus an unparked one's, and probing a Run's state versus acting on the answer. Those are worth re-deriving from the specification rather than reading for plausibility, since every one of them read as plausible while wrong. What is not finished is stated too: rollover bounds the Workflow Task rather than an activation, so an activation outliving the task timeout still reaches a dbg_panic in Core, and making that non-fatal is a design decision rather than a bug fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It named the repositories but not where they are, and listed the documents to read without linking them. A reviewer given this one path can now reach everything from it: the two checkouts on disk, which of them the build compiles, and every document in reading order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fifteen findings, four of them P0, each with the code it concerns and a test that would demonstrate it. Recorded verbatim before any triage, so what the review said is not confused with what was later agreed or disputed. Two spot-checked immediately and hold exactly as written: readiness treats a Stale acknowledgement as delivered, with a comment claiming a re-probe that cannot happen once prefetch has moved past the record; and merge() drains one subscription to exhaustion before touching the next, so a saturated low wait id starves every stream above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The annotation grammar was still the shape it had before bindings moved per wait, and said nothing about a subscription created after the header goes out. That binding rides its own frame, because Core appends observation deltas and never rewrites what it holds, so a header cannot be extended in place. The grammar, the merging rule, bound-exactly-once, and the late wait's own start cursor are all stated, with ADR-027 carrying the alternative that was rejected: re-emitting a header frame would leave a decoder meeting two of them to decide whether the later replaces the earlier. A segment's delivery list is the recorded global order across waits, and a replay drain takes from the front only while the front belongs to the wait that is asking. ADR-018 gains the distinction that makes both halves necessary: segmentation reproduces how many drains an activation ran, and the order within a segment reproduces which wait each of those drains served. A park intent needs two enforcement points to hold its invariant, not one. The intent is durable backend state while the record of which Worker installed it is not, so besides the resolve there is a reconciliation when a subscription is registered -- and that is sound because registering happens as user Workflow code runs, and no user code runs inside a park, so an intent found there belongs to a park that is over. A provider's key derivation must be injective. Identity components are user-chosen strings, so joining them without escaping maps two unrelated Workflows onto one physical key: they read each other's records, and each concludes the other's claim has taken responsibility for the wake, so neither signals and both wait forever on records already durable. Three documents overstated what a lease buys, saying it prevents a producer crashing between claim and Signal from stranding the generation. Expiry permits takeover; it schedules nobody to take over. What prevents the stranding is that a producer losing the claim sends the wake anyway, which costs almost nothing because a parked wake's request id ignores sender identity and the server collapses the duplicates. A granted claim saves a round trip, not a wakeup. The parking handshake no longer installs an intent for every subscription the Worker holds; the set is the one Core's job names, and a failed attempt withdraws every intent it installed. The review guide now carries a status against each of the fifteen findings, with the reviewer's own text untouched. Thirteen are fixed. Two remain open: payload decoding still running on the Workflow thread, and the failure taxonomy not reaching the activation failure path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…etion `python-runtime.md` claimed the Workflow thread only pops from a bounded buffer, which is only true if popping is the whole of getting a value -- and decoding is not. It now states the split that makes the claim hold: `DataConverter.decode` is three steps, only the last needs the topic's declared type, and that one is synchronous, so the asynchronous half runs on the Worker's loop and the type stays where the topic declared it. Running the asynchronous half on the Workflow thread does not merely block; the deterministic event loop turns a codec's await into a real Temporal timer command, so History comes to depend on a codec's internals. ADR-028 carries the alternatives. Sending the whole decode back to the Worker costs a Workflow Task per record and cannot serve replay at all, since a marker's segments are delivered inside one activation and there is no readiness round trip between them. Blocking the Workflow thread on the Worker's loop turns an await into a guaranteed deadlock-timeout failure and only moves which thread waits. Borrowing the converter through the runtime handle makes the Workflow-facing module depend on the manager's internals rather than on the handle's named methods. `failure-taxonomy.md` describes four rows kept apart because an operator does something different with each; it now says what makes that real. Reporting inspects the failed completion rather than an exception, because a delivery raises on the Workflow thread and the activation converts it before the async layer ever sees it -- keying on an exception would drop most of two rows. Classification happens at both halves of the split, and both delivery paths carry a validated range, so both land on the decode row rather than on integrity. The counter names were wrong in five documents, and the metrics table was missing the storage row entirely, so "three failing rows, three counters" was never actually stated anywhere. The review guide closes at fifteen of fifteen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The docs carried a deliberate carve-out: closing tidied the Workflow side and left the Worker's watcher prefetching and the park intent installed, and that gap was recorded as a gap rather than written up as design. It is closed (sdk-python 88c3578d, guarded in 8c2ff85b), so the carve-out goes and the specs state what closing is. python-runtime.md gains the section: which half is synchronous and which is handed to the manager, that undelivered records are dropped without being consumed so a successor receives them rather than stepping over them, and the single property closing changes permanently -- the wait can never block again, so Core can never be asked to retain a Workflow Task on behalf of a coroutine that is gone. wft-lifecycle.md gains the third point at which a park intent is taken back. ADR-029 records keeping the subscription's state and marking it closed. Deleting it is what the alternatives make expensive: an unbound wait replays as a wait the Workflow never created, against code that never changed, and a successor restarts the stream from the beginning. Keeping the binding and cursor somewhere else instead splits every reader of the registered set in two, and a reader that consults only the live half is wrong solely after a close -- rare, silent, and durable once it reaches a marker. ADR-030 records removing the intent in the cancel path rather than leaving it to the paths that would otherwise reach it. Neither can: the resolve iterates subscriptions cancellation has already dropped, and the reconciliation runs at registration, which a closed wait never sees again. The cost of getting this wrong is not a leak but a lost wake -- the record is appended, the Signal sent and acknowledged, and Core discards the wake as stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The specs described the paths these fixes changed, and described them as they were meant to work rather than as they did. They now state what holds. wft-lifecycle gains the owed removal: three moments decide a removal, and a ledger belonging to the Run carries one the backend did not confirm, drained under that Run's park lock by whatever reaches it next. The drain's match rule is there with its reason -- a stream key outlives Continue-As-New while wait ids restart at one -- and so is the admission that reading before removing narrows that window without closing it across Runs. backend-contract gains the requirement that makes the rule implementable: an intent must read back unmodified, because a delayed removal identifies its target by what it holds. annotation-format gains what replay checks about the Workflow's own code, and loses the sentence saying detection is not best-effort. That claim rested on a branch no marker this runtime writes can reach, and one that could not have done the job in any case: both its maps come from the annotation, so it compares an annotation against itself and never against the Workflow. The guarantee is real now, and it is stated with the case it does not cover -- an insertion among bindings that compare equal, which no check can catch without making an addition at the end illegal. python-runtime gains merge's ordering and the argument for why a rotating start does not disturb replay, including the premise the argument rests on; and gains the serialization context both halves now bind, per holder, with the reason each binds where it does. failure-taxonomy loses a sentence claiming row four reports in both directions, which was the same overstatement as the annotation one. Five records: an owed removal belongs to the Run, rollback catches cancellation and is not shielded, replay checks bound waits one-directionally, merge rotates its start position, and both sides of a stream bind the Workflow's context. Each carries what was rejected -- including, for the last, fixing only the consumer, which reads like the smaller change and breaks every deployment that currently works. ADR-030 is amended where this work falsified it: a failed removal is no longer only a log line. review-guide no longer says all fifteen are fixed. It points at the follow-up review, which now records all seven closed and names what remains open: the insertion case above, and a producer that stores with no target. Covers sdk-python 2763a25c. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records the third review round and updates the specs it changed. The findings themselves are in third-review.md; what the specs gained is the reasoning that outlives them: - annotation-format.md: why a fractional high-water mark is not a bound, and the four rules that make one -- reserved closing frames, delivery that stops before an unrecordable segment, a rollover request that rides the completion which crossed the line, and a refusal at subscribe(). Also why a zero-segment marker closes before the activation's drain while every other marker closes after it, and why a failing activation abandons a replay rather than closing it. - python-runtime.md: why the reposition after a replay is synchronous, and what makes that safe from the Workflow thread -- the epoch check moving inside the append being the half that is easy to miss. Plus the second delivery budget, which is the annotation's. - wake-signal.md: that all three steps after a durable append are inside publish()'s acknowledged-wake contract, and the two recoveries, which are not the same. - wft-lifecycle.md: that the shutdown sweep counts subscriptions rather than attempts, and the three cases it deliberately does not count. - failure-taxonomy.md: that an unreachable external payload store is row one, and that nothing labels it unless the preparation step does. - ADR-007 and ADR-018 gain the consequences the fixes established. empty-stream-replay-flake-handoff.md arrives with its own resolution: its two findings are the review's first two, all six of its required fix properties hold, its test-harness race is fixed, and its two proposed deterministic tests are cases 70-71. It also carries three corrections to its own analysis -- the zero-segment case needing a fix of its own, the bare `finally` masking real errors, and the flake being pre-existing rather than produced by the findings it names. That last one took three baseline full-suite runs with the fixes stashed to establish. Milestone 1's required-test list grows from 55 to 71 cases. New cases are appended rather than filed under the sections they belong to, because the gate maps cases by position and inserting one silently repoints every mapping after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fourth round's five findings are written up in fourth-review.md, with ADR-036 (cancellation after a durable append) and ADR-037 (a subscription has one consumer) carrying the reasoning for the two that needed a new decision. Reviewing those fixes then found one more P1 in this round's own code, and it is the reason for ADR-038. The cancellation fix closed the window after append() returned and kept the assumption underneath the old behaviour: that an append() which did not return had not happened. A remote backend commits on its own side and only then answers, so that is false -- cancelled or disconnected in between, the producer said nothing about a record that may already be durable, and both caller moves were wrong. The append's acknowledgement window is now its own outcome, carrying the exact record, with a recovery that re-appends it and a stream that refuses the publish that would duplicate it. spec/wake-signal.md states it; ADR-038 records why probing the backend and auto-resolving were rejected instead. Case 77 of required-tests/tests-m1.md is its test set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Validating ADR-038's recovery found three ways it could still duplicate or misroute a record. One diagnosis under all three: the producer kept the unsettled record when what it owed was the unsettled operation, and the recovery checked the record's session id when what makes a re-append a no-op is the whole of (stream, session, sequence, bytes). A refusal raised while an append was unsettled described the refused call rather than the unsettled one, so following its own instructions dropped a wake the older record owed. A record does not name its own stream and idempotency is scoped per stream, so the same record settled against another topic appended a second copy there and left the first stream blocked. And the session check passed on a replacement producer whose sequence and wake counters are back at zero, where settling leaves the next publish reusing a sequence number and the next unparked wake reusing a request ID. Recovery is now in-process, on the topic the error names, for the exact bytes it carried, and it defaults its wake and lease to what the interrupted call was doing. A producer that is gone recovers the way an Activity retry already does. Case 78 of required-tests/tests-m1.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A re-interrupted `resolve_append()` is now the interrupted operation: its effective wake and lease replace the earlier attempt's in the single retained entry, while cancellation accumulates, because once delivered it is still owed after every later failure. `spec/wake-signal.md` and ADR-038 state that rule; `fourth-review.md` records the defect it closes as case 79, and `tests-m1.md` carries the case, which moves the declared count to 79 and the partition to 91. `tests-m2.md` said 76 where `tests-m1.md` said 78, so that count is corrected too. `repeated-recovery-fix-validation.md` is the validation record for that fix: how the defect was reproduced by mutation-testing `_remember` back to its pre-fix body, why the replace-wake/accumulate-cancellation asymmetry holds, and two residual items — one docstring since corrected, one latent matching asymmetry in `_forget` that is unreachable today. It also records why the Milestone 1 gate stayed green while case 79 was unarmed: `m1_gate.py` parses these lists out of the vendored Core checkout, and nothing there fails on a mapping beyond the declared count. Case 78's paragraph said the first raise and every later refusal report the same recovery, which case 79 refines rather than keeps: they report the same *canonical* recovery, which a deliberate override may change between attempts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five files were back-and-forth between agents rather than deliverables: four `WORKFLOW_STREAMING_*.md` in the task directory and this set's own `repeated-recovery-fix-validation.md`. Their findings were already recorded -- the six older ones in `third-review.md`, cases 72-79 in `fourth-review.md` -- so what needed a home was the residue, not the findings. Three things moved: - **A third verification hazard.** The m1 gate parses the required-test lists out of the *vendored* Core checkout, so a case added here is not armed until the submodule pointer moves -- and nothing catches a mapping for a case the parsed list lacks, because the count check compares that list against its own heading. The gate stays green while checking nothing, which is the inverse of the two hazards already recorded and the more dangerous direction. - **Case 79's structural cause.** `_append` raised from its own local object while `_remember` kept the older one, so the error and the retained state were two objects for one operation. The fix is that they are now one, not a rule about which fields win. Its mutation test also showed why the reviews before it missed the defect: the raised error was internally consistent with itself, because `resolve_append()` had already defaulted its `wake` from the stale entry before the error was built out of those defaults. - **The `_forget` key-vs-record mismatch** went to that function's own docstring, where a latent code-level detail belongs under this set's conventions. `AppendNotAcknowledgedError`'s canonical-recovery wording was corrected in place when it was found, and the teardown warning the last validation noted is fixed in the Python repo rather than recorded.
The specs had said nothing about the last two rounds of fixes. Four facts were missing and one was wrong. The Continue-As-New header carries the whole binding a cursor is a position in, not just the cursor and a stream name, because a successor's first live read happens before it has written a marker that could make the comparison marker replay makes for every recorded range. Restoration splits a mismatch the way the annotation's binding table does — stream or backend name is nondeterminism, provider identity or format version is a storage failure — and the format version is compared by map membership, since the provider contract reserves no value and zero is one a provider may declare. That header is also snapshotted where no Workflow code can still run rather than where the command is created, which is now stated with what it prevents: the event loop keeps draining ready tasks after a terminal command is added, those consumptions do reach the predecessor's final marker, and a header taken earlier hands the successor a record twice. The write fence's claim is enforced rather than merely documented, and the spec now carries the three outcomes an earlier publish can have and what each does to the fence — including that resolve_append() is what decides between the durable and the refused case, and that a fence must read that decision rather than infer it from the record no longer being outstanding. A wait a marker bound is converted with a converter built from its recorded binding, not the runtime's own. Those are the same identity on a live Worker and are not the same under an offline Replayer. ADR-039 and ADR-040 record the alternatives, which is where the one wrong fact was. The document set claimed a Continue-As-New header could stay readable by an older Worker and carry the binding in a trailing extension. It cannot: syntactic compatibility is not semantic compatibility, and an old Worker that parses the header still runs the restoration it shipped with, accepts a cursor it cannot vouch for, and skips every record below the boundary if its registry resolves that backend name elsewhere. The binding is must-understand data and the schema version is what enforces it; the mixed-fleet cost is accepted and stated, in the direction ADR-014 takes everywhere else in this feature. fifth-review.md records nine findings across three passes, four of them closed in 5ab10ac5 and five in the commit this documents. The review index stopped at the fourth round while two more had happened. review-guide.md is restamped, with its change surface, commit lists, test counts and gate figures recomputed; it remains a snapshot and is stale again the moment this lands. Every file, symbol, ADR reference and link in the set resolves. Still not armed: none of the nine cases is in required-tests/tests-m1.md, so the 79/79 figure does not cover them, and arming one needs the vendored submodule pointer moved before the gate can see it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR-039 said `Continuation.schema_version` was what pinned a writer behind its readers. The field carries the choice, but nothing supplied it: a release that is its own first v2 reader and an unconditional v2 writer has no safe first step, so the reader stage the decision rests on did not exist. Name the two quantities separately, name the Worker setting and the constant that hold them, and record the two consequences the staging actually commits to: a version 1 writer stage restores cursors with the binding checks skipped, and the stages are ordered because a chain that has read version 2 must not be written back down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge section said what a pass takes and in what order, and nothing about the blocked set. The asymmetry it left unstated is a trap: a readiness activation resolves every pending future rather than only the waits it named, so a group wait always ends together, while at most one member comes out of it holding a record. Only that one is unblocked by taking it, so unblocking members individually leaves the rest marked blocked with their futures already discarded -- waits the quiescent snapshot still names, and Core retains and eventually parks a Workflow Task for. A caller that takes one value out of a merge and goes on to wait for a Signal has that event delayed behind the phantom wait until the idle timeout. Also why the resulting `wait_generation` bump on every member is paid for rather than avoided: an in-flight readiness report under the previous generation is answered `Stale`, and `rearm_readiness()` on every completion is what re-reports it. Suppressing the bump buys that coverage back by keeping the phantom wait. Found by review in sdk-python and fixed there; this is the statement the specs were missing, not a new decision. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Accepted` is the one readiness result that leaves nobody but Core holding the record: the watcher does nothing further and will not report that record again. Core returned it whenever a Workflow Task was open, and a task is open until `mark_wft_complete` -- which runs only once the server has answered the completion. Every completion that reports rather than retains sits inside that window: a replaying one, one carrying a server-bound command, one answering a query. Each registers the wait set lang described and reports the task, and neither path turns pending readiness into a resolve job, because `_check_more_activations` is not on either of them. Readiness accepted anywhere in that window was therefore stranded. Core held it in `ready`, no activation was issued for it, the task closed, and the watcher owed no Signal. The record stayed buffered behind a Run that Core believed it had already told, with nothing in the system waiting for anything: the Workflow blocked, the Worker idle, the server holding no task to schedule. Both halves are fixed in `prepare_complete_resp`, the single funnel every reported completion passes through. Local delivery ends when the completion is prepared rather than when the server answers it, so readiness arriving after that point is answered `NoOpenWorkflowTask` and the watcher sends the wake Signal itself. Readiness already accepted has no such fallback, so the completion asks for the replacement task the resolve job is issued on -- `apply_new_wft` re-opens the wait set and `_check_more_activations` ships it. This is the empty-stream replay flake, whose resumed Run walks straight into the window: the records are published while it is evicted, so the watcher its replayed `subscribe()` starts finds them already in the stream and reports readiness during the very activation whose completion registers the wait set. Measured on `test_replay_end_to_end.py` run whole, which is what makes it fail: without this, 2 of 3 then 1 of 3 runs; with it and the sdk-python wake-counter fix it needs, 18 of 18 clean. The case passes 8/8 on its own, and Core DEBUG logging suppresses it entirely -- the window is small enough that the logger's own writes close it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The handoff document had a resolution that fixed two real defects and not this flake -- as its own baseline measurement said, in a paragraph that has been the most useful thing in the file since. Replace the front of it with the root cause, keep that round below unchanged, and say plainly which evidence closed it: the readiness answer in the test's own stall diagnosis, the activation trace showing nothing after the replay job, and the four measured rates. ADR-041 records the decision the fix rests on, including the option not taken. Deciding retention from the machines' state *after* a completion is applied is the better shape -- `replaying` is sampled before lang's commands reach the machines, so it describes the activation and not its outcome -- but it changes what a Run of this shape writes to History, which is a versioned change, and it would not cover a completion carrying a server-bound command anyway. `spec/core-lang-protocol.md` gains the invariant the table implied and never stated: `Accepted` is a promise, so Core stops making it the moment a completion is prepared for the server, and keeps the ones already made on a replacement task. `spec/wake-signal.md` gains the scope of the wake counter. It already called it "a per-sender monotonic counter"; what was missing is that the sender outlives the Run, so a counter held on the subscription restarts at 1 when an evicted Run is rebuilt and re-derives the request ID that Run already used. The server deduplicates the wake and the Run stalls on records it is holding. Fixed in sdk-python; this is the sentence that would have caught it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR-031 chose the per-Run ledger and deferred the background retry task, for two stated reasons: the retry needed an owner that teardown awaits, and it has the widest read-to-delete window of any drain, which the generation check narrowed but did not close. Both prerequisites are now met, so the decision records B and C together rather than B with C deferred. `remove_park_intent_if_matches` becomes a provider obligation: compare the recorded Run ID and park generation and delete the intent and its claim in one operation. That is what makes a delayed cleanup safe against a Continue-As-New successor installing its own intent at the same `(stream key, wait_id)` -- a process-local park lock cannot order a predecessor's drain against a successor's install, because the two Runs may not even be held by the same Worker. With the window closed, the ledger gets its own retry: one manager-owned task per Run, capped exponential backoff, cancelled and awaited inside the Worker's shutdown grace. Lifecycle drains stay as eager fast paths, but they are no longer the only liveness mechanism -- a backend that recovers while the Run sits cached and idle is now enough to retire a stale intent, which is exactly the Run that never parks, resolves, registers or evicts again. `follow-up-review.md` and `review-guide.md` close the two owed-removal items their status lines had left open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six things the autonomous retry left unsaid, all of them about the difference between a mechanism and its best case. **Bounding.** Counting attempts does not bound an attempt that never returns. A provider call that hangs rather than raises leaves the retry loop stopped where it is -- the one thing left holding that removal -- and holding the Run's park lock while it waits, which the next park, resolve or eviction needs. So every hold of that lock taken *off* the activation path is bounded in time, and a hold that runs out is a failed attempt like any other. Holds an activation takes itself stay unbounded on purpose: that wait is the backend exposure the park handshake already owns, and bounding it would move the wait rather than remove it. **Owning.** Shutdown cancels the retry loops, and a loop cancelled during its backoff still had an attempt in it -- the failure that started the backoff is not the state of the backend now. Eviction used to be that last attempt and cannot be while shutting down, because it would wait on the lock a stuck loop may hold. The last attempt is a bounded pass after the loops are cancelled, and what survives it is logged: an intent still installed as the process exits suppresses the unparked wake for whoever picks the Run up next. Discovery needs the same ownership as the removal. The ledger records an intent's identity, so a read that never succeeds records nothing at all, and the reconciliation used to give up with no entry, no task and no owner -- while saying an autonomous retry was continuing. It now keeps retrying while this Worker holds the subscription, and hands over to the ledger the moment a read succeeds and only the removal fails. The subscription is what bounds it, and that bound is load-bearing: removing what sits at that key is justified by holding the Run with the wait registered on it. **Announcing.** Retiring an intent is not delivering what it silenced. Wakes sent while it was installed named the generation behind it, and Core discards a non-zero generation that is not the park it holds -- so the record from that window is buffered on the Worker with its wake counted as sent and its prefetch cursor already past it. Every path that retires an intent now announces a wait still holding records, the reconciliation's first-time success included: that is the *ordinary* path for an inherited intent, it races the watcher that sends the doomed wake, and it never goes near the ledger. Which means the provider cannot answer with a Boolean. A delete that commits and loses its reply leaves the retry meeting a key it cleared itself, so *absent* and *mismatch* are separate outcomes: the first ends the suppression and may owe an announcement, the second is a park to leave alone. Collapsing them costs a record, not an intent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ledger moved the *removal* off the Subscription and left the *wake* on it. Both halves of the announcement read the Subscription -- a buffer to check, a readiness channel to use -- and eviction and close deliberately drop that object while keeping the ledger and its retry alive. So the removal a recovered backend finally lets through is frequently the one with nothing left to announce through, and conditioning the announcement on a registered wait kept the leak fixed while putting the record loss straight back. With the wait gone the wake is unconditional, and that is a choice rather than an oversight. Whether a remote producer appended during the suppression window was never visible from this Worker, so "a record arrived" and "none did" are the same observation from out here -- and only one of them is silent if the guess goes the wrong way. One empty Workflow Task is the cheaper side of that trade. The other half is what a wake names. An owed removal says the installed generation is one Core has already discarded, so composing a wake from it produces a Signal the service accepts and Core ignores -- worse than a failure, because the send reports success. That is how the shutdown sweep came to count an obsolete-generation Signal as a handoff it had made. The manager holds the ledger, so the manager decides what a wake names; an entry in it, or a handoff for an intent just retired, both compose the unparked wake. The sweep's wake and that removal's handoff are then one obligation, recorded on the entry so the final pass does not send it twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`buffer_wft_if_outstanding_work` says it stores work "if there is any
outstanding WFT or activation for the run", and its condition never looked at
the WFT:
has_activation || about_to_issue_evict || self.more_pending_work()
`more_pending_work` is not a stand-in for that check. It is `wft.is_some() &&
machines.has_pending_jobs()`, so it answers false for the one state the
docstring names on its own -- an outstanding WFT whose machines have nothing
queued -- and a freshly polled task for that run went straight through to
`_incoming_wft`, which treats a second WFT for one run as a bug and
`dbg_panic!`s on it. A debug build loses the workflow-processing thread, and the
tokio worker then panics on `expect("Workflow processing terminates cleanly")`.
A release build logs and carries on into code that was never written to hold
two.
The window is real and is not a streaming regression: the method and the
invariant it claims date to 7473e80, the current condition to 2da4d2f. The
server can accept a completion and make the replacement task available to a
poller before Core processes the local post-completion message that clears
`ManagedRun.wft`, so a polled task can win that race. A task returned *in* the
completion response is already safe, because `process_post_activation` calls
`complete_wft` and finishes the activation before applying it; a separately
polled one had nothing holding it back.
Buffering is the safe answer rather than a heavier fix, because the drain side
already agrees: the buffer is emptied on the next run update that finds
`!has_any_pending_work(false, true)`, and that predicate does count
`wft.is_some()`. So the task waits exactly until the WFT in flight is cleared,
and `BufferedTasks::buffer` already handles a newer history-advancing task
replacing a buffered one.
The condition moves into `must_buffer_wft` so it is assertable on its own -- the
defect is the condition, and the test module here has no `ManagedRun` to build
one from. Three cases: an outstanding WFT alone must buffer (fails without this
fix, and only that case fails), nothing outstanding must still admit, and the
three reasons that already buffered still do. The end-to-end poll/completion
ordering needs control of Core's two input lanes and is left as separate work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Documentation only. Two investigation records move off the task volume, where nothing is version-controlled, into the set that already holds this genre beside `empty-stream-replay-flake-handoff.md`. `outstanding-issues.md` is the register: eight open items, what each one is, and whether it is diagnosed. The one currently failing the suite is a replay-after-eviction timeout whose cause is **not** established -- the server rejects Core's Workflow Task report as `BUSY_WORKFLOW`, "workflow operation can not be applied because workflow is closing", and Core evicts and replays into the same rejection exactly 85 times. It has been seen in two different tests with the identical signature and the same count, so it is one defect landing on whichever case is running rather than two flaky tests. `wft-double-dispatch-flake-handoff.md` is the evidence behind it, in three passes. The conclusions at the top supersede the archive below them, and the archive is kept because it shows which hypotheses led where. Both files say which of their own experiments cannot be trusted, and that is the point of committing them. A harness meant to settle whether the manager's wake Signals sustain that loop failed 15/15 while measuring its own broken delivery path; a baseline comparison confounded an instrumentation probe with the change under test. Either would cost a day to re-derive and neither proves anything. They also record what is ruled out and how: finding 14 is excluded by a probe that measured its new paths never executing during two reproductions, with the ledger populated -- reachable, and not taken -- rather than by reading the code, which is how two earlier readings here went wrong. The register carries the trap as well. `BUSY_WORKFLOW` is a transient signal, so the obvious fix -- treating it as terminal to stop the retry storm -- would drop a wake and lose a record, which is the failure the wake mechanism exists to prevent. `README.md` gains rows for both, under "what is still broken" and the flake chase, and the closing list marks them as records rather than design: they carry line numbers and dates, which the specs deliberately do not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three predicate tests exercise `must_buffer_wft` as a pure function. They prove the condition is right; they do not prove `ManagedRun` feeds it the live state. Add a stateful regression that drives a real `ManagedRun` into exactly the state the gate exists for -- a Workflow Task outstanding, activation finished, machines with nothing queued, no eviction pending -- and assert both halves: the replacement task buffers rather than reaching `_incoming_wft`, and it drains into the run once the outstanding task is reported. Without `has_wft` in the predicate all three remaining terms are false there, so the first assertion is what fails; the test cannot pass against the old gate. Record dispositions in the register: 1, 3, 4, 5 and 8 resolved, 2 and 7 open, 6 deferred. Say plainly that the optional lane-controlled race test was not added and why, rather than claiming it was unnecessary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replays Max's Option 7 prototype, External Workflow Streams in Core, onto the envelope base.
What changed?
It cherry-picks every commit on Max's
task/python-sdk-streamingsince it forked frommain, in order, onto the envelope in #38. Each commit keeps Max's authorship, message and date, and none came out empty.The content is Max's, unchanged, except for conflict resolution against upstream
main, which moved on under his branch. Themanaged_run.rsimports take Max's list plus upstream'sWftFailureKind, and its reset path keeps Max's three lines with upstream's(tt, attempt). The wake Signal filter takes upstream'sSignalWorkflow::from((attrs, event_id))in itselse. The bridge protos merged clean, so his activation jobs and commands keep the numbers they have on his branch.Two picks needed a compile fix against current upstream, made inside the pick that broke. The stray-command arm binds the command as
leaked @ (...), since upstream movescmd.variantout before the arm runs.add_cmd_to_wf_taskgetsDefault::default()where Max passedNone, since upstream takes the annotations by value, and therecord_markerconflict takes Max's side with the sameDefault::default().What the branch brings: the external stream protos and wait-set types, input routing and readiness, quiescence that retains the Workflow Task, the reserved wake Signal, the marker machine, the park handshake, finalization and rollover, Core-owned shutdown and eviction, the Workflow Task admission gate, workflow-originated output streams, and Max's ADR and design documents.
Part of AI-198 (epic AI-37).
Why?
This phase of the series wakes a parked reader with Max's reserved Signal and nothing else. The notification channel is paused per the 2026-10-05 design review, so the prototype now sits on the envelope and upstream
mainwith no channel below it, and the earlier version's channel conflicts are gone. A cherry-pick keeps his history reviewable commit by commit, with the authorship his. The fixes on top of the prototype are separate PRs, so this one is Max's work and nothing else.How did you test it?
Link to a test plan if any -
cargo fmt --check,cargo lint,cargo docwith warnings denied and the workspace lib suite pass. Two checks are red here, and the next PR fixes both:cargo test-lintflags atype_complexityin one of Max's tests, and the coverage reporter run with the ignored tests panics onExternalStreamMachine, which isn't registered yet. The integ suites weren't run, since they need a server.