Skip to content

Added the workflow publish and subscribe commands. - #6

Closed
moetemp wants to merge 29 commits into
moe/AI-198-srv-3-stream-servicefrom
moe/AI-198-srv-4-workflow-commands
Closed

moetemp wants to merge 29 commits into
moe/AI-198-srv-3-stream-servicefrom
moe/AI-198-srv-4-workflow-commands

Conversation

@moetemp

@moetemp moetemp commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

This PR adds the commands a workflow uses to publish to a stream and subscribe to one.

What changed?

  • chasm/lib/workflow/stream_commands.go holds the handlers for AppendStreamRecords and SubscribeStream, their events and the library that registers them. A publish goes to a co-located stream the workflow owns, so the batch and the frontier land in the task's commit. A publish clears the producer id, which is how a reader tells the owner's records apart.
  • Both commands write an offsets-only event. One tiny record and a thousand 2KB records write the same event, because the bodies stay in the stream. Every SDK matches commands against their events in order, and a command with no event puts that matching out of step.
  • A subscribe is staged, not resolved in the handler. A stream in another execution has to be pinned on its own shard, which a handler holding the state lock can't reach. respondworkflowtaskcompleted/stream_appends.go resolves the staged subscriptions before the commit. The pin lands before the cursor, so an interruption leaves a pin nobody reads.
  • A subscribe that names neither an owned nor a standalone stream creates the owned one. A workflow reads a topic before anyone writes it, and an outside producer appends there later. Only stream_absent from the stream's shard counts as that case. A routing miss stays an error. StreamCursors keys owned streams by name and standalone ones by id in one map. A collision is refused, but History doesn't say which origin a range came from.
  • A refused publish or subscribe fails the workflow task with its own cause, not the completion call. A failed completion leaves the worker retrying with nothing in History saying why.
  • Both commands answer stream.enabled. stream.maxSubscriptionsPerWorkflow bounds subscriptions, and all routed calls of one task share stream.RoutedSetBudget, so the lock hold doesn't grow with the count. A publish counts against the owner's shared byte budget.
  • start_position rides the subscribe command. A bad position fails the task with BAD_SUBSCRIBE_STREAM_ATTRIBUTES. The flush records the resolved absolute offset on the event, so replay and reset see a fixed number.
  • When a workflow's activity finishes, deleteTerminalActivity closes its streams, so a reader tailing it is released while the workflow runs. A stream the activity never wrote to reads as open until the workflow closes. A command naming the reserved activity/ prefix fails the task. service/history/history_engine.go installs the routed stream client.

Delivery to a subscriber comes in #7.

Part of AI-198 (epic AI-37).

Why?

The publish path is the one that makes the design worth having. It costs one event per batch, and it's atomic with the workflow task. That's easier to judge on its own than next to the delivery machinery.

How did you test it?

go build ./... and make fmt are clean, and the lint findings are fixed. Unit tests ran for ./chasm/lib/... and ./service/history/.... With -tags test_dep, the functional tests ran the publish, refusal, failed-task, activity-end and recorded-offset cases. The replay and consume tests arrive with delivery in #7. Limit: a subscription to an owned stream ends at continue-as-new, and the successor gets a new stream at offset zero. One on another execution's stream carries over.

  • Unit Tests
  • Staging
  • End to End Tests

A publish appends to a stream co-located with the workflow, so it commits with the workflow task and a task that fails takes the publish with it. Both commands write an offsets-only event, because a command that produces none puts the command-to-event matching every SDK replays against out of step.
…workflow-commands

# Conflicts:
#	chasm/lib/workflow/stream_commands.go
A subscribe no longer reads any NotFound as proof that the name is the
workflow's own, since a routing failure would then bind it to different, empty
data. Subscriptions per execution are bounded, publishes count against the
shared byte budget, and both commands answer the namespace switch.
Each call had a deadline of its own and nothing bounded how many a task could
carry, so the execution's lock hold grew with the subscription count.
The append command only ever names a stream this workflow owns, and the subscribe command takes a name or an id. The appended event now bounds its batch the way every other range does.
…workflow-commands

# Conflicts:
#	chasm/lib/workflow/stream_admission_test.go
#	chasm/lib/workflow/stream_commands.go
Its streams live in the workflow's map and the workflow is still running, so
the rule that ends a stream with its execution never reaches them. A retry is
not terminal, so the next attempt keeps writing to the same streams.
Covers retry inheritance, the parked reader released at completion, and the workflow's own stream staying open.
The command is checked where it is handled, so a position that names nothing, or one sent
alongside a start offset, fails the task with the subscribe cause. The flush resolves it against
the stream and records the absolute offset on the subscribed event.
Nothing observable marks a parked reader, so the wait is the only way
to have it parked before the activity completes.
…workflow-commands

# Conflicts:
#	chasm/lib/stream/config.go
@moetemp moetemp closed this Oct 1, 2026
@moetemp moetemp reopened this Oct 1, 2026
@moetemp

moetemp commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Replaced by #18, #19, #20, #21, #22, #23, #24, #25, #26, #27, #28, #29, #30, #31, #32, #33, #34, #35, #36 and #37.

Same content, split into 20 PRs in the v3 series: the notification channel first, then the streaming interface, then native streams and the rest. The branch stays as a pin.

@moetemp moetemp closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant