fix(worker): count activities where they run, not where they were scheduled - #440
Merged
Merged
Conversation
…eduled The self-update gate is meant to stop a worker restarting mid-activity. It could not work, for two independent reasons. Wrong process. ActivityWG was incremented by wfutils.Execute, which runs in the worker executing the *workflow*. Execute's whole job on the line above is to route the activity to the queue that owns it, so the count lands on a different worker than the work. The transcode, audio and live-ingest workers register no workflows at all (cmd/worker/main.go:245-251), so their counter is always zero — the gate was a no-op on precisely the workers running multi-hour ffmpeg jobs, while the worker-queue process held itself back for activities it was not running. Leaked counts. Add(1) ran on every replay, but Done() only ran if that workflow.Go coroutine was scheduled to completion. A coroutine still blocked when the workflow function returns is abandoned, so every fire-and-forget Execute leaked a count permanently, as does every eviction from the workflow cache. One leak is enough: ActivityWG.Wait() then blocks forever and that worker never self-updates again, silently. Verified with a throwaway test that a workflow.Go blocked at workflow return never runs the rest of its body. Accounting now happens in the activity interceptor, which runs in the process executing the activity and releases through defer, so it cannot be skipped. It also already wraps every activity, so Execute loses both the bookkeeping and the extra coroutine it spawned per call. That coroutine issued no commands, so removing it does not change history and is safe for in-flight workflows. sync.WaitGroup is replaced rather than moved: Add must not race with Wait once the counter has hit zero, and here activities start at arbitrary times relative to the reader, which panics with "WaitGroup misuse: Add called concurrently with Wait". ActivityCounter is an atomic counter handing out an idempotent release func. The gate itself is reordered to check the version first and idleness second, so a busy worker no longer blocks on idle every five minutes to discover there was nothing to install, and the check sits as close to the restart as it can. It also logs why an update was deferred, which was previously invisible. The residual race — an activity starting between the check and reload.Exec — is unchanged and needs a graceful worker.Stop to close; logged as a follow-up. analytics.GetService() gains a fallback for an uninitialised service. It returned nil, and every method dereferences the receiver, so any process registering the interceptor without calling Init panicked on its first activity. That is what a test registering the interceptor does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 12, 2026
sifferhans
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1/n of a stack. Base:
master.The self-update gate is meant to stop a worker restarting mid-activity. It could not work, for two independent reasons.
Wrong process.
ActivityWGwas incremented bywfutils.Execute, which runs in the worker executing the workflow. Execute's whole job on the line above is to route the activity to the queue that owns it, so the count lands on a different worker than the work. The transcode, audio and live-ingest workers register no workflows at all, so their counter is always zero — the gate was a no-op on precisely the workers running multi-hour ffmpeg jobs, while the worker-queue process held itself back for activities it was not running.Leaked counts.
Add(1)ran on every replay, butDone()only ran if thatworkflow.Gocoroutine was scheduled to completion. A coroutine still blocked when the workflow function returns is abandoned, so every fire-and-forgetExecuteleaked a count permanently, as does every eviction from the workflow cache. One leak is enough:ActivityWG.Wait()then blocks forever and that worker never self-updates again, silently. Verified with a throwaway test that aworkflow.Goblocked at workflow return never runs the rest of its body.Accounting now happens in the activity interceptor, which runs in the process executing the activity and releases through
defer.Executeloses both the bookkeeping and the extra coroutine it spawned per call; that coroutine issued no commands, so removing it does not change history and is safe for in-flight workflows.sync.WaitGroupis replaced rather than moved:Addmust not race withWaitonce the counter has hit zero, which panics with "WaitGroup misuse: Add called concurrently with Wait".The gate itself now checks the version first and idleness second, so a busy worker no longer blocks on idle every five minutes to discover there was nothing to install — and it logs why an update was deferred. The residual race (an activity starting between the check and
reload.Exec) is unchanged and needs a gracefulworker.Stopto close.analytics.GetService()gains a fallback for an uninitialised service: it returned nil and every method dereferences the receiver, so any process registering the interceptor without callingInitpanicked on its first activity — which is what a test registering the interceptor does.🤖 Generated with Claude Code