Skip to content

fix(worker): count activities where they run, not where they were scheduled - #440

Merged
KillerX merged 1 commit into
masterfrom
fix/activity-counter-on-activity-worker
Aug 14, 2026
Merged

fix(worker): count activities where they run, not where they were scheduled#440
KillerX merged 1 commit into
masterfrom
fix/activity-counter-on-activity-worker

Conversation

@KillerX

@KillerX KillerX commented Aug 12, 2026

Copy link
Copy Markdown
Member

1/n of a stack. Base: master.

The self-update gate is meant to stop a worker restarting mid-activity. It could not work, for two independent reasons.

Wrong process. ActivityWG was incremented by wfutils.Execute, which runs in the worker executing the workflow. Execute's whole job on the line above is to route the activity to the queue that owns it, so the count lands on a different worker than the work. The transcode, audio and live-ingest workers register no workflows at all, so their counter is always zero — the gate was a no-op on precisely the workers running multi-hour ffmpeg jobs, while the worker-queue process held itself back for activities it was not running.

Leaked counts. Add(1) ran on every replay, but Done() only ran if that workflow.Go coroutine was scheduled to completion. A coroutine still blocked when the workflow function returns is abandoned, so every fire-and-forget Execute leaked a count permanently, as does every eviction from the workflow cache. One leak is enough: ActivityWG.Wait() then blocks forever and that worker never self-updates again, silently. Verified with a throwaway test that a workflow.Go blocked at workflow return never runs the rest of its body.

Accounting now happens in the activity interceptor, which runs in the process executing the activity and releases through defer. Execute loses both the bookkeeping and the extra coroutine it spawned per call; that coroutine issued no commands, so removing it does not change history and is safe for in-flight workflows.

sync.WaitGroup is replaced rather than moved: Add must not race with Wait once the counter has hit zero, which panics with "WaitGroup misuse: Add called concurrently with Wait".

The gate itself now checks the version first and idleness second, so a busy worker no longer blocks on idle every five minutes to discover there was nothing to install — and it logs why an update was deferred. The residual race (an activity starting between the check and reload.Exec) is unchanged and needs a graceful worker.Stop to close.

analytics.GetService() gains a fallback for an uninitialised service: it returned nil and every method dereferences the receiver, so any process registering the interceptor without calling Init panicked on its first activity — which is what a test registering the interceptor does.

🤖 Generated with Claude Code

…eduled

The self-update gate is meant to stop a worker restarting mid-activity. It
could not work, for two independent reasons.

Wrong process. ActivityWG was incremented by wfutils.Execute, which runs in the
worker executing the *workflow*. Execute's whole job on the line above is to
route the activity to the queue that owns it, so the count lands on a different
worker than the work. The transcode, audio and live-ingest workers register no
workflows at all (cmd/worker/main.go:245-251), so their counter is always zero
— the gate was a no-op on precisely the workers running multi-hour ffmpeg jobs,
while the worker-queue process held itself back for activities it was not
running.

Leaked counts. Add(1) ran on every replay, but Done() only ran if that
workflow.Go coroutine was scheduled to completion. A coroutine still blocked
when the workflow function returns is abandoned, so every fire-and-forget
Execute leaked a count permanently, as does every eviction from the workflow
cache. One leak is enough: ActivityWG.Wait() then blocks forever and that
worker never self-updates again, silently. Verified with a throwaway test that
a workflow.Go blocked at workflow return never runs the rest of its body.

Accounting now happens in the activity interceptor, which runs in the process
executing the activity and releases through defer, so it cannot be skipped. It
also already wraps every activity, so Execute loses both the bookkeeping and
the extra coroutine it spawned per call. That coroutine issued no commands, so
removing it does not change history and is safe for in-flight workflows.

sync.WaitGroup is replaced rather than moved: Add must not race with Wait once
the counter has hit zero, and here activities start at arbitrary times relative
to the reader, which panics with "WaitGroup misuse: Add called concurrently
with Wait". ActivityCounter is an atomic counter handing out an idempotent
release func.

The gate itself is reordered to check the version first and idleness second,
so a busy worker no longer blocks on idle every five minutes to discover there
was nothing to install, and the check sits as close to the restart as it can.
It also logs why an update was deferred, which was previously invisible. The
residual race — an activity starting between the check and reload.Exec — is
unchanged and needs a graceful worker.Stop to close; logged as a follow-up.

analytics.GetService() gains a fallback for an uninitialised service. It
returned nil, and every method dereferences the receiver, so any process
registering the interceptor without calling Init panicked on its first
activity. That is what a test registering the interceptor does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@KillerX
KillerX merged commit ff310b9 into master Aug 14, 2026
@KillerX
KillerX deleted the fix/activity-counter-on-activity-worker branch August 14, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants