fix: bound activity retries instead of retrying forever

[Quentin-David-24]

Problem

Most activities Flows schedules can retry forever.

• InfiniteRetryContext (send/update stages: CreateTransaction, DebitWallet, …) sets no MaximumAttempts.
• The trigger activities (ListTriggers, EvalTriggerVariables, InsertTriggerOccurrence, SendEventForTriggerTermination) and the instance/stage bookkeeping activities (InsertNewInstance, UpdateInstance, InsertNewStage, …) set no RetryPolicy at all, i.e. Temporal's unlimited default.

Any deterministic failure missing from a NonRetryableErrorTypes list therefore wedges the workflow for good, with nothing surfaced to the caller. In production we have seen InsertTriggerOccurrence reach attempt 20,916 and EvalTriggerVariables spin indefinitely.

Fix

One bounded policy for every activity, matching the one PaymentInitiationRetryContext already used: 2s initial interval, ×2 backoff, 200s cap, 15 attempts (~30–43 min worst case before giving up).

• New internal/retry package: retry.ActivityContext(ctx, timeout, nonRetryableCodes...) and retry.ShortActivityContext(ctx) (10s per attempt).
• InfiniteRetryContext → LedgerRetryContext, bounded, same non-retryable codes (VALIDATION, CONFLICT, NO_SCRIPT, COMPILATION_FAILED, INSUFFICIENT_FUND).
• PaymentInitiationRetryContext — same numbers, now built on the shared helper.
• Trigger and bookkeeping activities use retry.ShortActivityContext, keeping their 10s timeout.

No activity is left unbounded. Signal waits and delays schedule no activities; child workflow options are unchanged.

Tests

• stages/internal/context_test.go — both stage contexts pinned against literal values; policies don't share their non-retryable slices.
• stages/send/run_test.go — CreateTransaction always failing retryably is called exactly 15 times, then the workflow fails with that error.
• internal/triggers/workflow_trigger_retry_test.go, internal/workflow/run_retry_test.go — trigger and bookkeeping activities give up after 15 attempts and fail their workflow.

go build ./..., go vet -tags it ./internal/... and go test -race -tags it ./... all pass.

Deploying this

Changing activity options is replay-safe (they are not part of command matching). But a retry policy is captured when the activity is scheduled, so this only applies to activities scheduled after the deploy. Activities already retrying forever keep their old policy and must be terminated or reset by hand.

Behavioural changes worth a second opinion

• Stage activity exhausts its attempts: the instance now ends failed with the last activity error after ~40 min instead of staying "running" forever. Ledger operations committed earlier in the same stage are not rolled back (same as any stage failure today).
• Idempotency: retries within one activity reuse the key (RunID-ActivityID), so they can't double-post. A manual re-run (new instance or Temporal reset) gets new keys — check ledger/wallet state first if an attempt might have committed before failing.
• Bookkeeping activity exhausts its attempts: the workflow fails and the DB can be left stale (e.g. an instance still shown running if UpdateInstance never succeeded). This is the main trade-off; I judged it better than wedging forever.

Follow-up

The bound is still applied per call site. A WorkflowOutboundInterceptor defaulting RetryPolicy/MaximumAttempts for every ExecuteActivity would enforce it for future activities too.

Companion PR: #215 (non-retryable expression errors in triggers, independent).

[NumaryBot]

The required automated review completed with no remaining findings.

[flemzord]

Reviewed the bounded retry policy, activity call sites, failure propagation, tests, and documentation at 28fc566. No blocking findings.