fix(triggers): make expression evaluation errors non-retryable

[Quentin-David-24]

Problem

A trigger whose variable expression cannot be evaluated against the event wedges ExecuteTrigger forever.

EvalTriggerVariables returned expr compile/runtime errors as plain Go errors, and the activity runs with Temporal's default (unlimited) retry policy. Errors against the event payload alone are deterministic — the same payload fails the same expression on every attempt — so the if err != nil { occurrence.Error = ... } branch in ExecuteTrigger, which records the failed occurrence and publishes FAILED_TRIGGER, was never reached.

Seen in production: a trigger on the Connectivity event SAVED_PAYMENT_INITIATION_ADJUSTMENT (payload has no transactions) with the variable

get(event.transactions[0].metadata, "payout_cycle") ?? ""

fails with cannot fetch 0 from <nil> (1:23) on every attempt, thousands of times.

Fix

• EvalTriggerVariables classifies compile errors and runtime errors before any successful link() fetch as non-retryable EXPRESSION_EVALUATION errors. Runtime errors after a successful fetch stay retryable because the remote data can change. The fetch marker is local to each expression evaluation, including concurrent calls; explicitly classified link() errors retain their original retryability. The classification lives at the activity boundary so the shared evaluator stays Temporal-agnostic: the TestTrigger API keeps returning the plain expr message.
• link() now classifies its own errors, since expr wraps custom-function errors the same way as its own engine errors (e.g. an invalid regexp), so they can't be told apart after the fact:
  • transient HTTP failures (GET failure, HTTP 408/429 or 5xx, body decode) → retryable ApplicationError (type LINK);
  • permanent HTTP 4xx responses (except 408/429) and deterministic argument errors (links unmarshal, non-string second parameter) → non-retryable APPLICATION, like the existing unknown/multiple-link cases.
• Errors after a successful link() fetch are conservatively retryable even when a later error in that same expression is unrelated to the fetched data. They remain bounded by the activity policy.
• Filter evaluation is unchanged: a failing filter is still a non-match.

Tests

• internal/triggers/expression_test.go — table test through EvalTriggerVariables: the production payload/expression, a compile error, an invalid regexp and link() argument errors are all non-retryable; permanent link() 4xx responses stop retries; HTTP 408/429/500, bad body and connection refused stay retryable; a failing filter still doesn't match.
• internal/triggers/workflow_trigger_expression_test.go — ExecuteTrigger with the production case or an HTTP 404 linked resource runs EvalTriggerVariables exactly once, records the occurrence with the expr error and no workflow instance, and sends the termination event. Additional workflow tests verify that incomplete HTTP 200 linked data succeeds on the second attempt and starts the child workflow once, while persistently incomplete data records a failed occurrence after 15 attempts. Classification tests also cover compile errors mentioning link(), unexecuted links, permanent link errors following a successful fetch, and isolation between concurrent expressions.

go build ./..., go vet -tags it ./internal/... and go test -race -tags it ./... all pass.

Deploying this

This is an activity-code change, so it does reach workflows already wedged on this error: their next retry runs on the new worker and fails non-retryably (within ~100s, the default max backoff). Each one then records a failed occurrence and publishes FAILED_TRIGGER — expect a burst of those events for affected stacks on rollout.

Minor behaviour changes

• TestTrigger API: a transient link() failure message now carries the Temporal suffix (type: LINK, retryable: true), like the non-retryable link() errors already did.
• The failed-occurrence error string gains (type: EXPRESSION_EVALUATION, retryable: false).

PR #216 (bounded retry policies for all activities) is merged into main and included in this branch. Already scheduled retries can still retain their old policy; permanent expression and linked-resource errors stop immediately with the new activity code.

[NumaryBot]

The required automated review completed with no remaining findings.

[NumaryBot]

The required automated review completed with no remaining findings.

[NumaryBot]

The required automated review completed with no remaining findings.