Skip to content

fix(workflow-executor): report runs that fail to hydrate instead of dropping them - #1830

Open
Scra3 wants to merge 3 commits into
mainfrom
fix/prd-956-report-unhydratable-runs
Open

fix(workflow-executor): report runs that fail to hydrate instead of dropping them#1830
Scra3 wants to merge 3 commits into
mainfrom
fix/prd-956-report-unhydratable-runs

Conversation

@Scra3

@Scra3 Scra3 commented Aug 14, 2026

Copy link
Copy Markdown
Member

Problem

/pending-run claims a run (runState='loading', lockedAt=NOW()) before the executor has hydrated it. Only an outcome report — or the 60s reaper — clears that claim.

In getAvailableRuns, a hydration failure that was not a WorkflowExecutorError was only logged, never reported:

if (error instanceof WorkflowExecutorError) {
  malformed.push(this.toMalformedInfo(run, error));   // reported
} else {
  this.logger('Error', 'Failed to hydrate pending run — unexpected error', {});
}                                                     // dropped silently

The log lands in the customer's infrastructure, so from our side the run just churns: claimed → dropped → reaped after 60s → re-claimed, forever.

Two Qonto prod runs (135900, 136259) cycled this way ~2 600 times over 3 days, blocked on Task_UpdateGoogleSheet — a bpmn:ServiceTask carrying a Camunda connector (io.camunda:google-sheets:1) that maps to none of the executor's 7 step types. reportMalformedRun already does the right thing; this case simply never reached it.

Fix

  • getAvailableRuns buckets every hydration failure as malformed, including non-domain ones. The existing "unexpected error" log is kept for the non-domain case.
  • getAvailableRun wraps non-domain errors in MalformedRunError too, so the Runner reports on the trigger path as well (that branch was previously istanbul ignored and untested).
  • toMalformedInfo accepts unknown and supplies a generic userMessage for raw errors. Its workflowHistory access is now null-safe — it runs inside the catch block that a non-array history reaches, so throwing there would have killed the whole poll batch.
  • The one genuinely unreportable case (toDispatch returning null — no available step to attach an error to) is now logged loudly instead of silently skipped.

Everything downstream already worked: report → orchestrator records context.error → run parks in Started → no longer re-dispatched.

⚠️ Behaviour change

On the trigger path, a raw hydration error now surfaces as 400 with a user-facing message ("This step could not be loaded and cannot be executed.") instead of an opaque 500. It is still logged at Error with the technical message.

Not covered by this PR

  • If the error report itself fails server-side, the loop resumes (better logged). The ticket's "bound the retry" direction is still open.
  • Rejecting unsupported step types at workflow publication time — the real product fix, separate ticket. This BPMN should never have reached prod.

Tests

4 tests added/updated in test/adapters/forest-server-workflow-port.test.ts, all verified to fail against main's port and pass here:

  • non-WorkflowExecutorError is reported and logged (previously asserted malformed was empty — that test pinned the bug)
  • a raw failure is reported against the correct pending step (stepId/stepIndex), mirroring the PRD-956 payload shape
  • getAvailableRun wraps a raw error in MalformedRunError (new coverage)
  • a run with no executable step is dropped and logged

Full package suite: 1512 passed, 0 failed. Lint: 0 errors. Build clean.

fixes PRD-956

🤖 Generated with Claude Code

Note

Report runs that fail to hydrate as malformed instead of silently dropping them

  • getAvailableRuns in forest-server-workflow-port.ts now pushes a MalformedRunInfo for every hydration failure, including non-WorkflowExecutorError types, rather than silently filtering them out.
  • toMalformedInfo now accepts unknown errors and safely handles malformed workflowHistory shapes; uses a generic user message for non-domain errors.
  • getAvailableRun now always throws MalformedRunError on hydration failure, including for non-domain errors, and logs unexpected failures before doing so.
  • Runs with no available step (unreportable) are still dropped but are now explicitly logged as an error instead of being silently discarded.

Macroscope summarized 51fd648.

…ropping them

/pending-run claims a run before the executor has hydrated it, and only an
outcome (or the 60s reaper) clears that claim. A hydration failure that was not
a WorkflowExecutorError was only logged customer-side, so the claim was never
cleared: reaped after 60s, re-claimed, forever.

Two Qonto prod runs cycled ~2 600 times over 3 days this way, blocked on a
Camunda connector step (io.camunda:google-sheets:1) the executor cannot map.

getAvailableRuns now buckets every hydration failure as malformed, and
getAvailableRun wraps non-domain errors in MalformedRunError so the Runner
reports on the trigger path too. toMalformedInfo accepts unknown and supplies a
generic userMessage for raw errors; its workflowHistory access is now
null-safe, since it runs inside the catch block that a non-array history
reaches.

The remaining unreportable case (toDispatch returning null: no available step
to attach an error to) is now logged instead of silently skipped.

Behaviour change: on the trigger path a raw hydration error now surfaces as
400 with a user-facing message instead of an opaque 500. It is still logged.

fixes PRD-956

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Aug 14, 2026

Copy link
Copy Markdown

PRD-956

@qltysh

qltysh Bot commented Aug 14, 2026

Copy link
Copy Markdown

Qlty


Coverage Impact

This PR will not change total coverage.

Modified Files with Diff Coverage (1)

RatingFile% DiffUncovered Line #s
Coverage rating: A Coverage rating: A
.../workflow-executor/src/adapters/forest-server-workflow-port.ts100.0%
Total100.0%
🚦 See full report on Qlty Cloud »

🛟 Help
  • Diff Coverage: Coverage for added or modified lines of code (excludes deleted files). Learn more.

  • Total Coverage: Coverage for the whole repository, calculated as the sum of all File Coverage. Learn more.

  • File Coverage: Covered Lines divided by Covered Lines plus Missed Lines. (Excludes non-executable lines including blank lines and comments.)

    • Indirect Changes: Changes to File Coverage for files that were not modified in this PR. Learn more.

No behaviour change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/workflow-executor/src/adapters/forest-server-workflow-port.ts Outdated
… optional chaining

toMalformedInfo runs inside the catch block that a malformed workflowHistory
reaches, so a throw there aborts the whole pending-runs batch instead of
reporting the run. `?.` only covers null/undefined: it still throws a TypeError
on {} and silently indexes a string ("abc".at(-1) === "c").

The two fixtures now use {} instead of null, which fails against the optional
chaining form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant