fix(workflow-executor): report runs that fail to hydrate instead of dropping them - #1830
Open
Scra3 wants to merge 3 commits into
Open
fix(workflow-executor): report runs that fail to hydrate instead of dropping them#1830Scra3 wants to merge 3 commits into
Scra3 wants to merge 3 commits into
Conversation
…ropping them /pending-run claims a run before the executor has hydrated it, and only an outcome (or the 60s reaper) clears that claim. A hydration failure that was not a WorkflowExecutorError was only logged customer-side, so the claim was never cleared: reaped after 60s, re-claimed, forever. Two Qonto prod runs cycled ~2 600 times over 3 days this way, blocked on a Camunda connector step (io.camunda:google-sheets:1) the executor cannot map. getAvailableRuns now buckets every hydration failure as malformed, and getAvailableRun wraps non-domain errors in MalformedRunError so the Runner reports on the trigger path too. toMalformedInfo accepts unknown and supplies a generic userMessage for raw errors; its workflowHistory access is now null-safe, since it runs inside the catch block that a non-array history reaches. The remaining unreportable case (toDispatch returning null: no available step to attach an error to) is now logged instead of silently skipped. Behaviour change: on the trigger path a raw hydration error now surfaces as 400 with a user-facing message instead of an opaque 500. It is still logged. fixes PRD-956 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Coverage Impact This PR will not change total coverage. Modified Files with Diff Coverage (1)
🛟 Help
|
No behaviour change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… optional chaining
toMalformedInfo runs inside the catch block that a malformed workflowHistory
reaches, so a throw there aborts the whole pending-runs batch instead of
reporting the run. `?.` only covers null/undefined: it still throws a TypeError
on {} and silently indexes a string ("abc".at(-1) === "c").
The two fixtures now use {} instead of null, which fails against the optional
chaining form.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Problem
/pending-runclaims a run (runState='loading',lockedAt=NOW()) before the executor has hydrated it. Only an outcome report — or the 60s reaper — clears that claim.In
getAvailableRuns, a hydration failure that was not aWorkflowExecutorErrorwas only logged, never reported:The log lands in the customer's infrastructure, so from our side the run just churns: claimed → dropped → reaped after 60s → re-claimed, forever.
Two Qonto prod runs (135900, 136259) cycled this way ~2 600 times over 3 days, blocked on
Task_UpdateGoogleSheet— abpmn:ServiceTaskcarrying a Camunda connector (io.camunda:google-sheets:1) that maps to none of the executor's 7 step types.reportMalformedRunalready does the right thing; this case simply never reached it.Fix
getAvailableRunsbuckets every hydration failure asmalformed, including non-domain ones. The existing "unexpected error" log is kept for the non-domain case.getAvailableRunwraps non-domain errors inMalformedRunErrortoo, so the Runner reports on the trigger path as well (that branch was previouslyistanbul ignored and untested).toMalformedInfoacceptsunknownand supplies a genericuserMessagefor raw errors. ItsworkflowHistoryaccess is now null-safe — it runs inside the catch block that a non-array history reaches, so throwing there would have killed the whole poll batch.toDispatchreturningnull— no available step to attach an error to) is now logged loudly instead of silently skipped.Everything downstream already worked: report → orchestrator records
context.error→ run parks inStarted→ no longer re-dispatched.On the trigger path, a raw hydration error now surfaces as 400 with a user-facing message (
"This step could not be loaded and cannot be executed.") instead of an opaque 500. It is still logged atErrorwith the technical message.Not covered by this PR
Tests
4 tests added/updated in
test/adapters/forest-server-workflow-port.test.ts, all verified to fail againstmain's port and pass here:WorkflowExecutorErroris reported and logged (previously assertedmalformedwas empty — that test pinned the bug)stepId/stepIndex), mirroring the PRD-956 payload shapegetAvailableRunwraps a raw error inMalformedRunError(new coverage)Full package suite: 1512 passed, 0 failed. Lint: 0 errors. Build clean.
fixes PRD-956
🤖 Generated with Claude Code
Note
Report runs that fail to hydrate as malformed instead of silently dropping them
getAvailableRunsin forest-server-workflow-port.ts now pushes aMalformedRunInfofor every hydration failure, including non-WorkflowExecutorErrortypes, rather than silently filtering them out.toMalformedInfonow acceptsunknownerrors and safely handles malformedworkflowHistoryshapes; uses a generic user message for non-domain errors.getAvailableRunnow always throwsMalformedRunErroron hydration failure, including for non-domain errors, and logs unexpected failures before doing so.Macroscope summarized 51fd648.