Skip to content

feat!: Standardize Error Handling and Support Custom Error Reasons - #196

Open
natemort wants to merge 1 commit into
cadence-workflow:mainfrom
natemort:errors_v2
Open

natemort wants to merge 1 commit into
cadence-workflow:mainfrom
natemort:errors_v2

Conversation

@natemort

Copy link
Copy Markdown
Member

This is a breaking change!

Standardize the exception hierarchy, introducing CadenceError as the root of all Errors and FailureError as the root of any Failure expected in Workflow code. We call it FailureError rather than WorkflowError as ApplicationError is raised by Activity implementations.

Add ApplicationError. This allows users to explicitly specify the error reason, details, and any FailureOptions from a Workflow or an Activity. If a Workflow or Activity raises any other type of Error then the reason will be the name of the error type and the details will be a serialized representation of the error. We don't attempt to make this representation of the error round-trippable. It's purely for manual inspection and debugging, not to propagate stacktraces through a Workflow.

Within Workflows we introduce a consistent hierarchy for receiving exceptions from Activities or Child Workflows.

For Activities:

  • If cancelled, we raise asyncio.CancelledError
  • Else, we raise ActivityError. The reason for the error is the backing __cause__, which we also expose as a property:
    • ApplicationError if the activity failed. This contains the reason and details.
    • CadenceTimeoutError if the activity timed out.

For Child Workflows:

  • If cancelled, we raise asyncio.CancelledError
  • If we failed to start the Child Workflow, we raise StartChildWorkflowExecutionError (extends ChildWorkflowError)
  • Else, we raise ChildWorkflowError. Again the reason is the backing cause:
    • ApplicationError if the child workflow failed.
    • CadenceTimeoutError if the child workflow timed out.
    • TerminatedError if the child worfklow was terminated.

SignalFailure is removed, it was an implementation detail that was never exposed to users. Usage was replaced with FatalDecisionError.

The breaking changes are:

  • Removed most subtypes of ChildWorkflowError to align it with Activities.
  • Renamed all errors to end with Error:
    • ActivityFailure -> ActivityError
    • WorkflowFailure -> FailureError
    • SignalExternalWorkflowFailed -> SignalExternalWorkflowError
    • StartChildWorkflowExecutionError -> StartChildWorkflowExecutionError
  • Renamed StartChildWorkflowExecutionFailedError.cause -> failure_cause to avoid confusion with Error cause.

We add aliases for removed error types to reduce the impact of breaking changes.

What changed?

  • Added consistent behavior around error types and handling in Workflows.
  • Added ApplicationError to support an explicit failure reason

Why?

  • Make error handling clearer and consistent.
  • Support arbitrary error reasons, similar to Go SDK.

How did you test it?

  • Unit and integration tests.

Potential risks

  • This is a breaking change.

Release notes

  • breaking: Error types were refactored, see

Documentation Changes

Comment thread cadence/_internal/failure.py
Comment thread cadence/error.py Outdated
Comment thread cadence/error.py
This is a breaking change!

Standardize the exception hierarchy, introducing CadenceError as the root of all Errors and FailureError as the root of any Failure expected in Workflow code. We call it FailureError rather than WorkflowError as ApplicationError is raised by Activity implementations.

Add ApplicationError. This allows users to explicitly specify the error reason, details, and any FailureOptions from a Workflow or an Activity. If a Workflow or Activity raises any other type of Error then the reason will be the name of the error type and the details will be a serialized representation of the error. We don't attempt to make this representation of the error round-trippable. It's purely for manual inspection and debugging, not to propagate stacktraces through a Workflow.

Within Workflows we introduce a consistent hierarchy for receiving exceptions from Activities or Child Workflows.

For Activities:
- If cancelled, we raise asyncio.CancelledError
- Else, we raise ActivityError. The reason for the error is the backing `__cause__`, which we also expose as a property:
  - ApplicationError if the activity failed. This contains the reason and details.
  - CadenceTimeoutError if the activity timed out.

For Child Workflows:
- If cancelled by the Workflow, we raise asyncio.CancelledError
- If we failed to start the Child Workflow, we raise StartChildWorkflowExecutionError (extends ChildWorkflowError)
- Else, we raise ChildWorkflowError. Again the reason is the backing cause:
  - ApplicationError if the child workflow failed.
  - WorkflowCancelledError if the child workflow was cancelled externally
  - CadenceTimeoutError if the child workflow timed out.
  - TerminatedError if the child worfklow was terminated.

Distinguishing between cancellation from within the Workflow and external Cancellation is important because asyncio.CancelledError extends BaseException instead of Exception. It's unlikely that Workflow code expects a non-Exception type if they're not the one cancelling the Workflow. This aligns with the Java SDK.

SignalFailure is removed, it was an implementation detail that was never exposed to users. Usage was replaced with FatalDecisionError.

The breaking changes are:
- Removed most subtypes of ChildWorkflowError to align it with Activities.
- Renamed all errors to end with Error:
  - ActivityFailure -> ActivityError
  - WorkflowFailure -> FailureError
  - SignalExternalWorkflowFailed -> SignalExternalWorkflowError
  - StartChildWorkflowExecutionError -> StartChildWorkflowExecutionError
- Renamed StartChildWorkflowExecutionFailedError.cause -> failure_cause to avoid confusion with Error `cause`.
Comment on lines +156 to +157
if cancel_requested:
self.result.force_cancel(event.details.data.decode())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Edge Case: Strict UTF-8 decode of child cancel details can crash the decision task

When the parent has requested cancellation, handle_canceled calls self.result.force_cancel(event.details.data.decode()). This decodes the raw details payload as strict UTF-8. That payload comes from the pluggable DataConverter (or from whichever client canceled the child), so it is not guaranteed to be valid UTF-8; a binary converter such as protobuf or msgpack will produce invalid bytes. When it is not, a UnicodeDecodeError is raised inside the history event handler, which fails the decision task, and replaying the same history fails again every time. The rest of this PR decodes untrusted payload bytes with errors="replace" (see ApplicationError._from_payload). Do the same here, or leave the details out of the cancel message entirely.

Decode leniently so malformed bytes can't raise in the event handler:

self.result.force_cancel(
    event.details.data.decode("utf-8", errors="replace")
)
  • Apply fix

Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎

Comment thread cadence/error.py
Comment on lines +246 to +260
class ActivityError(FailureError):
"""An activity failed or timed out.

``cause`` is an :class:`ApplicationError` if the activity failed, or a
:class:`CadenceTimeoutError` if it timed out. Cancellation is raised as
:class:`asyncio.CancelledError` instead.
"""

class SignalExternalWorkflowFailed(Exception):
def __init__(self, message: str, cause: Any) -> None:
def __init__(self, message: str, *, activity_id: str, activity_type: str) -> None:
super().__init__(message)
self.cause = cause
self.activity_id = activity_id
self.activity_type = activity_type

@property
def cause(self) -> BaseException | None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Cross-Repo: Cadence-Docs Python error guide still uses removed error classes

Fix here and in cadence-workflow/Cadence-Docs

Breaks in: cadence-workflow/Cadence-Docs - docs/07-python-client/09-error-handling.md - Python error handling guide (Activity failures, Child workflow failures, Signal failures sections and Error reference table)

The Python error-handling guide in cadence-workflow/Cadence-Docs (docs/07-python-client/09-error-handling.md) imports and catches ActivityFailure, StartChildWorkflowExecutionFailed, ChildWorkflowExecution{Failed,Canceled,TimedOut,Terminated}, SignalExternalWorkflowFailed, WorkflowFailure and SignalFailure from cadence.error. This PR removes or renames all of these to ActivityError, ChildWorkflowError (with a cause), StartChildWorkflowExecutionError, SignalExternalWorkflowError and FailureError, and cadence/error.py defines no aliases for the old names. So every snippet in the guide now raises ImportError. The guide's statement that the five child-workflow classes are subclasses of ChildWorkflowError is also no longer true: a failure now arrives as a ChildWorkflowError whose cause is an ApplicationError, CadenceTimeoutError, TerminatedError or WorkflowCanceledError, and cancellation the parent requested surfaces as asyncio.CancelledError. The break is intentional, but the docs need a coordinated update. The PR description also promises aliases that the code does not provide.

Here: Add the deprecated aliases the PR description promises so the old imports keep resolving during migration, and link the docs update from the release notes.

There: Rewrite the Activity, Child workflow and Signal failure sections and the Error reference table in docs/07-python-client/09-error-handling.md for the new hierarchy: catch ActivityError / ChildWorkflowError and inspect .cause, use StartChildWorkflowExecutionError.failure_cause and SignalExternalWorkflowError, drop SignalFailure, and document ApplicationError and asyncio.CancelledError for cancellation the workflow itself requested.

In cadence-workflow/Cadence-Docs - docs/07-python-client/09-error-handling.md:

from cadence.error import ActivityError, ApplicationError, CadenceTimeoutError

try:
    await workflow.execute_activity(...)
except ActivityError as e:
    if isinstance(e.cause, ApplicationError):
        print(e.cause.reason, e.cause.details())
    elif isinstance(e.cause, CadenceTimeoutError):
        print(e.cause.timeout_type)

Hold the contract here:

# Deprecated aliases kept for backwards compatibility
ActivityFailure = ActivityError
WorkflowFailure = FailureError
SignalExternalWorkflowFailed = SignalExternalWorkflowError
StartChildWorkflowExecutionFailed = StartChildWorkflowExecutionError
  • Apply fix

Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎

@gitar-bot

gitar-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown
CI failed: Two integration tests in test_failures.py fail on application-error propagation: one hits an IndexError on an empty workflow history, and the reraised-error test fails with no visible cause in the truncated log. Relation to this PR is unclear for both, since the PR touches error handling but no specific behavior change explains the failures.

Overview

Two unique failure templates across 4 logs (2 jobs, both showing the same two failing integration tests in tests/integration_tests/workflow/test_failures.py). Both logs show the same pattern, so this is one underlying issue reported across two runs. Overall assessment: real test failures that need investigation, not clearly caused by this change.

Failures

Application-error round-trip and reraised-error integration tests fail (confidence: medium)

  • Type: test
  • Affected jobs: 116692875165, 116692875308
  • Related to change: unclear
  • Root cause: test_activity_application_error_details_round_trip fails at tests/integration_tests/workflow/test_failures.py:210 with IndexError because _close_event reads response.history.events[-1] from an empty history, meaning no close event was recorded. test_reraised_activity_application_error_fails_workflow also fails, but its traceback is truncated so the exact cause is not visible. The gRPC 'Channel is closed' warning is cleanup noise, not the failure. The PR modifies error handling (cadence/error.py, a new internal failure module), which overlaps this area, but no specific behavior change is identified that explains the failures.
  • Suggested fix: Pull the full traceback for the reraised-error test and the worker/server logs. Verify that activity application errors are converted and propagated so the workflow reaches a terminal failed state with history events recorded. If closure is asynchronous, make _close_event wait for a terminal state before reading history. Rerun both tests.

Summary

  • Change-related failures: None confirmed. Both failures are marked unclear, since they touch the error-handling area changed by this PR without a demonstrated causal link.
  • Infrastructure/flaky failures: None identified. The gRPC channel-closed message is a cleanup warning and was excluded.
  • Recommended action: Obtain the complete traceback for the reraised-error test, then determine whether the empty history reflects a real behavior regression from the error-handling changes or a test timing issue. Treat these as potential regressions in application-error propagation until ruled out.
Code Review ⚠️ Changes requested 3 closed / 5 findings

🔴 High risk · The breaking error hierarchy and new ApplicationError retry controls could break existing exception handlers and alter activity retry timing.

Standardizes the Python client error hierarchy around CadenceError, FailureError, ActivityError, and ChildWorkflowError, and adds ApplicationError for explicit failure reasons. Changes requested: the PR promises aliases for removed error names but cadence/error.py defines none, so old imports raise ImportError; the Cadence-Docs Python error guide still uses the removed classes and needs a coordinated update. Also, child cancel details are decoded as strict UTF-8 in child_workflow_execution_state_machine.py, which can raise UnicodeDecodeError and fail the decision task. The rename list in the description also has a typo.

⚠️ Cross-Repo: Cadence-Docs Python error guide still uses removed error classes

📄 cadence/error.py:246-260 📄 cadence/error.py:57-62

Fix here and in cadence-workflow/Cadence-Docs

Breaks in: cadence-workflow/Cadence-Docs - docs/07-python-client/09-error-handling.md - Python error handling guide (Activity failures, Child workflow failures, Signal failures sections and Error reference table)

The Python error-handling guide in cadence-workflow/Cadence-Docs (docs/07-python-client/09-error-handling.md) imports and catches ActivityFailure, StartChildWorkflowExecutionFailed, ChildWorkflowExecution{Failed,Canceled,TimedOut,Terminated}, SignalExternalWorkflowFailed, WorkflowFailure and SignalFailure from cadence.error. This PR removes or renames all of these to ActivityError, ChildWorkflowError (with a cause), StartChildWorkflowExecutionError, SignalExternalWorkflowError and FailureError, and cadence/error.py defines no aliases for the old names. So every snippet in the guide now raises ImportError. The guide's statement that the five child-workflow classes are subclasses of ChildWorkflowError is also no longer true: a failure now arrives as a ChildWorkflowError whose cause is an ApplicationError, CadenceTimeoutError, TerminatedError or WorkflowCanceledError, and cancellation the parent requested surfaces as asyncio.CancelledError. The break is intentional, but the docs need a coordinated update. The PR description also promises aliases that the code does not provide.

Here: Add the deprecated aliases the PR description promises so the old imports keep resolving during migration, and link the docs update from the release notes.

There: Rewrite the Activity, Child workflow and Signal failure sections and the Error reference table in docs/07-python-client/09-error-handling.md for the new hierarchy: catch ActivityError / ChildWorkflowError and inspect .cause, use StartChildWorkflowExecutionError.failure_cause and SignalExternalWorkflowError, drop SignalFailure, and document ApplicationError and asyncio.CancelledError for cancellation the workflow itself requested.

In cadence-workflow/Cadence-Docs - docs/07-python-client/09-error-handling.md:

from cadence.error import ActivityError, ApplicationError, CadenceTimeoutError

try:
    await workflow.execute_activity(...)
except ActivityError as e:
    if isinstance(e.cause, ApplicationError):
        print(e.cause.reason, e.cause.details())
    elif isinstance(e.cause, CadenceTimeoutError):
        print(e.cause.timeout_type)
Hold the contract here
# Deprecated aliases kept for backwards compatibility
ActivityFailure = ActivityError
WorkflowFailure = FailureError
SignalExternalWorkflowFailed = SignalExternalWorkflowError
StartChildWorkflowExecutionFailed = StartChildWorkflowExecutionError
💡 Edge Case: Strict UTF-8 decode of child cancel details can crash the decision task

📄 cadence/_internal/workflow/statemachine/child_workflow_execution_state_machine.py:156-157

When the parent has requested cancellation, handle_canceled calls self.result.force_cancel(event.details.data.decode()). This decodes the raw details payload as strict UTF-8. That payload comes from the pluggable DataConverter (or from whichever client canceled the child), so it is not guaranteed to be valid UTF-8; a binary converter such as protobuf or msgpack will produce invalid bytes. When it is not, a UnicodeDecodeError is raised inside the history event handler, which fails the decision task, and replaying the same history fails again every time. The rest of this PR decodes untrusted payload bytes with errors="replace" (see ApplicationError._from_payload). Do the same here, or leave the details out of the cancel message entirely.

Decode leniently so malformed bytes can't raise in the event handler
self.result.force_cancel(
    event.details.data.decode("utf-8", errors="replace")
)
✅ 3 closed
✅ Bug: Activity timeout no longer exposes heartbeat details or last failure

📄 cadence/_internal/failure.py:88-97 📄 cadence/error.py:165-170 📄 tests/cadence/_internal/workflow/statemachine/test_activity_state_machine.py:103-117
Before this PR, an activity timeout produced ActivityFailure(..., failure_details=<last_failure>, heartbeat_details=event.details). The old docstring documented heartbeat_details so workflows could decode the activity's last heartbeat progress. The new activity_timed_out_error reads only event.timeout_type. It drops event.details (the heartbeat payload) and event.last_failure (the last application failure before a retry ended in timeout). CadenceTimeoutError and ActivityError have nowhere to keep them, so workflow code can no longer resume from heartbeat progress or see why earlier attempts failed. The PR description does not list this as one of the breaking changes. The updated state-machine test (test_activity_state_machine.py:106) still builds the event with heartbeat details and last_failure but no longer asserts on either. Suggested fix: carry them on the timeout error, e.g. a heartbeat_details payload or decoder on CadenceTimeoutError, plus the last failure as an ApplicationError (or as its __cause__).

✅ Bug: Deprecated child-workflow aliases now catch unrelated failures

📄 cadence/error.py:366-380 📄 tests/cadence/test_error.py:39-50
ChildWorkflowExecutionCanceled, ChildWorkflowExecutionTimedOut, ChildWorkflowExecutionTerminated and ChildWorkflowExecutionFailed all now resolve to the same ChildWorkflowError. Existing code like except ChildWorkflowExecutionTimedOut: <retry/fallback> or except ChildWorkflowExecutionCanceled: return keeps importing without error, but it now also catches child failures, terminations and timeouts. That silently swallows or misroutes errors the old handler never meant to handle. The old canary in tests/canary/cancellation.py had exactly this except (CancelledError, ChildWorkflowExecutionCanceled): return pattern. On top of that, a real cancellation is now asyncio.CancelledError, so the Canceled alias never matches the case it is named for. These aliases make broken code look compatible, which is worse than an ImportError. Suggested fix: alias only the true renames (ActivityFailure, WorkflowFailure, SignalExternalWorkflowFailed, StartChildWorkflowExecutionFailed), and have __getattr__ raise an AttributeError with migration guidance for the removed subtypes.

✅ Quality: ApplicationError docstring references removed ChildWorkflowExecutionFailed

📄 cadence/error.py:93-96
The ApplicationError docstring says workflow code receives "an :class:ActivityError or :class:ChildWorkflowExecutionFailed". This PR removes that class, keeping it only as a deprecated alias of ChildWorkflowError. Readers are pointed at a name that now raises a DeprecationWarning, and the Sphinx cross-reference no longer resolves. It should reference ChildWorkflowError.

🤖 Prompt for agents
Code Review: Standardizes the Python client error hierarchy around `CadenceError`, `FailureError`, `ActivityError`, and `ChildWorkflowError`, and adds `ApplicationError` for explicit failure reasons. Changes requested: the PR promises aliases for removed error names but `cadence/error.py` defines none, so old imports raise `ImportError`; the Cadence-Docs Python error guide still uses the removed classes and needs a coordinated update. Also, child cancel details are decoded as strict UTF-8 in `child_workflow_execution_state_machine.py`, which can raise `UnicodeDecodeError` and fail the decision task. The rename list in the description also has a typo.

1. 💡 Edge Case: Strict UTF-8 decode of child cancel details can crash the decision task
   Files: cadence/_internal/workflow/statemachine/child_workflow_execution_state_machine.py:156-157

   When the parent has requested cancellation, `handle_canceled` calls `self.result.force_cancel(event.details.data.decode())`. This decodes the raw details payload as strict UTF-8. That payload comes from the pluggable `DataConverter` (or from whichever client canceled the child), so it is not guaranteed to be valid UTF-8; a binary converter such as protobuf or msgpack will produce invalid bytes. When it is not, a `UnicodeDecodeError` is raised inside the history event handler, which fails the decision task, and replaying the same history fails again every time. The rest of this PR decodes untrusted payload bytes with `errors="replace"` (see `ApplicationError._from_payload`). Do the same here, or leave the details out of the cancel message entirely.

   Fix (Decode leniently so malformed bytes can't raise in the event handler):
   self.result.force_cancel(
       event.details.data.decode("utf-8", errors="replace")
   )

2. ⚠️ Cross-Repo: Cadence-Docs Python error guide still uses removed error classes
   Files: cadence/error.py:246-260, cadence/error.py:57-62

   **Fix here and in [`cadence-workflow/Cadence-Docs`](https://github.com/cadence-workflow/Cadence-Docs)**
   
   **Breaks in:** [`cadence-workflow/Cadence-Docs`](https://github.com/cadence-workflow/Cadence-Docs) - [`docs/07-python-client/09-error-handling.md`](https://github.com/cadence-workflow/Cadence-Docs/blob/HEAD/docs/07-python-client/09-error-handling.md) - `Python error handling guide (Activity failures, Child workflow failures, Signal failures sections and Error reference table)`
   
   The Python error-handling guide in cadence-workflow/Cadence-Docs (`docs/07-python-client/09-error-handling.md`) imports and catches `ActivityFailure`, `StartChildWorkflowExecutionFailed`, `ChildWorkflowExecution{Failed,Canceled,TimedOut,Terminated}`, `SignalExternalWorkflowFailed`, `WorkflowFailure` and `SignalFailure` from `cadence.error`. This PR removes or renames all of these to `ActivityError`, `ChildWorkflowError` (with a `cause`), `StartChildWorkflowExecutionError`, `SignalExternalWorkflowError` and `FailureError`, and `cadence/error.py` defines no aliases for the old names. So every snippet in the guide now raises `ImportError`. The guide's statement that the five child-workflow classes are subclasses of `ChildWorkflowError` is also no longer true: a failure now arrives as a `ChildWorkflowError` whose `cause` is an `ApplicationError`, `CadenceTimeoutError`, `TerminatedError` or `WorkflowCanceledError`, and cancellation the parent requested surfaces as `asyncio.CancelledError`. The break is intentional, but the docs need a coordinated update. The PR description also promises aliases that the code does not provide.
   
   **Here:** Add the deprecated aliases the PR description promises so the old imports keep resolving during migration, and link the docs update from the release notes.
   
   **There:** Rewrite the Activity, Child workflow and Signal failure sections and the Error reference table in docs/07-python-client/09-error-handling.md for the new hierarchy: catch ActivityError / ChildWorkflowError and inspect `.cause`, use StartChildWorkflowExecutionError.failure_cause and SignalExternalWorkflowError, drop SignalFailure, and document ApplicationError and asyncio.CancelledError for cancellation the workflow itself requested.
   
   In cadence-workflow/Cadence-Docs - `docs/07-python-client/09-error-handling.md`:
   
   ````
   from cadence.error import ActivityError, ApplicationError, CadenceTimeoutError
   
   try:
       await workflow.execute_activity(...)
   except ActivityError as e:
       if isinstance(e.cause, ApplicationError):
           print(e.cause.reason, e.cause.details())
       elif isinstance(e.cause, CadenceTimeoutError):
           print(e.cause.timeout_type)
   ````

   Fix (Hold the contract here):
   # Deprecated aliases kept for backwards compatibility
   ActivityFailure = ActivityError
   WorkflowFailure = FailureError
   SignalExternalWorkflowFailed = SignalExternalWorkflowError
   StartChildWorkflowExecutionFailed = StartChildWorkflowExecutionError

Verification Repos 3 inspected · PRs 2 inspected

Functional validation No results

Rules No rules evaluated

Broader impact Repos 3 inspected PRs 2 inspected
Cross-repo 3 repositories inspected 1 open impact

cadence-workflow/cadence-samples Inspected
cadence-workflow/ai-skills Inspected
cadence-workflow/Cadence-Docs Inspected

Cross-PR 2 PRs inspected 0 open impacts

PR #1107: feat: Add ApplicationException Inspected
PR #191: fix(worker)!: Worker.run() now waits on failures or Worker.close() Inspected

For more verification details, comment gitar display:verbose.

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant