Repository navigation
Conversation
…self-target
A DBOS operation called from a workflow committed its effect and the
checkpoint recording it in separate transactions, so a crash between the
two let a replay run the operation again. SystemDatabase gains
runTransactionalStep: from the workflow body, one transaction checks the
step (replaying a recorded output or error), runs the work, and records
it under the ownership check. If the work fails, the transaction rolls
back and the failure is recorded in a fresh one; a failure the retries
would have kept trying, a lost ownership, or a mismatched checkpoint is
not recorded. From inside a step nothing is recorded, but the work
commits only while the caller owns the workflow. Outside a workflow it
is a plain transaction.
cancelWorkflows, resumeWorkflows and deleteWorkflows now run through it,
in place of runDbosFunctionAsStep. A cancel with children is one
transaction that cancels each level before reading its children, so a
child whose insert holds its parent's lock is read once that insert
commits, and one inserted later is refused by the ownership check on its
parent. SendCaller becomes StepCaller, the caller every such operation
takes.
In one transaction, a workflow that cancels, resumes or deletes itself
would fail its own checkpoint's ownership check and wait forever. Such a
call is now refused with IllegalArgumentException, from the workflow
body or a step: the caller among the targets before any write, and a
cancel or delete whose tree contains the caller ("workflow X cannot
cancel the tree of R, which contains it"), rolling back any level
already cancelled. A workflow cancelling its own child's tree is
unaffected. Step names and recorded output shapes are unchanged.
Every DBOS cancel, resume and delete overload now says, under @throws IllegalArgumentException, that a call from inside a target, or one of its steps, or with children included from inside a target's descendants, is refused and changes nothing.
…in tests From review: - runTransactionalStep sets READ COMMITTED instead of inheriting the pool's level. A tree walk depends on it: each level's children read must see a child committed while the level's write waited on its parent, which a REPEATABLE READ snapshot would hide. - deleteWorkflows walks the tree as cancelWorkflows does, deleting each level before reading its children, so a child started mid-delete is either found once its insert commits or refused because its parent is gone; it read the whole tree first, and a child committed between the read and the delete was left behind. Both share walkTree, which carries each workflow's root down the walk, so a refusal names the tree without reading rows the walk may already have changed; the ancestry reads go. - A failure is recorded only when the exception that escaped is the work's own, or the retries' wrapping of it. - Replaying a step with neither output nor error recorded throws, as the old path did. The cancel, resume and delete log lines run inside the work, so a replay or a refused call does not log them. - sendBulkAs takes its caller from stepCaller(). Tests: the commit triggers fire only on the test's own workflow, and each test asserts its trigger fired; a child whose insert holds its parent is cancelled with the tree, and one inserted after the cancel is never created (a new debug trigger holds the insert open); a fork replays a recorded refusal instead of deciding it again; isRetryableFailure.
…docs A cancel, resume or delete whose work still fails once its retries run out, a serialization failure or deadlock included, now records that failure as the step's error. Before, the step ID was used up with no row behind it, so a workflow that caught the failure and carried on could take the other branch on replay, when the call might succeed. Only a failure an interrupt cut short is left unrecorded, along with a lost ownership, a mismatched checkpoint and a failed commit, none of which come from the work. The cancel, resume and delete log lines now follow the DAO call, so a refused self-target is not logged. The checkOwner javadoc no longer claims fenced writes cannot deadlock: two workflows cancelling each other can, and the retry resolves it. The runTransactionalStep javadoc now says a lost commit acknowledgement is replayed only for a call from the workflow body; from a step or a client the work runs again. Tests cover a recorded conflict and its replay, a stale caller's cancel rolling back from the body and from a step, and a child inserted during or after a delete of its parent's tree.
devhawk
marked this pull request as draft
October 30, 2026 17:38
…solation The cancel, resume and delete log lines ran inside the work, so they repeated on every retry and appeared even when the ownership check then rolled the step back. They now log once, after the commit, and only when the work ran rather than a replay of its checkpoint. The runTransactionalStep javadoc now says that only what the work throws is recorded: a conflict the database raises on the step check or on the record around it is not, any more than a lost ownership is. New tests cover a workflow that changes hands while the work runs, whose record is then refused and whose work rolls back, and a tree cancel on a pool that defaults to REPEATABLE READ, which lets a concurrently inserted child escape unless the step sets READ COMMITTED itself.
A tree walk wrote each level and then read its children, one round trip per statement: two a level for a cancel, five for a delete with its input, output and step rows. Each level's statements and the read of its children now go in one prepared statement, which the driver sends as separate statements in a single flush. They still run one after another, each on a snapshot of its own under READ COMMITTED, so the read still sees a child whose insert held its parent's lock until the level's write could proceed. A single statement would not, as its one snapshot is taken before the wait. Every statement takes the level's IDs as its only parameter; the cancel's state names are now literals.
…he native format A conflict that outlasts its retries is now the step's answer whichever statement the database raised it in, not only when the work raised it. An internal step's recorded error is serialized in the native format, so a replay rethrows the type the workflow caught under a custom serializer. Adds tests for a bulk cancel naming its caller, a tree refusal from a step, resume's rollback and lost acknowledgement, two workflows cancelling each other, and a send in a startWorkflow lambda. The lambda refusal now names DBOS operations, and getDirectChildren reuses childrenSql.
devhawk
added a commit
that referenced
this pull request
Nov 12, 2026
Called from inside a workflow, or from one of its steps, `cancelWorkflow(s)`, `resumeWorkflow(s)` and `deleteWorkflow(s)` now throw `IllegalArgumentException` when their targets include the calling workflow. Nothing is written, and the workflow can catch the exception and carry on. ## Why Each of these clears or removes the caller's own `workflow_status` row, so under the ownership fence the caller's next checkpoint fails: - a self-cancel ended `CANCELLED`, with its cancel step never recorded; - a self-delete ended with the workflow gone; - a self-resume put the workflow back on the queue before its own step was recorded. Each dispatch replayed to the same resume and did it again, with no end short of deleting the workflow. ## How The check runs first inside the call's internal step, before any database call: - From a workflow body, the refusal is recorded as the error of `DBOS.cancelWorkflow` / `DBOS.resumeWorkflow` / `DBOS.deleteWorkflow`, so a recovery or fork rethrows it instead of deciding again. - From a step body, the call takes no step of its own, and the refusal is thrown into the enclosing step. Calls with no workflow context (outside a workflow, `DBOSClient`, Conductor, the admin server) are unaffected, as are calls that target other workflows, including the caller's own children's trees. This PR refuses only a caller that names itself. Refusing a cancel or delete of an ancestor's tree that contains the caller (`cancelChildren` / `deleteChildren`) needs the cascade and its step record in one transaction, and comes with #656. ## Tests - New `SelfTargetTest` covers: - self-cancel, self-resume and self-delete, from a workflow body and from a step; - a bulk cancel naming the caller, which leaves the other targets untouched; - replay of a recorded refusal through a fork; - an uncaught refusal becoming the step's error; - cancelling another workflow and one's own child's tree, which still work. - `OwnershipFenceTest`: removes the three tests that recorded the old self-cancel, self-delete and self-resume behaviour. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DBOS operation called from a workflow committed its effect and its step checkpoint in separate transactions. A crash between the two made recovery run the operation again. This PR adds one helper that commits both together, and moves cancel, resume and delete onto it. It is the first of three stacked PRs that move every DBOS step onto the helper.
The helper
SystemDatabase.runTransactionalStep(caller, stepName, work)runs in one transaction at READ COMMITTED. The whole transaction is retried on connection failures, 40001 and 40P01.If the work fails, the transaction rolls back and the failure is recorded as the step's error in a fresh transaction, so a replay is refused the same way. A conflict that outlasted its retries is recorded too, whichever statement the database raised it in: the work, the step check, or the record and owner check after it. The error is recorded in the native format whatever serializer is configured, so a replay rethrows the type the workflow caught.
Some failures are not recorded, and a workflow that catches one may take another branch on replay:
Cancel, resume and delete
IllegalArgumentException, which the workflow can catch. Before, its own step's ownership check would roll the call back and leave the workflow waiting on an outcome nothing writes.Behaviour changes
DBOS.cancelWorkflow(DBOS.workflowId())used to end the workflow CANCELLED; it now throwsIllegalArgumentException. A workflow that cancelled itself under an earlier release, and is recovered after upgrading, hits the refusal at that step.DBOS.sendfrom inside astartWorkflowlambda now throws, as other DBOS steps there already did.DBOSClientnow log too.Notes
WorkflowDAO.deleteWorkflowssignature this changes.Testing
New tests:
InternalStepTransactionTest,SelfTargetTestandTransactionalStepTest, 30 tests, all passing.OwnershipFenceTestpassed before the last commit and was not rerun after it.spotlessCheck,compileTestJavaandjavadocpass.🤖 Generated with Claude Code