Skip to content

Commit fork, rewind, delay and attribute changes with their checkpoint - #657

Draft
devhawk wants to merge 9 commits into
transactional-internal-stepsfrom
transactional-fork-rewind-delay
Draft

devhawk wants to merge 9 commits into
transactional-internal-stepsfrom
transactional-fork-rewind-delay

Conversation

@devhawk

@devhawk devhawk commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Stacked on the PR that adds runTransactionalStep. This one moves forkWorkflow, rewindWorkflow, setWorkflowDelay and updateWorkflowAttributes onto that helper. Each now commits its effect and its step checkpoint in one transaction, under the step name it already used.

Before, a crash between the two commits made recovery repeat the operation. A fork ran a second time under a new random ID. A second rewind was refused, because the first had left the target ENQUEUED.

Details

  • Checks stay inside the step. The executor's queue check, and a rewind's clearing of the transactional step factories' checkpoints, run after the checkpoint check. A replay therefore returns the recorded result without repeating them, even if the queue has since been unregistered.
  • Rewind's checkpoint deletes commit on their own. Those deletes go through the factories' own connections, a second one from the system database's pool if a factory shares it. A rewind that then rolls back leaves them deleted, with the workflow still terminal. That was already the case before this change.
  • The fork's ID is chosen before the transaction. A retry that re-runs the work, from a client after a lost commit acknowledgement, fails on the fork it already wrote rather than forking again under a second ID.
  • A relative delay is resolved inside the work. It counts from the attempt that commits, as a started workflow's deadline does. A delay past the latest instant is still recorded as the step's error.
  • DAO and client changes. The DAO methods take the caller's connection, and the batch fork runs through the helper with no caller. DBOSClient calls pass no caller.

Testing

New tests in InternalStepTransactionTest:

  • a fork that fails before commit leaves no fork;
  • a lost commit acknowledgement replays the fork;
  • a lost commit acknowledgement does not rewind again.

Targeted run: 235 of 235 pass. The classes were:

  • InternalStepTransactionTest, ForkTest, RewindTest, SelfTargetTest
  • WorkflowAttributesTest, OversizedDurationTest, DelayReleaseClockTest
  • PayloadTableReadTest, WorkflowMgmtTest, SystemDatabaseTest
  • ClientTest, DBOSConductorEngineTest

spotlessCheck, compileTestJava and javadoc pass.

🤖 Generated with Claude Code

forkWorkflow, rewindWorkflow, setWorkflowDelay and
updateWorkflowAttributes now run through runTransactionalStep, so each
commits its effect and its step checkpoint in one transaction under the
step name it already used. A crash between the two can no longer make
recovery repeat the operation: a second fork under a new random ID, or
a second rewind that is refused because the first left the target
ENQUEUED. Called from inside a step, each commits only while the caller
still owns its workflow; from a client, it runs in a plain transaction.

The executor's queue check, and a rewind's clearing of the
transactional step factories' checkpoints, still run inside the step,
after the checkpoint check, so a replay returns the recorded result
without repeating them. A fork's ID is chosen before the transaction,
so a retry that re-runs the work fails on the fork it already wrote
rather than forking again. A relative delay is resolved inside the
work, so a delay past the latest instant is still recorded as the
step's error.

The DAO methods take the caller's connection instead of opening their
own transaction, and the batch fork runs through the same helper with
no caller.
@devhawk
devhawk added this pull request to stack #659 October 30, 2026 17:34
@devhawk
devhawk marked this pull request as draft October 30, 2026 17:39
…ounts

setWorkflowDelay resolved a relative delay inside the step's work, so
every retry of the transaction pushed it later by that attempt's
backoff, contradicting resolveDelayUntil's own contract. It is now
resolved once, before the transaction; a delay that cannot be resolved
is thrown from inside the work, so the refusal is still recorded as the
step's error.

The rewind comments no longer claim nothing can run against a partly
cleared history: a rewind that rolls back after the step factories'
deletes leaves the workflow terminal, and a resume would still run it
without those checkpoints.

New tests: attributes and a delay whose step fails before commit change
nothing and record nothing, and the lost-acknowledgement rewind test
counts the target's runs instead of relying on its status.
The transactional step factories' checkpoints were deleted inside the
rewind step's transaction. A factory sharing the system database's pool
then asked that pool for a second connection while the step held one,
so enough concurrent rewinds could exhaust it, and a rewind from a
workflow would record the pool timeout as its permanent error.

The deletes now run before the transaction opens, as they did before
the rewind became a transactional step, with the queue and rewindable
checks ahead of them. A step already recorded skips them, so a replay
does not delete again; one recorded by a rival meanwhile is found by
the step and replayed. A refusal reached before the transaction is
thrown from inside the step, so it is still recorded as the step's
error.

RewindTest now counts the deletes and checks that recovering a workflow
that rewound another does not repeat them.
Before its step's transaction, a rewind asked whether the step was
already recorded, which took two round trips through the full step
check, and then read the target's status in a third. One statement now
reads whether the calling step has a row, whether the caller still
owns its workflow, and the target's status. The full step check still
runs in the step's transaction, which replays a recorded step and
refuses a caller that has lost its workflow; the pre-check only decides
whether to clear the step factories' checkpoints first, which a caller
that has lost its workflow now leaves alone.
…ewind-delay

# Conflicts:
#	transact/src/test/java/dev/dbos/transact/workflow/InternalStepTransactionTest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant