Skip to content

fix(protocol): a cancelled event move stays cancelled (#254) - #374

Draft
Hafeok wants to merge 18 commits into
mainfrom
claude/event-move-race-254
Draft

Hafeok wants to merge 18 commits into
mainfrom
claude/event-move-race-254

Conversation

@Hafeok

@Hafeok Hafeok commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Completes: #254

What this changes, and why

#254: EventMoveTests.A_catalogue_that_does_not_translate_one_to_one_cancels_the_move_and_writes_reopen failed now and then with 409 event-catalogue-moved after a cancel. The server answered the cancel only once it was in force. The fleet's interval loop then undid it: it had picked the dataset as Moving, and its run of the move took varve:InDataset as a move to start and froze the catalogue again. With a catalogue that translates, that run went on to Moved, which is final. The cause, the plan and the maintainer's answers are on #254.

The fixes, each its own commit, with its own tests:

  1. (a) A run of the move never undoes a cancel.
    • A run with no request never freezes.
    • A run that has seen Moving, by its own freeze or by reading it, commits nothing more when it then finds InDataset.
    • A request's run answers the new 409 event-move-cancelled, which has a page and "What to do" (ADR 0242). The client does not retry it.
  2. (b) A cancel whose write failed answers that failure, not "cancelled".
  3. (c) A write to a dataset that is Moving here advances the system follower once before its grants are pinned.
    • What it adds to such a write: one read of the store, then the reconcile under the advance's lock. The read is a probe of the next log slot on a follower, the writer record on the system writer on an object store, and nothing on a fleet of one.
    • When the plane cannot be read: a catalogue write to a dataset that was Moving is 503 management-behind, which the client already sends again (ADR 0246 point 10). It is never 409 event-catalogue-moved. Moved stays 409.
  4. A management write answered NoChange is applied before its answer. The model-based test found this; the plan did not list it.
  5. One run of a dataset's move at a time (IManagementPlane.EnterMoveAsync and TryEnterMove, default members). A request waits for a run in progress; the loop's resume does not. eng/repeat.cs --load found this after the first fixes, and so did the hosted 500 on Windows.
  6. ADR 0167 amended 2026-11-19, filed unaccepted, with three keys. The last code commit cites them, so this head is red on CS0618 alone (ADR 0066). The tested head is 1cd717d.
  7. A dispatch-only repeat job in nightly.yml, running eng/repeat.cs, described in docs/testing.md §11. It is never a required check, and the required-checks gate refuses it as one (proven).

Unaccepted decisions cited (ADR 0066), each by Set.Key at its citation site:

  • EventDeclarationsMoveToTheManagementPlane.ResumeOnlyFinishesAMove on src/Varve.Server/Fleet.cs;
  • EventDeclarationsMoveToTheManagementPlane.CancelIsAnsweredOnlyWhenInForce on src/Varve.Protocol/Endpoints/ManagementAnswers.cs;
  • EventDeclarationsMoveToTheManagementPlane.MovingWriteAdvancesFirst on src/Varve.Server/MovingDatasetWrites.cs.

Public API. Varve.Protocol.Client is unchanged. Varve.Protocol is grown, not broken (api freeze green at 1cd717d):

  • ProblemType.EventMoveCancelled;
  • EventCatalogueState;
  • a second EventCatalogueValidator constructor;
  • two default members on IManagementPlane.

What was measured

The flaky test, under load Linux Windows
Before, c754bcb, this container 11 of 300 failed 3 hosted sightings
After the first fixes, 627ca92 / a247813 0 of 500 hosted; 1 of 300 here 1 of 500 hosted
After every fix, 1cd717d, the gate 0 of 500 hosted (run); 0 of 300 here 0 of 500 hosted (run)
  • The red-test head a3af2c8 failed on exactly the two new tests, on Linux, Windows and the devcontainer.
  • The model-based test fails 12 runs of 12, of 300 schedules each, with any one fix removed. With every fix, 1,000 schedules ran clean.
  • Durations, before (c754bcb) and after (1cd717d), one run each:
    • server suite: Linux 9.9 s → 9.9 s, Windows 14.2 s → 12.8 s;
    • fleet (services): 1 min 38 s → 1 min 59 s, not attributed, since one run is too few;
    • handover (services): 34 s → 36 s.
  • IdleFleetTests passed, unchanged.

What was found

What is left

  • The maintainer's acceptance of the three keys, which turns the head green.
  • The close-out is docs/sessions/2026-11-event-move-race.md, and the record is docs/traceability/2026-11-19-issue-254-event-move-race.md.

Checklist

  • Issue referenced: every commit carries Refs #254, and the last Closes #254 (eng/issue-refs.cs ok)
  • ADR cited: ADR 0167 amended, filed unaccepted
  • Specification updated, or n/a: n/a; the operator guide and the problem pages updated
  • API baseline updated: PublicAPI.Unshipped.txt in Varve.Protocol
  • Conformance baseline updated, or n/a: n/a
  • Tests: real host and fleets; doubles only at the test seams on Fleet
  • A new gate's failure path was proven: the required-checks gate refused repeat (ubuntu-latest) as required; eng/repeat.cs exits 2 for a filter that runs nothing
  • Signed and signed off (eng/dco.cs ok)
  • A traceability record exists
  • dotnet run eng/ci.cs passes locally: not run whole; the server, protocol and Web UI suites, api freeze, required-checks, decision-sets, issue-refs, dco and model-names were run

🤖 Generated with Claude Code

https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR

… a signal

Two tests through the real host hold the interval loop's resume after it
has picked a dataset as Moving, apply a cancel, then release it:

- a cancel by apply, with a catalogue that translates: the resume must
  neither freeze again nor finish the move;
- the move's own cancel, for a catalogue that does not translate (the
  path of the flaky test): the resume must commit nothing.

They fail on the code as it is, the first because the resume finishes
the cancelled move to Moved, the second because it freezes and cancels
again. Neither waits on time.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Hafeok and others added 4 commits November 19, 2026 15:35
The fleet's interval loop picked a dataset as Moving from its applied
snapshot and ran the move with no caller. When a request had cancelled
the move meanwhile, the run read InDataset and froze the catalogue again
(EventMoves.cs:236), so a catalogue write after an acknowledged cancel
was refused 409 event-catalogue-moved; a catalogue that translates was
even moved to Moved, which is final. The same happened to a request's
run whose cancel went stale.

Now, as ADR 0167 point 3 says, the resume only finishes a move:
- a run with no caller never freezes;
- a run that has seen the move in progress, by its own freeze or by
  reading Moving, and then finds InDataset commits nothing more. A
  request's run answers 409 event-move-cancelled, a new problem type
  with a page and "What to do" (ADR 0242), which the client does not
  retry; its own untranslatable catalogue's cancel that lost to an
  equal one answers the same 422 as before;
- a cancel whose write came back with a problem answers that problem,
  not "cancelled".

Two test seams on Fleet, null in production: Advancing, awaited at the
start of each advance, where an event move's run steps; and
RefusingManagementWrites, which makes the system dataset's validator
refuse. Two more tests use them: a move cancelled under its request,
and a cancel the move could not commit.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A cancel the system writer has answered was not in force at a dataset's
writer that is another instance until that instance's next interval
(1 s by default): a catalogue write there was refused 409
event-catalogue-moved, which the client does not retry, after the
caller was told the move was cancelled.

What it adds to a write while a dataset is Moving here: one advance of
the system follower before the write's grants are pinned, the step the
host already takes when the dataset's log records a later position
(ADR 0165 point 5). That is one read of the store: on a follower of the
system dataset, a probe of its next log slot; on its writer on an object
store, its writer record; on a fleet of one, none. Then the datasets are
reconciled to the declarations, under the advance's lock, so writes to
a moving dataset take that lock one at a time. Writes to a dataset that
is not Moving are unchanged.

What a write gets when the management plane cannot be read then: the
advance fails and the grants stay at the last applied position (ADR 0163
point 8). A catalogue write to a dataset that was Moving there is
refused 503 management-behind, which the client already sends again
(ADR 0246 point 10), and never 409 event-catalogue-moved, which it does
not. Every other write is decided as before. Moved is final, so a
catalogue write to a moved dataset is still 409.

EventCatalogueValidator takes the catalogue's state, Open, Frozen or
Unconfirmed (EventCatalogueState, unshipped API), in place of a bool.

Two tests with two fleets over a store in memory: the write after the
answered cancel is accepted, and with the store unreadable it is
management-behind. Both fail without this change, with 409
event-catalogue-moved.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…ce applied

Found by the model-based test of the event move. The move had
committed its own cancel and not yet applied it, waiting for the
advance's lock; an operator's cancel by apply then equalled the head,
so the system dataset answered NoChange, and the write was answered
204 without an advance. This instance was still at the freeze, so a
catalogue write after that 204 was refused 409 event-catalogue-moved.

A management write answered NoChange now advances the plane before it
answers, as a committed one does: its answer says the head is so, and
the head is then in force here. A test holds the move between its
cancel's commit and its advance, and applies the operator's cancel.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…erleavings

One server per schedule. Two actors run the move's code and are held
between its steps by signals: one round of the interval loop
(Fleet.TickAsync, the loop's body, which RunAsync now calls unchanged)
and a move request. Between their steps a schedule freezes and cancels
by apply, writes to the catalogue, and switches the system dataset's
refusal of writes on or off; the catalogue translates or does not.
After every step:

1. the declarations never move after an acknowledged cancel, unless a
   move was started since;
2. with no move in progress they are in-dataset, and a catalogue write
   is accepted;
3. an answer that says the move was cancelled holds when it is given.

40 schedules by default; VARVE_EVENT_MOVE_SCHEDULES raises it.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A test that fails now and then is found by running it many times in the
conditions it failed in, with the whole output of every failure kept
(#254). eng/repeat.cs builds a test project once in Release and runs
one xUnit filter N times; with --load, beside a competing Release build
of the solution from a second checkout and one busy thread per
processor. A filter that runs no test exits 2.

The nightly workflow runs it as the job `repeat (<os>)`, on a dispatch
with run: repeat, on any branch, with the project, filter, count, OS
and load as inputs. It never runs on the schedule, a push or a pull
request, and it is not a required check; eng/required-checks.cs refuses
a required name only a workflow not run on pull requests reports, which
was proven by requiring `repeat (ubuntu-latest)`:

  FAIL: 1 required check(s) nothing can satisfy, or anything can (ADR 0088):
    'repeat (ubuntu-latest)' is required, and no job in workflows/ reports it

docs/testing.md §11 says how to use it, and lists the event move's
model-based test with its variable.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Hafeok and others added 2 commits November 19, 2026 15:59
…t with a cancel

The model let any move request in flight at an acknowledged cancel move
after it, which also excused a request that had frozen, was cancelled
under it, and froze again (row 5 of #254's plan). A request is
concurrent with the cancel only while it has not read the state: it is
held at its first pause, before its first read. Once stepped past it,
the cancel closes its move. 1,000 schedules ran clean.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A dated amendment to ADR 0167 (ADR 0068), filed unaccepted (ADR 0066):
the resume finishes a move and never starts one; a cancel found under a
run stands, and a request's run answers 409 event-move-cancelled; a
cancel is answered cancelled only once in force here, a management
write answered with no change included; a write to a moving dataset is
decided after one advance, and is 503 management-behind for a catalogue
graph when the plane cannot be read.

Three keys added to the decision set with no acceptance:
ResumeOnlyFinishesAMove, CancelIsAnsweredOnlyWhenInForce and
MovingWriteAdvancesFirst. The problem page of event-catalogue-moved and
the operator guide's cancelling section say what a caller sees now.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Hafeok and others added 2 commits November 19, 2026 16:08
…s for

ManagementEndpoints and the event move's local writes each advanced the
plane after a write to the system dataset, by their own code, and only
the first did it after a write that changed nothing. Both now call
ManagementAnswers.AppliedAsync: after a write that committed or changed
nothing, this instance applies the head before the answer. For the
move's own writes that adds an advance after a write with no change,
which the move did not need and does no harm, so that the rule has one
place (ADR 0167, amended for #254).

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Whether a write advances the system follower before its grants are
pinned, the dataset recording a later position (ADR 0165 point 5) or
its event declarations moving here (ADR 0167, amended for #254), is
MovingDatasetWrites.AdvanceFirst, which the host's management-position
step calls. Behaviour is unchanged.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Hafeok and others added 4 commits November 19, 2026 16:42
Found by eng/repeat.cs --load after the first fixes, 1 run in 300: the
flaky test's move answered 409 where its own cancel gives 422. The
interval loop resumed the move while the request ran it; the loop judged
the catalogue and cancelled first, and the request, which had seen the
move in progress but not judged it, answered 409 event-move-cancelled,
"cancelled by another request". The cancel stood; the reason misled.

Now the host keeps one run of a dataset's move at a time. A move
request waits for a run in progress (IManagementPlane.EnterMoveAsync),
so its answer is about its own run; the interval loop's resume does not
wait (TryEnterMove), and leaves the move to the run that holds it. 409
event-move-cancelled is left to a cancel by apply. Both are default
members that let every run proceed, unshipped API in Varve.Protocol.
The Fleet keeps one semaphore per dataset; a waiting request passes
through the test seam, which now says where it holds a flow.

A test holds a move request after its freeze and runs a round of the
loop: it commits nothing, and the request answers 422. Without the gate
the loop's cancel lands under the request.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…swer gives

A fourth invariant: 409 event-move-cancelled only when a cancel by apply
was acknowledged while the move request was in flight, and 422
"cancelled" only for a catalogue that does not translate. Without the
one-run gate it fails in 6 runs of 6. A move request waiting for a run
of the move has read nothing, so it stays concurrent with a cancel
until it is let go from an advance.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
The amendment's first ruling, and ResumeOnlyFinishesAMove's statement,
filed unaccepted in this pull request, now say that the instance runs
one move of a dataset at a time: a request waits, the resume never.
docs/testing.md gives the model's fourth invariant.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
The api freeze gate refused the replaced constructor: it shipped in
v0.1.0-preview.22, so removing it is a breaking change (CP0002). The
constructor with Func<bool> stays, true meaning Frozen, beside the one
with EventCatalogueState, which the server uses. Varve.Protocol is only
grown.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Hafeok and others added 2 commits November 19, 2026 17:25
ResumeOnlyFinishesAMove on Fleet, the interval loop that resumes a move;
CancelIsAnsweredOnlyWhenInForce on ManagementAnswers, the advance a
management write's answer waits for; MovingWriteAdvancesFirst on
MovingDatasetWrites, the host's decision to advance before it authorises
a write. The keys are filed without acceptance, so each citation is
CS0618 and the build is red on that alone until the maintainer accepts
them (ADR 0066). The head before this commit, 1cd717d, is the one
tested; with -p:WarningsNotAsErrors=CS0618 nothing else is left:

  warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.CancelIsAnsweredOnlyWhenInForce' is obsolete
  warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.ResumeOnlyFinishesAMove' is obsolete
  warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.MovingWriteAdvancesFirst' is obsolete
  0 Error(s)

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
docs/sessions/2026-11-event-move-race.md: the cause, the reproduction
rates before and after, the move's transitions, the interleavings the
model-based test found, the durations per OS, and what is left.
docs/traceability/2026-11-19-issue-254-event-move-race.md: the prompt,
the maintainer's answer, and where the report is.

Closes #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>

Hafeok commented Nov 19, 2026

Copy link
Copy Markdown
Owner Author

Head 716f791: CI run 44577622689 is red on CS0618 alone, the ADR 0066 shape, and agent review is green. I read every failed job:

  • 22 jobs failed, each with error CS0618 and no other compiler error and no test failure. That covers both build jobs, the devcontainer pipeline, api freeze, conformance ×2, native aot ×5, auth ×2, storage, fleet and handover (services), samples ×2, the Web UI browser tests, container ×2, and the decision report. A build stops at the first citation, ManagementAnswers in Varve.Protocol, so the two citations in Varve.Server are not reached.
  • 2 jobs were cancelled with no runner, durability (macos-latest) and native aot (macos-15): "The job was not acquired by Runner of type hosted even after multiple attempts" (macOS arm64 capacity). They are not re-run, since they would reach the same CS0618.

The unaccepted keys are EventDeclarationsMoveToTheManagementPlane.ResumeOnlyFinishesAMove, .CancelIsAnsweredOnlyWhenInForce and .MovingWriteAdvancesFirst.

Head 1cd717d, the one before the citations: every job is green, including native aot (macos-15) on its re-run, which got a runner. The gate's 500 runs under load passed there: 0 failures on ubuntu-latest and 0 on windows-latest.

The pull request stays a draft. Accepting the three keys turns it green.


Generated by Claude Code

The maintainer asked how much of the fleet suite's 1 min 38 s to
1 min 59 s, one run each, is #254's fix. The fleet suite needs Azurite
and RustFS, and the commit before the fix has no repeat job, so the job
now takes:

- repeat-compare: a second commit, built from a checkout of its own and
  run alternately with the first, the side that goes first changing
  every round, so the runner's drift falls on both alike; the summary
  gives each side's failures and mean, median, least and most time;
- repeat-at: the commit to measure when the branch's head does not
  build, as a head red on an unaccepted decision's CS0618 (ADR 0066);
- repeat-services: the object stores of the fleet (services) job.

Each run's TRX is kept in the artifact, so a difference can be put on
the tests that make it. Still dispatch only, never required.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
At the maintainer's request: five runs a side put the fix at +3.3 s on
109 s, within the spread, all in the active-active property; step by
step against a control, no part reproduces a cost. #380 is the one
transport failure of that property found on the way.

Refs #254

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR
Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>

Hafeok commented Nov 19, 2026

Copy link
Copy Markdown
Owner Author

Head 4d31f20: CI run 44601289730 is red on CS0618 alone, as at 716f791. I read all 23 failed jobs: each has error CS0618, no other compiler error, and no test failure. No job was cancelled this time.

The two commits since 716f791:

The tested product head is still 1cd717d.


Generated by Claude Code

This branch was successfully deployed

1 active deployment
gates — 4d31f204 Deployed Nov 19, 2026 by Hafeok via agent review (judge) #1623
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant