Repository navigation
Conversation
… a signal Two tests through the real host hold the interval loop's resume after it has picked a dataset as Moving, apply a cancel, then release it: - a cancel by apply, with a catalogue that translates: the resume must neither freeze again nor finish the move; - the move's own cancel, for a catalogue that does not translate (the path of the flaky test): the resume must commit nothing. They fail on the code as it is, the first because the resume finishes the cancelled move to Moved, the second because it freezes and cancels again. Neither waits on time. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
The fleet's interval loop picked a dataset as Moving from its applied snapshot and ran the move with no caller. When a request had cancelled the move meanwhile, the run read InDataset and froze the catalogue again (EventMoves.cs:236), so a catalogue write after an acknowledged cancel was refused 409 event-catalogue-moved; a catalogue that translates was even moved to Moved, which is final. The same happened to a request's run whose cancel went stale. Now, as ADR 0167 point 3 says, the resume only finishes a move: - a run with no caller never freezes; - a run that has seen the move in progress, by its own freeze or by reading Moving, and then finds InDataset commits nothing more. A request's run answers 409 event-move-cancelled, a new problem type with a page and "What to do" (ADR 0242), which the client does not retry; its own untranslatable catalogue's cancel that lost to an equal one answers the same 422 as before; - a cancel whose write came back with a problem answers that problem, not "cancelled". Two test seams on Fleet, null in production: Advancing, awaited at the start of each advance, where an event move's run steps; and RefusingManagementWrites, which makes the system dataset's validator refuse. Two more tests use them: a move cancelled under its request, and a cancel the move could not commit. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A cancel the system writer has answered was not in force at a dataset's writer that is another instance until that instance's next interval (1 s by default): a catalogue write there was refused 409 event-catalogue-moved, which the client does not retry, after the caller was told the move was cancelled. What it adds to a write while a dataset is Moving here: one advance of the system follower before the write's grants are pinned, the step the host already takes when the dataset's log records a later position (ADR 0165 point 5). That is one read of the store: on a follower of the system dataset, a probe of its next log slot; on its writer on an object store, its writer record; on a fleet of one, none. Then the datasets are reconciled to the declarations, under the advance's lock, so writes to a moving dataset take that lock one at a time. Writes to a dataset that is not Moving are unchanged. What a write gets when the management plane cannot be read then: the advance fails and the grants stay at the last applied position (ADR 0163 point 8). A catalogue write to a dataset that was Moving there is refused 503 management-behind, which the client already sends again (ADR 0246 point 10), and never 409 event-catalogue-moved, which it does not. Every other write is decided as before. Moved is final, so a catalogue write to a moved dataset is still 409. EventCatalogueValidator takes the catalogue's state, Open, Frozen or Unconfirmed (EventCatalogueState, unshipped API), in place of a bool. Two tests with two fleets over a store in memory: the write after the answered cancel is accepted, and with the store unreadable it is management-behind. Both fail without this change, with 409 event-catalogue-moved. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…ce applied Found by the model-based test of the event move. The move had committed its own cancel and not yet applied it, waiting for the advance's lock; an operator's cancel by apply then equalled the head, so the system dataset answered NoChange, and the write was answered 204 without an advance. This instance was still at the freeze, so a catalogue write after that 204 was refused 409 event-catalogue-moved. A management write answered NoChange now advances the plane before it answers, as a committed one does: its answer says the head is so, and the head is then in force here. A test holds the move between its cancel's commit and its advance, and applies the operator's cancel. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…erleavings One server per schedule. Two actors run the move's code and are held between its steps by signals: one round of the interval loop (Fleet.TickAsync, the loop's body, which RunAsync now calls unchanged) and a move request. Between their steps a schedule freezes and cancels by apply, writes to the catalogue, and switches the system dataset's refusal of writes on or off; the catalogue translates or does not. After every step: 1. the declarations never move after an acknowledged cancel, unless a move was started since; 2. with no move in progress they are in-dataset, and a catalogue write is accepted; 3. an answer that says the move was cancelled holds when it is given. 40 schedules by default; VARVE_EVENT_MOVE_SCHEDULES raises it. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A test that fails now and then is found by running it many times in the conditions it failed in, with the whole output of every failure kept (#254). eng/repeat.cs builds a test project once in Release and runs one xUnit filter N times; with --load, beside a competing Release build of the solution from a second checkout and one busy thread per processor. A filter that runs no test exits 2. The nightly workflow runs it as the job `repeat (<os>)`, on a dispatch with run: repeat, on any branch, with the project, filter, count, OS and load as inputs. It never runs on the schedule, a push or a pull request, and it is not a required check; eng/required-checks.cs refuses a required name only a workflow not run on pull requests reports, which was proven by requiring `repeat (ubuntu-latest)`: FAIL: 1 required check(s) nothing can satisfy, or anything can (ADR 0088): 'repeat (ubuntu-latest)' is required, and no job in workflows/ reports it docs/testing.md §11 says how to use it, and lists the event move's model-based test with its variable. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…t with a cancel The model let any move request in flight at an acknowledged cancel move after it, which also excused a request that had frozen, was cancelled under it, and froze again (row 5 of #254's plan). A request is concurrent with the cancel only while it has not read the state: it is held at its first pause, before its first read. Once stepped past it, the cancel closes its move. 1,000 schedules ran clean. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
A dated amendment to ADR 0167 (ADR 0068), filed unaccepted (ADR 0066): the resume finishes a move and never starts one; a cancel found under a run stands, and a request's run answers 409 event-move-cancelled; a cancel is answered cancelled only once in force here, a management write answered with no change included; a write to a moving dataset is decided after one advance, and is 503 management-behind for a catalogue graph when the plane cannot be read. Three keys added to the decision set with no acceptance: ResumeOnlyFinishesAMove, CancelIsAnsweredOnlyWhenInForce and MovingWriteAdvancesFirst. The problem page of event-catalogue-moved and the operator guide's cancelling section say what a caller sees now. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…s for ManagementEndpoints and the event move's local writes each advanced the plane after a write to the system dataset, by their own code, and only the first did it after a write that changed nothing. Both now call ManagementAnswers.AppliedAsync: after a write that committed or changed nothing, this instance applies the head before the answer. For the move's own writes that adds an advance after a write with no change, which the move did not need and does no harm, so that the rule has one place (ADR 0167, amended for #254). Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Whether a write advances the system follower before its grants are pinned, the dataset recording a later position (ADR 0165 point 5) or its event declarations moving here (ADR 0167, amended for #254), is MovingDatasetWrites.AdvanceFirst, which the host's management-position step calls. Behaviour is unchanged. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
Found by eng/repeat.cs --load after the first fixes, 1 run in 300: the flaky test's move answered 409 where its own cancel gives 422. The interval loop resumed the move while the request ran it; the loop judged the catalogue and cancelled first, and the request, which had seen the move in progress but not judged it, answered 409 event-move-cancelled, "cancelled by another request". The cancel stood; the reason misled. Now the host keeps one run of a dataset's move at a time. A move request waits for a run in progress (IManagementPlane.EnterMoveAsync), so its answer is about its own run; the interval loop's resume does not wait (TryEnterMove), and leaves the move to the run that holds it. 409 event-move-cancelled is left to a cancel by apply. Both are default members that let every run proceed, unshipped API in Varve.Protocol. The Fleet keeps one semaphore per dataset; a waiting request passes through the test seam, which now says where it holds a flow. A test holds a move request after its freeze and runs a round of the loop: it commits nothing, and the request answers 422. Without the gate the loop's cancel lands under the request. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
…swer gives A fourth invariant: 409 event-move-cancelled only when a cancel by apply was acknowledged while the move request was in flight, and 422 "cancelled" only for a catalogue that does not translate. Without the one-run gate it fails in 6 runs of 6. A move request waiting for a run of the move has read nothing, so it stays concurrent with a cancel until it is let go from an advance. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
The amendment's first ruling, and ResumeOnlyFinishesAMove's statement, filed unaccepted in this pull request, now say that the instance runs one move of a dataset at a time: a request waits, the resume never. docs/testing.md gives the model's fourth invariant. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
The api freeze gate refused the replaced constructor: it shipped in v0.1.0-preview.22, so removing it is a breaking change (CP0002). The constructor with Func<bool> stays, true meaning Frozen, beside the one with EventCatalogueState, which the server uses. Varve.Protocol is only grown. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
ResumeOnlyFinishesAMove on Fleet, the interval loop that resumes a move; CancelIsAnsweredOnlyWhenInForce on ManagementAnswers, the advance a management write's answer waits for; MovingWriteAdvancesFirst on MovingDatasetWrites, the host's decision to advance before it authorises a write. The keys are filed without acceptance, so each citation is CS0618 and the build is red on that alone until the maintainer accepts them (ADR 0066). The head before this commit, 1cd717d, is the one tested; with -p:WarningsNotAsErrors=CS0618 nothing else is left: warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.CancelIsAnsweredOnlyWhenInForce' is obsolete warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.ResumeOnlyFinishesAMove' is obsolete warning CS0618: 'EventDeclarationsMoveToTheManagementPlane.MovingWriteAdvancesFirst' is obsolete 0 Error(s) Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
docs/sessions/2026-11-event-move-race.md: the cause, the reproduction rates before and after, the move's transitions, the interleavings the model-based test found, the durations per OS, and what is left. docs/traceability/2026-11-19-issue-254-event-move-race.md: the prompt, the maintainer's answer, and where the report is. Closes #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
|
Head
The unaccepted keys are Head The pull request stays a draft. Accepting the three keys turns it green. Generated by Claude Code |
The maintainer asked how much of the fleet suite's 1 min 38 s to 1 min 59 s, one run each, is #254's fix. The fleet suite needs Azurite and RustFS, and the commit before the fix has no repeat job, so the job now takes: - repeat-compare: a second commit, built from a checkout of its own and run alternately with the first, the side that goes first changing every round, so the runner's drift falls on both alike; the summary gives each side's failures and mean, median, least and most time; - repeat-at: the commit to measure when the branch's head does not build, as a head red on an unaccepted decision's CS0618 (ADR 0066); - repeat-services: the object stores of the fleet (services) job. Each run's TRX is kept in the artifact, so a difference can be put on the tests that make it. Still dispatch only, never required. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
At the maintainer's request: five runs a side put the fix at +3.3 s on 109 s, within the spread, all in the active-active property; step by step against a control, no part reproduces a cost. #380 is the one transport failure of that property found on the way. Refs #254 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR Signed-off-by: Emil Okkels Klein <emil@okkels-klein.dk>
|
Head The two commits since
The tested product head is still Generated by Claude Code |
Completes: #254
What this changes, and why
#254:
EventMoveTests.A_catalogue_that_does_not_translate_one_to_one_cancels_the_move_and_writes_reopenfailed now and then with409 event-catalogue-movedafter a cancel. The server answered the cancel only once it was in force. The fleet's interval loop then undid it: it had picked the dataset asMoving, and its run of the move tookvarve:InDatasetas a move to start and froze the catalogue again. With a catalogue that translates, that run went on toMoved, which is final. The cause, the plan and the maintainer's answers are on #254.The fixes, each its own commit, with its own tests:
Moving, by its own freeze or by reading it, commits nothing more when it then findsInDataset.409 event-move-cancelled, which has a page and "What to do" (ADR 0242). The client does not retry it.Movinghere advances the system follower once before its grants are pinned.Movingis503 management-behind, which the client already sends again (ADR 0246 point 10). It is never409 event-catalogue-moved.Movedstays409.NoChangeis applied before its answer. The model-based test found this; the plan did not list it.IManagementPlane.EnterMoveAsyncandTryEnterMove, default members). A request waits for a run in progress; the loop's resume does not.eng/repeat.cs --loadfound this after the first fixes, and so did the hosted 500 on Windows.CS0618alone (ADR 0066). The tested head is1cd717d.repeatjob innightly.yml, runningeng/repeat.cs, described in docs/testing.md §11. It is never a required check, and the required-checks gate refuses it as one (proven).Unaccepted decisions cited (ADR 0066), each by
Set.Keyat its citation site:EventDeclarationsMoveToTheManagementPlane.ResumeOnlyFinishesAMoveonsrc/Varve.Server/Fleet.cs;EventDeclarationsMoveToTheManagementPlane.CancelIsAnsweredOnlyWhenInForceonsrc/Varve.Protocol/Endpoints/ManagementAnswers.cs;EventDeclarationsMoveToTheManagementPlane.MovingWriteAdvancesFirstonsrc/Varve.Server/MovingDatasetWrites.cs.Public API.
Varve.Protocol.Clientis unchanged.Varve.Protocolis grown, not broken (api freezegreen at1cd717d):ProblemType.EventMoveCancelled;EventCatalogueState;EventCatalogueValidatorconstructor;IManagementPlane.What was measured
c754bcb, this container627ca92/a2478131cd717d, the gatea3af2c8failed on exactly the two new tests, on Linux, Windows and the devcontainer.c754bcb) and after (1cd717d), one run each:IdleFleetTestspassed, unchanged.What was found
AppliedAsyncgives up silently.NoChangeanswer. Repeating under load found a fourth: the loop and a request running one move together. The model catches it with its fourth invariant (the answer's reason).1cd717d,native aot (macos-15)was cancelled: no hosted runner took it (macOS arm64 capacity). It was re-run.a12f5e4) refers in a doc comment to a test class the next commit adds; only the head is built by CI.What is left
docs/sessions/2026-11-event-move-race.md, and the record isdocs/traceability/2026-11-19-issue-254-event-move-race.md.Checklist
Refs #254, and the lastCloses #254(eng/issue-refs.csok)PublicAPI.Unshipped.txtinVarve.ProtocolFleetrepeat (ubuntu-latest)as required;eng/repeat.csexits 2 for a filter that runs nothingeng/dco.csok)dotnet run eng/ci.cspasses locally: not run whole; the server, protocol and Web UI suites,api freeze,required-checks,decision-sets,issue-refs,dcoandmodel-nameswere run🤖 Generated with Claude Code
https://claude.ai/code/session_01NbWSYShh3XyxMHBQtApVnR