Repository navigation
CI: Add the upgrade E2E test for each release - #2712
Open
Haven Xia (HavenXia) wants to merge 3 commits into
Open
Haven Xia (HavenXia) wants to merge 3 commits into
Haven Xia (HavenXia) wants to merge 3 commits into
Conversation
Haven Xia (HavenXia)
force-pushed
the
upgrade-e2e
branch
from
October 15, 2026 17:52
f2efb5a to
70ac4a4
Compare
Haven Xia (HavenXia)
force-pushed
the
upgrade-e2e
branch
from
October 19, 2026 02:56
74efe66 to
359959c
Compare
Bowei Du (bowei)
added
area/tests
Haven Xia (HavenXia)
force-pushed
the
upgrade-e2e
branch
from
October 19, 2026 20:59
359959c to
c067a30
Compare
The suite takes a cluster running a release with actors on it and upgrades it to the checkout under test in the order of docs/upgrade.md: podcertificate-controller, atelet, the WorkerPool's workerImage, the API server, ate-controller, atenet, the SandboxConfig, then the rest of the system bundle. Every step except the workerImage edit is one ate-setup command, run from a copy of the install record. The suite stands in for the actors' harness and suspends an actor as soon as its worker pod starts terminating. Each step that deploys a component has to move it to new images, and no actor may be CRASHED at any step. Only the pool roll rolls the workers, plus ate-controller if its pod template changed, and only those steps may evict an actor. Afterwards each actor resumes on a worker running the new image with its memory intact. The suite upgrades the cluster it runs on, so it lives outside internal/e2e/suites, where the default e2e run would pick it up. It runs on gVisor only. The workflow runs it for every push to a release branch, against the newest earlier release on that branch, so a patch release is checked before it is tagged. It also runs on demand from a given release. Two installs on one kind cluster are too slow for every PR.
CI requires every e2e run to pass --preview, so the upgrade workflow passes it empty: the previous release is installed with no preview gates. Without it the suite fails before running a test. The suite also expected evictions only for the actors that happened to be running after the atelet step. It now requires both running actors to still be running there, and expects the pool roll to evict exactly those two and the actor resumed onto the spare old worker.
…y mode The skew policy keeps the worker pod template the same within a major, so the suite now fails if the new ate-controller rolls the pool instead of allowing the roll and its evictions. PauseActor is gone and SuspendActor defaults to an asynchronous suspend. The paused actor is now suspended asynchronously, and the suspended actor and the final checks suspend synchronously. The imports move to the go.ate.dev/substrate module path.
Haven Xia (HavenXia)
force-pushed
the
upgrade-e2e
branch
from
October 19, 2026 22:25
c067a30 to
15c7829
Compare
Member
Author
|
green |
Haven Xia (HavenXia)
requested review from
Dmitry Berkovich (dberkov) and
Julian Gutierrez Oschmann (juli4n)
October 20, 2026 17:20
Member
Author
|
Dmitry Berkovich (@dberkov) Julian Gutierrez Oschmann (@juli4n) Adding you both for your API expertise |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds an e2e suite and a workflow that upgrade a running install with actors on it, following
docs/upgrade.mdstep by step.The workflow installs the newest earlier release on the branch with that release's own installer, and the suite starts counter actors that are running, paused, or suspended.
It then runs each runbook step with this checkout's
ate-setupand a copy of the install record: podcertificate-controller, atelet, the WorkerPool'sworkerImage, the API server, ate-controller, atenet, the SandboxConfig, thendeploy ate-system --setup-csi=none. While it runs, it acts as the actors' harness and suspends an actor as soon as its worker pod starts terminating.After each step, the step's component has to run new images and no actor may be
CRASHED. Only the pool roll may roll the workers and evict actors, plus ate-controller if its pod template changed. At the end, every actor, plus one created after the upgrade, has to resume on a new worker with its counter intact.Cadence
Patch releases are cut by merging fixes into
release-X.Yand tagging there. The workflow runs on every push to arelease-X.Ybranch and upgrades from the newest earlier tag on that branch, so the commit that becomes v0.6.1 is tested asv0.6.0→v0.6.1before it is tagged.A push with no earlier tag yet, such as the branch cut itself, is skipped with a notice. It also runs on demand.
After GA 1.0 we supports upgrading to the next minor, for example 1.1.x → 1.2.y, so a push to
release-1.2should also upgrade from the newest 1.1 release. That second lane comes with 1.0.Tested locally on kind:
unknown service atelet.AteomHerder, because rename ateom -> ateworker #2254 renamed the atelet and worker RPCs. That is a pre-1.0 minor upgrade, which the runbook already says needs a reinstall, so the suite caught a real break.Not covered yet: rollback, and a micro-VM lane
Part of #1784