feat(runner-image): carry the Runner, KVM userland and guest assets in a container image (RIG-3720) - #1164
Merged
Merged
Conversation
…lity (RIG-3336) Decides whether the Compass Runner is containerized for Kubernetes, exactly what the pod is granted, and the generic object contract any conformant cluster can run. Containerize as the DaemonSet delivery + lifecycle unit, rootless with a scoped /dev/kvm device, never `privileged: true`. The ruling is explicitly contingent on a real-hardware spike (R7): a rootless HOST user has affordances a pod does not (setuid newuidmap, unrestricted unshare, no seccomp filter), so "the Runner already runs rootless" does not establish that a locked-down pod can boot the composition. If it cannot, the container-vs-host tradeoff reopens rather than a capability being granted. Three findings worth the reader's attention: - Guest RAM is POD RAM. cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup. With low requests the pod is Burstable with a large overage, which the kubelet ranks first for node-pressure eviction -- and evicting it kills every session on the node. - Container restart implies full session teardown, forced rather than elected: without shareProcessNamespace the Runner is pid 1, so its death tears down the pid namespace and reaps every VMM descendant. - The hostPath char-device route is non-functional, not merely worse posture ([INFERENCE] on cgroup-v2 device-controller behaviour, flagged as such and given a negative control in the spike). Scope is the core capability only. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred per the public/private boundary; the record cites the boundary convention rather than restating it. spike-findings.md records the verification's questions and negative controls BEFORE anyone runs them, so results cannot pick the questions. Each item states the control that must fail; a control that unexpectedly passes shrinks the contract. Mints DL-357. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…RIG-3336) Six findings from the review loop, all real. - The hostPath-over-emptyDir justification was wrong. emptyDir is pod-scoped and DOES survive a container restart, so the axis I contrasted did not distinguish them. Rejustified on outliving the POD: pod recreation and node reboot are when a previous pod's stale pidfile must still be reachable for orphan-reaping. - A DAC permission denial is EACCES, not EPERM (verified: errno 13). This is load-bearing rather than cosmetic, because the errno is what separates the two layers the record is at pains to distinguish -- cgroup device-controller denial is EPERM, filesystem denial is EACCES. Spike S3 now records the errno verbatim and states that seeing EPERM would mean the cgroup layer denied, confounding S3 with S2. - Guest RAM is charged to the pod's cgroup because the VMM is an ordinary child in that cgroup, NOT because of `--memory shared=on`. shared=on's consequence here is an accounting one: guest memory is shared mappings, so summing RSS across cloud-hypervisor, virtiofsd and the Runner double-counts it. Size against PSS or the configured guest total. - Fully-absent requests make the pod BestEffort, not Burstable. The eviction conclusion is unchanged and slightly stronger. - A probe-driven restart is the same node-wide session teardown as a rollout, but nothing rate-limits it the way maxUnavailable bounds a rollout. The contract now requires a conservative or absent liveness probe, and a terminationGracePeriodSeconds sized to the reap budget. - A node drain is equally session-affecting and was covered by nothing, since it is not a delivery mechanism. Named in the constraints, with the note that a fleet PDB bounds concurrent drains but cannot protect a pod on its own draining node. Gates on this tree: root:markdownlint rc=0, orion-ref-gate rc=0, ledger gate rc=0 (OK -- 319 rows, 133 headers). Co-authored-by: Matt Wilkinson <matt@rigel.build>
…h main (RIG-3336) CI caught a duplicate ledger id: main minted its own DL-357 (RIG-3561's pr_number wire-field row, #1037) while this branch was in review, so two rows claimed 357 and design-ledger-gate failed. My local gate had passed because the clone's main was 9 commits stale -- the id was genuinely free when I allocated it and taken by the time I pushed. A ledger id is only free at the moment the gate runs against current main, so it needs re-checking after any merge, not just at allocation. Renumbered to DL-358 (free on current main) and synced the record's three references: the Ledger-impact header, R6, and the tasks table. Merges current main, which also swaps the markdown gate from markdownlint-cli2 to rumdl (RIG-3663, #1094) -- verified via root:ci on the merged tree rather than the retired task. Gates on the merged tree: design-ledger-gate rc=0 (OK -- 320 rows, 133 headers), orion-ref-gate rc=0, root:ci rc=0. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…gency (RIG-3336) The record read as though containerizing the Runner might not work. It does not not work. The composition already boots on Linux with /dev/kvm -- the frozen microVM CI/dev enablement record runs KVM-backed boot tests as a required leg on GitHub Actions ubuntu-latest. What a pod adds over that is confinement: a cgroup device controller, a seccomp filter, a memory cgroup. So R8 asks which grants the confinement makes necessary, not whether microVMs run in containers. Every plausible outcome costs at most a wider pod spec this record already specifies -- a seccomp profile, a device plugin, a supplemental gid. The valuable outcome is a smaller contract, since a grant that proves inert drops out. The one result that would reopen the container-vs-host ruling is a requirement for a Linux capability or privileged: true, and no identified mechanism in the composition needs one. That is now stated as the low-probability tail rather than the expected case. Changes: - OQ-1 reframed from "is the zero-privilege pod sufficient" (load-bearing feasibility) to "which grants does confinement make necessary" (non-load-bearing spec narrowing). OQ-2 stays load-bearing -- the seccomp profile is the only item with a real cost attached -- but a needed profile is now named as a bounded cost, not a setback. - Approach, Global Constraints, Privilege shape, R8 and the spike header all cite the established KVM baseline instead of treating the boot as unproven. - S1 rescoped from "does the composition boot" to "does the confinement hold". - All 8 "contingent on R8" phrasings removed. Also supersedes a stale measurement: the frozen enablement record reads /dev/kvm as crw-rw-rw-, but this box measures crw-rw---- root:kvm (2026-09-12). The mode decides whether the supplementalGroups grant is required or inert, so R8 must record the node's own mode. Gates: design-ledger-gate rc=0 (320 rows, 133 headers), orion-ref-gate rc=0, root:ci rc=0. Co-authored-by: Matt Wilkinson <matt@rigel.build>
The reframe commit corrected design.md but left two things stale. The DL-358 ledger row still said the ruling was "explicitly CONTINGENT" on the spike and that the container-vs-host tradeoff "reopens" if it disappoints. That is the framing the reframe removed, sitting in the durable one-line summary a reader hits before the record. Rewrite it as a grant-set question, cite the KVM CI leg as the reason feasibility is settled, and state the capability/privileged tail as the only reopening outcome. Also restate the seccomp profile as specified-but-open (OQ-2) rather than flatly required. Prose referred to the privilege spike as R8 in 17 places; the task is R7 and no R8 exists, so every "R8 verifies it" pointed at nothing.
…age (RIG-3336) R1 said "a nix-built container image project ... mirroring the existing agent-image lane". That contradicts the private monorepo's frozen first-party-image-builds spec, which splits the two mechanisms by what the image's RUNTIME is rather than by what built the artifact: an image that IS a Nix environment (a CI step image, a dev/agent shell) earns nix2container, while a prebuilt application on a minimal base is a Dockerfile built by rootless BuildKit. The spec forecloses the exact inference R1 made — a first-party image does not earn the nix path "merely because Nix built it or it ships a compiled binary" — and rejects nix2container for app images on cost. The Runner is a prebuilt application: a static CGO_ENABLED=0 binary that exec's three userland binaries and runs no toolchain at runtime. agent-image sits on the other row by name, because that container's job is to be a toolchain; sharing the *-image/ directory shape is not sharing the mechanism. Distroless suits a managed multi-tenant cluster, but the KVM userland comes from nixpkgs and carries store-path interpreter/rpath references, so R1 must settle how those binaries run on a minimal base. R1's test cycle now asserts each carried binary EXECUTES, which a layer-contents check alone would miss. R2 gains the build mechanism and the digest-pinning contract: GHCR has no server-side tag immutability, so R3 pins repo@sha256, never a tag.
…n a container image (RIG-3720) R1 of the Runner containerization record. A Dockerfile on distroless, built by rootless BuildKit — the ruled mechanism for a prebuilt application. Nix still builds every carried artifact; it is the builder, never the runtime. The image carries the compass-runner binary, the KVM userland it exec's (cloud-hypervisor, virtiofsd, passt) and the three guest assets the microVM backend direct-boots, so a Runner pod needs no host toolchain. WHY DISTROLESS STATIC AND NOT :base. The carried binaries are dynamically linked but nix-closed: each names an absolute /nix/store interpreter and resolves every NEEDED library through its own RPATH, so the closure supplies its own ld.so and glibc and the base is never consulted for a library. That makes the base's libc dead weight. Measured, not assumed: with the closure staged all four binaries execute on distroless-static; with only cloud-hypervisor's own store path copied and its glibc omitted, the same exec fails "missing dynamic library" — the control proving the loader comes from the closure, not the base. Two consequences that are easy to get wrong, both encoded in the lane: - The staged context is the TRANSITIVE closure, not the five roots. A context missing the shared glibc yields an image whose binaries cannot exec at all. - The entrypoint is a RELATIVE symlink inside store/. An absolute symlink at the context root dangles on the build host, and BuildKit checksums context entries before any COPY runs, so it fails the build outright rather than resolving later inside the image. The realise-and-stage logic is TypeScript (the no-bash-gate task), split pure-core/thin-runner like the microvm-boot-test lane: build-core.ts holds the out-path parsing, the artifact->build-arg mapping and the closure-root selection, covered by 13 unit tests. The image build itself is runInCI:false — it needs a buildkitd, which no compass CI step carries until R2 — so the gate coverage is that suite, which is where a mapping drift is actually caught. Verified end-to-end on a rootless buildkitd: the lane builds the real image; its entrypoint runs as uid 65532; all four binaries execute in-image; passt resolves by name off PATH; all three guest assets match their host sizes byte for byte.
rigel-mintaka
added this pull request to stack #1165
September 12, 2026 19:10
|
Compass engineering docs preview: https://compass-managed-rig-3720-run.compass-eng-docs.pages.dev Deployed from |
Base automatically changed from
compass-managed/rig-3336-runner-containerization-capability
to
main
September 12, 2026 19:20
|
😎 Stack merged successfully - details. |
rigel-mintaka
added a commit
that referenced
this pull request
Sep 12, 2026
…3326) DL-358 was claimed on main by #1129 (RIG-3336, Compass Runner containerization) while this record sat in review, and open #1164 claims 358 as well. Main's ledger max is DL-363, so 358/359 were both stale reservations sized against an older main. Renumbered to DL-364/DL-365, above main's max and free across every open PR that touches DECISIONS.md. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…root uid (RIG-3720) Review findings on the R1 lane. The load-bearing one: the image was NOT reproducible, contradicting R1's acceptance and a Dockerfile comment that claimed the COPY layer was "bit-identical to the staged tree". Two builds of identical inputs produced different manifest digests. Two independent causes, both measured rather than reasoned about: - cpSync defaults preserveTimestamps:false, so every staged file got a wall-clock mtime instead of the 1s nix normalises its store to. - BuildKit carries the context's mtimes into the layer tar regardless. Fixing only the first was NOT sufficient: the staged tree hashed bit-identical across runs while the `COPY store` layer digest still moved. The export needs rewrite-timestamp=true plus a fixed SOURCE_DATE_EPOCH. With both, two consecutive builds yield the same digest. The publish lane asserts exactly this, so the property is now real instead of asserted. Also from review: - Drop the RUNNER_BIN build-arg. It was declared and supplied but never consumed — the entrypoint became a staged symlink and the arg was left behind. The new drift test is what catches this class: it parses the Dockerfile's own ARG lines and asserts set-equality with what build-core supplies. Confirmed it fails when an ARG is renamed and passes when restored. - Add an explicit USER 65532:65532 rather than inheriting nonroot from an opaque base digest, so a future digest bump onto the root variant cannot silently produce a root-running container. - Assert the build host is x64. The platform is a manifest LABEL BuildKit applies without checking the staged files; today only flake.nix's systems list keeps them honest, which this lane never reads. - Narrow the build task's inputs from `**/*` to named tracked files: moon's native walker hashes contents directly and does not skip gitignored paths, so the glob would hash the multi-GB closure this task itself writes. - Cut the mechanism rationale from five restatements to one plus pointers, drop forward task references from source, and delete a tautological test that could not fail.
trunk-io Bot
pushed a commit
that referenced
this pull request
Sep 12, 2026
…3326) (#997) * docs(server): forge self-origin notification suppression record (RIG-3326) Design record for suppressing self-origin forge notifications at the notify-router fan-out: an agent no longer wakes on its own COMMENT / REVIEW / OPENED / STATE on a subscribed artifact, matched on owner-qualified Compass handles, failing open on any ambiguity. CHECKS is never suppressed. A suppressed notification advances delivered_revision only when the subscriber was already caught up (CAS-guarded, artifact-scope only), so the reconcile sweep neither resurrects it nor masks a prior undelivered real notification. The owner half of that identity is not on the wire today: AgentAttribution is {agent_handle} only, so the parse discards the owner it already produces and the COMMENT/REVIEW arm would be structurally inert. T0 restores owner_handle and adds an internal ForgeEvent.Actor carrier for STATE (RIG-3331 populates it). Sibling of Record A (forge self-delegate write path, PR #900); depends on RIG-3331 (agent-driven state-transition write op) for the STATE arm to suppress on the real transition actor rather than an author-row proxy. Ledger-impact: adds DL-334 (self-origin notification suppression) and DL-335 (owner_handle carriage, amending the DL-094 clause DL-186 carried forward; DL-186 stays Active). Co-authored-by: Matt Wilkinson <matt@rigel.build> resolve * docs(server): resolve the STATE actor from RIG-3331's memo, and cite symbols not lines (RIG-3326) Two changes to the self-origin suppression record, both consequences of rulings that landed after it was frozen. 1. The STATE actor carrier is RIG-3331's memo, not a field on the event. This record froze the carrier as a new ForgeEvent.Actor field added by T0, on the reasoning that ForgeEvent has no actor slot and COMMENT/REVIEW are symmetric only because CommentRef holds theirs. Matt ruled the opposite on 2026-09-07 (RIG-3331 OQ-1): the actor travels in a durable, tenant-scoped forge_state_transitions memo written at the write-path chokepoint and consumed on match at the notify lane. A provider webhook is the forge's own statement of what happened, and stamping a Compass field onto the struct parsed from it invents an event the forge never sent. So T0 adds no ForgeEvent field. The field would have shipped inert: COMMENT/REVIEW resolve through CommentRef.Agent, OPENED through the DL-055 row, and STATE -- its only remaining client -- now resolves through the memo, leaving no populator and no consumer (rule://no-inert-gating). The fail-open interim is unchanged: a memo miss resolves no actor exactly as a nil field did, so STATE delivers until RIG-3331's op is writing memos. Amends the STATE row of the actor-source matrix, the carrier section, the T0 interface block, the T0 checklist entry, and DL-338's actor-source clause. Also states the dependency direction explicitly: RIG-3331 is the PREREQUISITE, this record is the consumer. 2. Symbol-plus-path citations. Replaces every path:line citation in the record and in DL-338/DL-339 with the symbol form. A line number is stale as soon as the file moves, and the drift is observable on main: PR #916 had to chase six prose citations in a merged record when resolver.go grew a few lines. All 104 were resolved to their enclosing symbol against the source tree and verified present in the cited file; the scrub was scoped to my own rows, leaving DL-064's citation into the vendored fork tree alone rather than resolving it by basename into an unrelated file. Refs RIG-3326 Co-authored-by: Matt Wilkinson <matt@rigel.build> * fix(docs): renumber the suppression record's ledger rows to DL-351/352 (RIG-3326) main gained DL-338 from RIG-3490 after this branch claimed 338/339, so design-ledger-gate:check failed on the merge-queue trial branch with a duplicate ledger id. Move both rows to the free range above main's max. Co-authored-by: Matt Wilkinson <matt@rigel.build> * fix(docs): renumber the suppression record's ledger rows to DL-358/359 (RIG-3326) Another lane claimed DL-351/352 on main (#1092, RIG-3655) while this record sat in review, so the post-merge ledger carried two rows for each id. Main's max is now DL-357; move to DL-358/359. Co-authored-by: Matt Wilkinson <matt@rigel.build> * fix(ui): ignore the e2e visual-run output dir (RIG-3326) `apps/ui/e2e/.gitignore` ignored `/.output/` but not its sibling `/.output-visual/`, so a local visual run's `.last-run.json` was snapshotted into a commit and failed `root:lint` on formatting. Co-authored-by: Matt Wilkinson <matt@rigel.build> * docs(design): renumber suppression ledger rows to DL-364/DL-365 (RIG-3326) DL-358 was claimed on main by #1129 (RIG-3336, Compass Runner containerization) while this record sat in review, and open #1164 claims 358 as well. Main's ledger max is DL-363, so 358/359 were both stale reservations sized against an older main. Renumbered to DL-364/DL-365, above main's max and free across every open PR that touches DECISIONS.md. Co-authored-by: Matt Wilkinson <matt@rigel.build> --------- Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
marked this pull request as ready for review
September 12, 2026 20:18
The CI `setup` job failed at "Generate the concern matrix", blocking every downstream leg. tools/ci-matrix requires each moon project to carry exactly one `ci-group.*` tag and fails loud otherwise — the fail-loud posture that lets moon own discovery instead of a stale YAML enumeration — and the new compass-runner-image project shipped with `tags: []`. ci-group.nix, with the sibling image projects: the real build closure is the nix one those legs already provision. The project contributes no CI task today (`build` is runInCI:false), so the tag places it in the graph rather than adding a leg. `moon ci` does not run the matrix generator, so a green local sweep said nothing about this. Run `bun run tools/ci-matrix/index.ts` when adding a project.
mattwilkinsonn
approved these changes
Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack containing 2 PRs:
mainR1 of the Runner containerization record. A Dockerfile on distroless, built by
rootless BuildKit — the ruled mechanism for a prebuilt application. Nix still
builds every carried artifact; it is the builder, never the runtime.
The image carries the compass-runner binary, the KVM userland it exec's
(cloud-hypervisor, virtiofsd, passt) and the three guest assets the microVM
backend direct-boots, so a Runner pod needs no host toolchain.
WHY DISTROLESS STATIC AND NOT :base. The carried binaries are dynamically
linked but nix-closed: each names an absolute /nix/store interpreter and
resolves every NEEDED library through its own RPATH, so the closure supplies
its own ld.so and glibc and the base is never consulted for a library. That
makes the base's libc dead weight. Measured, not assumed: with the closure
staged all four binaries execute on distroless-static; with only
cloud-hypervisor's own store path copied and its glibc omitted, the same exec
fails "missing dynamic library" — the control proving the loader comes from
the closure, not the base.
Two consequences that are easy to get wrong, both encoded in the lane:
missing the shared glibc yields an image whose binaries cannot exec at all.
the context root dangles on the build host, and BuildKit checksums context
entries before any COPY runs, so it fails the build outright rather than
resolving later inside the image.
The realise-and-stage logic is TypeScript (the no-bash-gate task), split
pure-core/thin-runner like the microvm-boot-test lane: build-core.ts holds the
out-path parsing, the artifact->build-arg mapping and the closure-root
selection, covered by 13 unit tests. The image build itself is runInCI:false —
it needs a buildkitd, which no compass CI step carries until R2 — so the gate
coverage is that suite, which is where a mapping drift is actually caught.
Verified end-to-end on a rootless buildkitd: the lane builds the real image;
its entrypoint runs as uid 65532; all four binaries execute in-image; passt
resolves by name off PATH; all three guest assets match their host sizes byte
for byte.