feat(runner-image): publish the runner image by digest (RIG-3721) - #1181
Merged
Conversation
…lity (RIG-3336) Decides whether the Compass Runner is containerized for Kubernetes, exactly what the pod is granted, and the generic object contract any conformant cluster can run. Containerize as the DaemonSet delivery + lifecycle unit, rootless with a scoped /dev/kvm device, never `privileged: true`. The ruling is explicitly contingent on a real-hardware spike (R7): a rootless HOST user has affordances a pod does not (setuid newuidmap, unrestricted unshare, no seccomp filter), so "the Runner already runs rootless" does not establish that a locked-down pod can boot the composition. If it cannot, the container-vs-host tradeoff reopens rather than a capability being granted. Three findings worth the reader's attention: - Guest RAM is POD RAM. cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup. With low requests the pod is Burstable with a large overage, which the kubelet ranks first for node-pressure eviction -- and evicting it kills every session on the node. - Container restart implies full session teardown, forced rather than elected: without shareProcessNamespace the Runner is pid 1, so its death tears down the pid namespace and reaps every VMM descendant. - The hostPath char-device route is non-functional, not merely worse posture ([INFERENCE] on cgroup-v2 device-controller behaviour, flagged as such and given a negative control in the spike). Scope is the core capability only. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred per the public/private boundary; the record cites the boundary convention rather than restating it. spike-findings.md records the verification's questions and negative controls BEFORE anyone runs them, so results cannot pick the questions. Each item states the control that must fail; a control that unexpectedly passes shrinks the contract. Mints DL-357. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…RIG-3336) Six findings from the review loop, all real. - The hostPath-over-emptyDir justification was wrong. emptyDir is pod-scoped and DOES survive a container restart, so the axis I contrasted did not distinguish them. Rejustified on outliving the POD: pod recreation and node reboot are when a previous pod's stale pidfile must still be reachable for orphan-reaping. - A DAC permission denial is EACCES, not EPERM (verified: errno 13). This is load-bearing rather than cosmetic, because the errno is what separates the two layers the record is at pains to distinguish -- cgroup device-controller denial is EPERM, filesystem denial is EACCES. Spike S3 now records the errno verbatim and states that seeing EPERM would mean the cgroup layer denied, confounding S3 with S2. - Guest RAM is charged to the pod's cgroup because the VMM is an ordinary child in that cgroup, NOT because of `--memory shared=on`. shared=on's consequence here is an accounting one: guest memory is shared mappings, so summing RSS across cloud-hypervisor, virtiofsd and the Runner double-counts it. Size against PSS or the configured guest total. - Fully-absent requests make the pod BestEffort, not Burstable. The eviction conclusion is unchanged and slightly stronger. - A probe-driven restart is the same node-wide session teardown as a rollout, but nothing rate-limits it the way maxUnavailable bounds a rollout. The contract now requires a conservative or absent liveness probe, and a terminationGracePeriodSeconds sized to the reap budget. - A node drain is equally session-affecting and was covered by nothing, since it is not a delivery mechanism. Named in the constraints, with the note that a fleet PDB bounds concurrent drains but cannot protect a pod on its own draining node. Gates on this tree: root:markdownlint rc=0, orion-ref-gate rc=0, ledger gate rc=0 (OK -- 319 rows, 133 headers). Co-authored-by: Matt Wilkinson <matt@rigel.build>
…h main (RIG-3336) CI caught a duplicate ledger id: main minted its own DL-357 (RIG-3561's pr_number wire-field row, #1037) while this branch was in review, so two rows claimed 357 and design-ledger-gate failed. My local gate had passed because the clone's main was 9 commits stale -- the id was genuinely free when I allocated it and taken by the time I pushed. A ledger id is only free at the moment the gate runs against current main, so it needs re-checking after any merge, not just at allocation. Renumbered to DL-358 (free on current main) and synced the record's three references: the Ledger-impact header, R6, and the tasks table. Merges current main, which also swaps the markdown gate from markdownlint-cli2 to rumdl (RIG-3663, #1094) -- verified via root:ci on the merged tree rather than the retired task. Gates on the merged tree: design-ledger-gate rc=0 (OK -- 320 rows, 133 headers), orion-ref-gate rc=0, root:ci rc=0. Co-authored-by: Matt Wilkinson <matt@rigel.build>
…gency (RIG-3336) The record read as though containerizing the Runner might not work. It does not not work. The composition already boots on Linux with /dev/kvm -- the frozen microVM CI/dev enablement record runs KVM-backed boot tests as a required leg on GitHub Actions ubuntu-latest. What a pod adds over that is confinement: a cgroup device controller, a seccomp filter, a memory cgroup. So R8 asks which grants the confinement makes necessary, not whether microVMs run in containers. Every plausible outcome costs at most a wider pod spec this record already specifies -- a seccomp profile, a device plugin, a supplemental gid. The valuable outcome is a smaller contract, since a grant that proves inert drops out. The one result that would reopen the container-vs-host ruling is a requirement for a Linux capability or privileged: true, and no identified mechanism in the composition needs one. That is now stated as the low-probability tail rather than the expected case. Changes: - OQ-1 reframed from "is the zero-privilege pod sufficient" (load-bearing feasibility) to "which grants does confinement make necessary" (non-load-bearing spec narrowing). OQ-2 stays load-bearing -- the seccomp profile is the only item with a real cost attached -- but a needed profile is now named as a bounded cost, not a setback. - Approach, Global Constraints, Privilege shape, R8 and the spike header all cite the established KVM baseline instead of treating the boot as unproven. - S1 rescoped from "does the composition boot" to "does the confinement hold". - All 8 "contingent on R8" phrasings removed. Also supersedes a stale measurement: the frozen enablement record reads /dev/kvm as crw-rw-rw-, but this box measures crw-rw---- root:kvm (2026-09-12). The mode decides whether the supplementalGroups grant is required or inert, so R8 must record the node's own mode. Gates: design-ledger-gate rc=0 (320 rows, 133 headers), orion-ref-gate rc=0, root:ci rc=0. Co-authored-by: Matt Wilkinson <matt@rigel.build>
The reframe commit corrected design.md but left two things stale. The DL-358 ledger row still said the ruling was "explicitly CONTINGENT" on the spike and that the container-vs-host tradeoff "reopens" if it disappoints. That is the framing the reframe removed, sitting in the durable one-line summary a reader hits before the record. Rewrite it as a grant-set question, cite the KVM CI leg as the reason feasibility is settled, and state the capability/privileged tail as the only reopening outcome. Also restate the seccomp profile as specified-but-open (OQ-2) rather than flatly required. Prose referred to the privilege spike as R8 in 17 places; the task is R7 and no R8 exists, so every "R8 verifies it" pointed at nothing.
…age (RIG-3336) R1 said "a nix-built container image project ... mirroring the existing agent-image lane". That contradicts the private monorepo's frozen first-party-image-builds spec, which splits the two mechanisms by what the image's RUNTIME is rather than by what built the artifact: an image that IS a Nix environment (a CI step image, a dev/agent shell) earns nix2container, while a prebuilt application on a minimal base is a Dockerfile built by rootless BuildKit. The spec forecloses the exact inference R1 made — a first-party image does not earn the nix path "merely because Nix built it or it ships a compiled binary" — and rejects nix2container for app images on cost. The Runner is a prebuilt application: a static CGO_ENABLED=0 binary that exec's three userland binaries and runs no toolchain at runtime. agent-image sits on the other row by name, because that container's job is to be a toolchain; sharing the *-image/ directory shape is not sharing the mechanism. Distroless suits a managed multi-tenant cluster, but the KVM userland comes from nixpkgs and carries store-path interpreter/rpath references, so R1 must settle how those binaries run on a minimal base. R1's test cycle now asserts each carried binary EXECUTES, which a layer-contents check alone would miss. R2 gains the build mechanism and the digest-pinning contract: GHCR has no server-side tag immutability, so R3 pins repo@sha256, never a tag.
…n a container image (RIG-3720) R1 of the Runner containerization record. A Dockerfile on distroless, built by rootless BuildKit — the ruled mechanism for a prebuilt application. Nix still builds every carried artifact; it is the builder, never the runtime. The image carries the compass-runner binary, the KVM userland it exec's (cloud-hypervisor, virtiofsd, passt) and the three guest assets the microVM backend direct-boots, so a Runner pod needs no host toolchain. WHY DISTROLESS STATIC AND NOT :base. The carried binaries are dynamically linked but nix-closed: each names an absolute /nix/store interpreter and resolves every NEEDED library through its own RPATH, so the closure supplies its own ld.so and glibc and the base is never consulted for a library. That makes the base's libc dead weight. Measured, not assumed: with the closure staged all four binaries execute on distroless-static; with only cloud-hypervisor's own store path copied and its glibc omitted, the same exec fails "missing dynamic library" — the control proving the loader comes from the closure, not the base. Two consequences that are easy to get wrong, both encoded in the lane: - The staged context is the TRANSITIVE closure, not the five roots. A context missing the shared glibc yields an image whose binaries cannot exec at all. - The entrypoint is a RELATIVE symlink inside store/. An absolute symlink at the context root dangles on the build host, and BuildKit checksums context entries before any COPY runs, so it fails the build outright rather than resolving later inside the image. The realise-and-stage logic is TypeScript (the no-bash-gate task), split pure-core/thin-runner like the microvm-boot-test lane: build-core.ts holds the out-path parsing, the artifact->build-arg mapping and the closure-root selection, covered by 13 unit tests. The image build itself is runInCI:false — it needs a buildkitd, which no compass CI step carries until R2 — so the gate coverage is that suite, which is where a mapping drift is actually caught. Verified end-to-end on a rootless buildkitd: the lane builds the real image; its entrypoint runs as uid 65532; all four binaries execute in-image; passt resolves by name off PATH; all three guest assets match their host sizes byte for byte.
…root uid (RIG-3720) Review findings on the R1 lane. The load-bearing one: the image was NOT reproducible, contradicting R1's acceptance and a Dockerfile comment that claimed the COPY layer was "bit-identical to the staged tree". Two builds of identical inputs produced different manifest digests. Two independent causes, both measured rather than reasoned about: - cpSync defaults preserveTimestamps:false, so every staged file got a wall-clock mtime instead of the 1s nix normalises its store to. - BuildKit carries the context's mtimes into the layer tar regardless. Fixing only the first was NOT sufficient: the staged tree hashed bit-identical across runs while the `COPY store` layer digest still moved. The export needs rewrite-timestamp=true plus a fixed SOURCE_DATE_EPOCH. With both, two consecutive builds yield the same digest. The publish lane asserts exactly this, so the property is now real instead of asserted. Also from review: - Drop the RUNNER_BIN build-arg. It was declared and supplied but never consumed — the entrypoint became a staged symlink and the arg was left behind. The new drift test is what catches this class: it parses the Dockerfile's own ARG lines and asserts set-equality with what build-core supplies. Confirmed it fails when an ARG is renamed and passes when restored. - Add an explicit USER 65532:65532 rather than inheriting nonroot from an opaque base digest, so a future digest bump onto the root variant cannot silently produce a root-running container. - Assert the build host is x64. The platform is a manifest LABEL BuildKit applies without checking the staged files; today only flake.nix's systems list keeps them honest, which this lane never reads. - Narrow the build task's inputs from `**/*` to named tracked files: moon's native walker hashes contents directly and does not skip gitignored paths, so the glob would hash the multi-GB closure this task itself writes. - Cut the mechanism rationale from five restatements to one plus pointers, drop forward task references from source, and delete a tautological test that could not fail.
The CI `setup` job failed at "Generate the concern matrix", blocking every downstream leg. tools/ci-matrix requires each moon project to carry exactly one `ci-group.*` tag and fails loud otherwise — the fail-loud posture that lets moon own discovery instead of a stale YAML enumeration — and the new compass-runner-image project shipped with `tags: []`. ci-group.nix, with the sibling image projects: the real build closure is the nix one those legs already provision. The project contributes no CI task today (`build` is runInCI:false), so the tag places it in the graph rather than adding a leg. `moon ci` does not run the matrix generator, so a green local sweep said nothing about this. Run `bun run tools/ci-matrix/index.ts` when adding a project.
Publish the runner image from the release workflow with rootless BuildKit and report the resolved immutable digest. The deployed contract is `repo@sha256:...`, not a tag: GHCR has no server-side tag immutability, so anyone holding `packages:write` can re-point a tag at other bytes. The `:git-<sha12>` tag exists only for build addressability. buildctl exposes no `push` verb -- a push is an exporter on `build` -- so the publish step re-runs the build with `--output push` rather than uploading a previously written layout. This keeps one staging implementation: a second copy could drift in the details that make the digest reproducible. The lane mirrors the established publish discipline: a 0600 auth file in a private dir (never on argv), a numbered exit-code taxonomy, a secret scan of the built image config's env *before* the push, and a post-push digest assertion. A `--expect-digest` flag turns reproducibility into a gate. Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
Compass engineering docs preview: https://compass-managed-rig-3721-pub.compass-eng-docs.pages.dev Deployed from |
Base automatically changed from
compass-managed/rig-3720-runner-image
to
main
September 12, 2026 21:54
|
😎 Merged successfully - details. |
…media types (RIG-3721) Three defects found by running the lane end to end against a local registry. The push run cleared the OCI layout the digest gate reads, so publishing destroyed its own evidence and then crashed on the missing index.json with a raw ENOENT instead of a named failure. Only a layout-producing run clears it now, and the guard checks for index.json rather than the directory the build creates before the exporter writes into it. The push output spec now states `oci-mediatypes=true` instead of inheriting it. Media-type strings live inside the manifest, so a Docker-media-type push of identical blobs would hash differently from the layout and red every run after the bytes were already published. `rewrite-timestamp` forces OCI types today -- measured: a push and a layout of the same context both yield sha256:e6e98c76... -- so this removes a dependency on that coupling. Verified end to end: a fresh build reproduced the same digest, publish exited 0 and printed the digest ref, the layout survived, and a planted GITHUB_TOKEN in the image config blocked the push with exit 3 before any upload. Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
marked this pull request as ready for review
September 12, 2026 22:14
mattwilkinsonn
approved these changes
Sep 12, 2026
…e config secret scan (RIG-3721) Review findings on the publish lane. The publish job had no changed-path gate, so every push to main ran the ~90-minute guest-rootfs build and republished a byte-identical image. It now self-gates over a new `RUNNER_IMAGE_CLOSURE_PATHS` set, taken from the image project's own declared build inputs, with the same err-toward-publishing fallbacks the agent image's gate uses. Driven over eight changed-file sets: a docs-only push and an unrelated Go command both resolve false; each closure path resolves true. The secret scan now covers config labels as well as env, and adds the keyword forms a name-level guard needs (bare `KEY`, `PAT`, `PASSPHRASE`, `BEARER`) — `SSH_KEY` and `DEPLOY_KEY` previously slipped. The word boundary keeps `PATH` and `KEYRING` from matching, which a bare `PAT`/`KEY` keyword would otherwise fire on and red every publish. Its comments now state what it does not read: layer contents, recorded build-arg values, and value shapes. Readiness is now `buildctl debug workers` rather than the socket file existing, since a present socket does not prove the worker accepts connections, and buildkitd's output is kept so a startup failure is diagnosable instead of a bare timeout. Also: a malformed local layout reports its own exit code rather than borrowing the usage one; the build tag truncates the sha deterministically, as `--short=12` returns the shortest unique length of at least 12; the registry auth blob is masked, because Actions matches the raw token and not a transformed form; and `--expect-digest` is gone, having had no caller.
mattwilkinsonn
approved these changes
Sep 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is part of a stack containing 2 PRs:
mainPublish the runner image from the release workflow with rootless BuildKit and
report the resolved immutable digest.
The deployed contract is
repo@sha256:..., not a tag: GHCR has no server-sidetag immutability, so anyone holding
packages:writecan re-point a tag atother bytes. The
:git-<sha12>tag exists only for build addressability.buildctl exposes no
pushverb -- a push is an exporter onbuild-- so thepublish step re-runs the build with
--output pushrather than uploading apreviously written layout. This keeps one staging implementation: a second copy
could drift in the details that make the digest reproducible.
The lane mirrors the established publish discipline: a 0600 auth file in a
private dir (never on argv), a numbered exit-code taxonomy, a secret scan of
the built image config's env before the push, and a post-push digest
assertion. A
--expect-digestflag turns reproducibility into a gate.Co-authored-by: Matt Wilkinson matt@rigel.build