From ea3fe8e67a4fc72f748d65bf4d985e1dd37ea705 Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 11 Sep 2026 21:27:44 -0400 Subject: [PATCH 1/8] docs(design): record Compass Runner containerization as a core capability (RIG-3336) Decides whether the Compass Runner is containerized for Kubernetes, exactly what the pod is granted, and the generic object contract any conformant cluster can run. Containerize as the DaemonSet delivery + lifecycle unit, rootless with a scoped /dev/kvm device, never `privileged: true`. The ruling is explicitly contingent on a real-hardware spike (R7): a rootless HOST user has affordances a pod does not (setuid newuidmap, unrestricted unshare, no seccomp filter), so "the Runner already runs rootless" does not establish that a locked-down pod can boot the composition. If it cannot, the container-vs-host tradeoff reopens rather than a capability being granted. Three findings worth the reader's attention: - Guest RAM is POD RAM. cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup. With low requests the pod is Burstable with a large overage, which the kubelet ranks first for node-pressure eviction -- and evicting it kills every session on the node. - Container restart implies full session teardown, forced rather than elected: without shareProcessNamespace the Runner is pid 1, so its death tears down the pid namespace and reaps every VMM descendant. - The hostPath char-device route is non-functional, not merely worse posture ([INFERENCE] on cgroup-v2 device-controller behaviour, flagged as such and given a negative control in the spike). Scope is the core capability only. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred per the public/private boundary; the record cites the boundary convention rather than restating it. spike-findings.md records the verification's questions and negative controls BEFORE anyone runs them, so results cannot pick the questions. Each item states the control that must fail; a control that unexpectedly passes shrinks the contract. Mints DL-357. Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 1 + .../compass-runner-containerization/design.md | 419 ++++++++++++++++++ .../spike-findings.md | 104 +++++ 3 files changed, 524 insertions(+) create mode 100644 docs/designs/platform/compass-runner-containerization/design.md create mode 100644 docs/designs/platform/compass-runner-containerization/spike-findings.md diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index ee3403cb4..13324ad57 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -470,3 +470,4 @@ check enforces the mechanical half. Full rationale: | DL-347 | Multi-actor comms coverage is tiered: cross-agent conversation, fan-out/isolation, and offline redelivery are asserted at the podman e2e tier with agent authorship carried by the real Runner session→account binding (no per-agent credential — `relay_comms.go:7-15`), and a non-admin observer client proves the credentialed transport path; the D9 visibility/leak matrix stays at the pgtest tier with e2e proving only one transport positive+negative. every agent-reachable relay arm is covered by a unit test at the runnerhub tier (podman is spent on the tool→relay→server JOIN, never on arm dispatch), and each native agent tool has at least one real-server execution. No new build tag or CI tier — new legs ride the existing e2e skip-guard | Active (Matt, 2026-09-08) | [multi-actor comms coverage](server/compass-comms-multi-actor-test-coverage/design.md#resolved-decisions) | | DL-348 | The UI emits the PostHog session id on outbound Connect requests via a `sessionIdInterceptor` in `@compass/client` (sibling to `traceResponseInterceptor`), completing the J1 correlation-key seam whose inbound half stamps semconv `session.id` on backend spans. `X-POSTHOG-SESSION-ID` ONLY — `X-POSTHOG-DISTINCT-ID` is permanently excluded because it identifies a person and backend spans land in Grafana/Tempo, the plane J1 keeps identity out of; identity resolution stays PostHog-side where `identify()` holds it. Supersedes the 2026-09-05 HOLD (RIG-3233), whose two reasons — no inbound consumer, and a lazy-getter-over-nothing boot workaround — are discharged by the server merge and by the boot reorder respectively. posthog-js's own `TracingHeaders` extension is rejected: it monkey-patches global fetch/XHR and also sends the distinct-id header | Active (Matt, 2026-09-07) | [outbound session header](ui/compass-outbound-session-header/design.md#the-outbound-interceptor) | | DL-349 | Boot constructs analytics BEFORE the live clients (`createAnalytics` → `createLiveClients` → `bootCaller` → `identify`), so the transport's session-id getter closes over a real `Analytics` rather than a mutable ref slot or a forward `let`. Rejected alternatives: mirroring the `clients.traceId` sink (a workaround for a construction-order problem the reorder deletes) and a forward `let analytics` (a TDZ-shaped `undefined` window every reader must guard). Accepted trade, stated not silent: on the WhoAmI-failure early return an analytics-enabled deployment now emits an anonymous PostHog session (init-time remote-config egress) where it previously emitted nothing — the reorder is unavoidable while the transport needs the getter at construction, and the egress itself is declined-not-absent (`advanced_disable_flags` would suppress it, at the cost of remote config), with nothing captured either way. The sender-side guard is printable ASCII + `.length ≤ 200`, STRICTER than the server's `≤200 bytes` + valid-UTF-8 pair and NARROWER than `Headers.set` itself, because `Headers.set` is a WebIDL ByteString: a well-formed id above U+00FF THROWS inside the interceptor and would fail the RPC, and U+0080–U+00FF is accepted by `set` but a browser then emits it as a SINGLE RAW HIGH BYTE on the wire, which fails Go's `utf8.ValidString`, so `sessionIDFromHeader` returns `""` and the id is DROPPED — silent loss, the same failure class as every other rejected value, NOT a wrong correlation key the server accepts (measured: raw-TCP wire bytes `736573732de9` from Node/undici, which serializes like a browser; and real Chromium → real Go `net/http` running a verbatim `sessionIDFromHeader` copy — `len` 6, `utf8.ValidString` false, result `""`. A Bun-client-to-`Bun.serve` round-trip measures Bun's own encode/decode pair, not the wire, and is NOT valid evidence here) | Active (Matt, 2026-09-07) | [outbound session header](ui/compass-outbound-session-header/design.md#the-boot-reorder-chosen--approved-by-matt) | +| DL-357 | The Compass Runner is containerized as the Kubernetes delivery and lifecycle unit — a DaemonSet pod running rootless with a scoped `/dev/kvm` device, NEVER `privileged: true`. The container wins over a host systemd service on four operator properties (atomic digest-named userland, one declarative fleet object, userland isolated from the node, no node mutation on upgrade); the privilege question is the price, and the ruling is explicitly CONTINGENT on a real-hardware spike (R7) proving a zero-privilege pod boots cloud-hypervisor + virtiofsd + passt — if it cannot, the container-vs-host tradeoff reopens rather than a capability being granted. `/dev/kvm` arrives by device plugin, not a hostPath char device: [INFERENCE] the cgroup device controller denies `open()` without CRI `Devices` injection, so hostPath functions only under privileged mode — R7 verifies it with a negative control. Guest RAM is POD RAM (cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup), therefore DaemonSet requests must be sized to the node's aggregate session capacity or the kubelet ranks the Runner first for node-pressure eviction and kills every session on the node. Container restart implies full session teardown — FORCED by the pid namespace (the Runner is pid 1, so its death reaps every VMM descendant), not elected. The image carries the whole KVM userland and guest assets, holding hostPath to exactly two mounts. A custom `Localhost` seccomp profile is required because the composition needs `unshare`/`mount`/`pivot_root` that `RuntimeDefault` restricts — the host-rootless affordance a pod removes; if R7 shows `RuntimeDefault` suffices, the profile and its on-node staging both drop. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred | Active (Matt, 2026-09-12) | [runner containerization](platform/compass-runner-containerization/design.md#resolved-decisions) | diff --git a/docs/designs/platform/compass-runner-containerization/design.md b/docs/designs/platform/compass-runner-containerization/design.md new file mode 100644 index 000000000..d860d96e6 --- /dev/null +++ b/docs/designs/platform/compass-runner-containerization/design.md @@ -0,0 +1,419 @@ +# Containerizing the Compass Runner + +Status: Draft — freezes on merge. The §Privilege shape contract is contingent on R8's real-hardware verification. + +Ledger-impact: mints DL-357 + +## Problem / Intent + +Compass ships a Runner that launches each agent session as a microVM +(cloud-hypervisor + virtiofsd + passt) rootless, as an ordinary host process. +Operators who run Compass on Kubernetes have no supported way to deploy it: +there is no Runner container image, no lane that builds one, and no recorded +answer to what such a container would have to be granted in order to boot a +microVM at all. + +That last question is the hard one, and it is why this is a design record +rather than a packaging task. A rootless *host* user has affordances a +container does not — setuid `newuidmap`, unrestricted `unshare(2)`, no seccomp +filter — so "the Runner already runs rootless" does not establish that a +locked-down pod can run it. This record decides whether the Runner is +containerized, states exactly what the container is granted and what it is +denied, says where the KVM userland and guest assets live, and defines the +Kubernetes object contract for a fleet whose real workload — the session +microVMs — is invisible to Kubernetes. + +Scope is the **core capability**: the image, the privilege shape, and the +generic Kubernetes object contract any conformant cluster can run. Choosing a +cloud, a node provisioner, or a GitOps delivery path for a particular +deployment is an operator concern and out of scope here. + +## Approach + +**Containerize the Runner as the Kubernetes delivery and lifecycle unit, +running rootless inside an unprivileged pod with a scoped `/dev/kvm` device — +never `privileged: true`.** + +The apparent conflict with the runtime record's "launched rootless as an +ordinary host process per session" +(`docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-runner.md`, +§D6) dissolves once the two layers are separated. *Rootless* is a property of +the uid the Runner and its VMM children run as. *Ordinary host process* +distinguishes the VMM from a CRI-owned pod-sandbox runtime. A container is a +namespaced process tree: cloud-hypervisor, virtiofsd, and passt run as +ordinary children of the containerized Runner at a non-zero uid, exactly as +they would under systemd. + +What the runtime record forbids is *privilege* — its D7 discussion pushes +anything needing a capability the rootless Runner lacks out to operator +provisioning plus preflight verification. That sentence rules out capabilities +as a *requirement*; it does not establish that a `drop: ["ALL"]` + +`RuntimeDefault` pod grants *enough*, because a pod denies affordances the +host-rootless model silently assumed. So the containerize ruling is explicitly +**contingent on the privilege shape verifying on real hardware (R8)**: if a +zero-privilege pod cannot run the frozen virtiofsd/passt composition, the +container-vs-host tradeoff reopens. + +### Container vs host process: why the container wins + +A host systemd service and a container both satisfy the runtime record. The +container wins on four properties an operator needs and systemd does not +provide: + +1. **Atomic, rollback-able version delivery.** The image digest names the + entire userland — Runner binary, cloud-hypervisor, virtiofsd, passt, guest + kernel and rootfs — as one unit. A host service versions each of those + independently, and a partial upgrade is a supported state. +2. **A declarative fleet object.** One DaemonSet describes the Runner on every + eligible node, with a rollout strategy and a health surface, rather than N + nodes' worth of drifted unit files. +3. **Isolation of the userland from the host.** The VMM binaries and guest + assets live in the image, not in the node's filesystem, so a node never + accumulates Compass-specific packages. +4. **No node mutation on upgrade.** Changing the Runner version is a pod + template change, not a node reprovision. + +The cost is the privilege question, which is the subject of §Privilege shape +and R8. + +### Net backend: passt, implemented, not an open pick + +Networking needs no capability. A rootless VMM cannot create a host tap, so +networking is a userspace concern the backend provides — and the implemented +D6 backend is passt, an unprivileged userspace forwarder. Guest networking is +in-guest; the Runner dials **out** to the Server over gRPC with its per-Runner +token (`go/cmd/compass-runner/main.go`), so the pod needs no host port and no +inbound service. + +### Privilege shape (the pod spec — contract contingent on verification) + +Every grant is justified; anything not listed is denied. This is a +contract-contingent-on-verification: the ruled privilege shape and the ruled +`/dev/kvm` grant must still verify on real hardware (R8) before R3 encodes +them. The rulings pick what R8 verifies first; they do not remove the +verification. + +- **`/dev/kvm` via a device plugin — NOT `privileged: true`, NOT a raw + hostPath char-device mount.** `/dev/kvm` is a world-irrelevant char device + the KVM API gates by fd; the runtime record's D3 requires only that it + "exists and is openable by the Runner uid". A generic device plugin + advertises a `devices.kubelet.io/kvm`-style resource and the container + runtime injects the device node with the correct cgroup device-controller + allowance; the pod requests it via `resources.limits`. + + The hostPath char-device route is rejected as **non-functional**, not merely + as a worse posture. **[INFERENCE]** A hostPath mount exposes the node's + device into the mount namespace, but the cgroup device controller (eBPF-backed + on cgroup v2) still denies `open()` unless the runtime injects the device via + CRI's `Devices` field — which is exactly what a device plugin's + `ContainerAllocateResponse` does — so a hostPath char device functions only + under privileged mode, which is banned here. Marked inference because this is + upstream Kubernetes/CRI/cgroup-v2 behaviour, grounded in no artifact in this + repo, and because it is what demotes hostPath from *worse-posture* to *not a + candidate*. If it is wrong, the option set reopens. **R8 verifies it**: + attempt the hostPath route on a real node and confirm `open()` actually fails + without device-plugin injection. + + The device-plugin implementation is an operator pick, not frozen here; any + community plugin image an operator pins is thinly maintained and should be + pinned by digest. The contract this record carries, contingent on R8, is + *scoped device node, zero capabilities, no privileged mode*. + +- **`securityContext`:** `runAsNonRoot: true`, `runAsUser`/`runAsGroup` fixed + to a dedicated runner uid, `allowPrivilegeEscalation: false`, + `capabilities.drop: ["ALL"]`, and seccomp `type: Localhost` with a custom + profile permitting `unshare`/`mount`/`pivot_root`. The custom profile is a + new deliverable: authored with R3, staged on-node under the kubelet's seccomp + root by operator node provisioning, and asserted by R3's test. + + `RuntimeDefault` is not sufficient, which is the whole reason the profile + exists — see §Alternatives considered. + + No capability is added. Project-quota *assignment* + (`FS_IOC_FSSETXATTR` + `quotactl`) needs `CAP_SYS_ADMIN` the rootless Runner + lacks, so it stays an operator-provisioning concern verified by Runner + preflight (the runtime record's D7). Networking needs nothing (passt). + + One filesystem grant IS required. **[INFERENCE]** `/dev/kvm` is typically + `root:kvm 0660`, and device injection grants a *cgroup allowance*, not + filesystem permission — so the non-root runner uid needs the `kvm` gid via + `securityContext.supplementalGroups` (or a node-provisioning chmod), or the + first real `open()` fails `EPERM`. Marked inference because the device node's + mode and ownership are properties of the node image's udev rules, not of + anything in this repo, and because it is the sole justification for the + `supplementalGroups` grant. **R8 verifies it directly**, and the negative + control is the one that matters: confirm a non-root uid *without* the kvm gid + actually fails to open the device. If it opens without the gid, the grant is + unnecessary and drops out of the contract. + +- **`hostNetwork: false`, `hostPID: false`.** Guest networking is in-guest and + the Runner dials out, so no host ports and no inbound service. hostPID is + unnecessary because the VMM/virtiofsd children are the Runner's own + descendants inside the container pid namespace. + + The restart semantics follow deterministically rather than being chosen: + without `shareProcessNamespace` the container's pid 1 **is** the Runner, so + Runner death tears down the pid namespace and the kernel kills every + descendant (VMM, virtiofsd, passt); a kubelet-driven restart kills the + container cgroup regardless. "Container restart implies full session + teardown" is therefore forced, not elected — and it guarantees no stranded + VMM processes. The hostPath runtime dir then correctly serves the reap path: + after a restart the backend finds only stale pidfiles, which it already + handles by pidfile plus process-liveness check. + +- **hostPath mounts, exactly two:** + 1. the session-volume tree (the project-quota filesystem operator + provisioning supplies) — read-write, `type: Directory`; + 2. the Runner runtime dir (`--runtime-dir`, default `/run/compass`) plus the + microVM runroot (`--microvm-runroot`) — a single host tree, read-write. + + Host-visible rather than `emptyDir` so pidfiles survive a container restart + for orphan-reaping. Nothing else: the VMM/virtiofsd binaries and the guest + kernel/rootfs/initrd ship **in the image**, so no hostPath reaches them. + +- **Explicitly absent:** `privileged`, every capability, `hostNetwork`, + `hostPID`, `hostIPC`, any `/dev` directory mount, any container-runtime + socket. + +### Pod resources, QoS, and eviction: guest RAM is pod RAM + +cloud-hypervisor's virtio-fs path requires `--memory shared=on` (the argv +construction in `go/internal/runtime/microvm/launch.go`; `BootConfig.MemoryMB` +in `go/internal/runtime/microvm/config.go` is documented as always launched +with `shared=on`). Guest memory is therefore memfd/shared mappings charged to +the pod's memory cgroup: **every session microVM's RAM counts against the +Runner pod.** + +With absent or low memory requests the pod is Burstable with a large +usage-over-requests overage — precisely the pod the kubelet's node-pressure +eviction ranks first — and evicting it kills every session on the node. A +priorityClass helps preemption and eviction ranking only if requests are +honest. The DaemonSet's requests and limits MUST account for the aggregate +guest RAM of the node's session capacity, not just the Runner process itself. +R3's test cycle asserts requests and limits are present and sized to the +session-capacity model. + +### Where the KVM userland and guest assets live: in the image + +The image carries the Runner binary, cloud-hypervisor, virtiofsd, passt, and +the guest kernel/rootfs/initrd. This is deliberately image-heavy: it is what +makes the digest name the whole userland (§Container vs host process, +property 1) and what keeps hostPath down to two mounts. The guest assets dominate the +image size; that cost is accepted in exchange for atomic version delivery. + +### Kubernetes object contract + +Stated as generic Kubernetes so any conformant cluster can run it. An operator +supplies the *values* (node labels, taints, the device-plugin resource name, +capacity sizing); this record fixes the *shape*. + +- **DaemonSet**, so the Runner lands on every eligible node. + - `nodeSelector` on an operator-chosen label key, and a matching + `toleration` for the taint that keeps non-Runner workloads off those nodes. + Keys and values are the operator's; the contract is that both exist. + - `updateStrategy` with a bounded `maxUnavailable`: a Runner pod replacement + terminates that node's sessions (§Privilege shape, restart semantics), so a + rollout is session-affecting and must be rate-limited. + - `securityContext`, device resource request, and the two hostPath mounts per + §Privilege shape. + - `spec.nodeName` via `fieldRef` into the environment, so the Runner can + identify its node. + - A scrape annotation for the metrics endpoint. +- **RBAC**: a dedicated ServiceAccount with the narrowest role the Runner + actually needs. The Runner dials out to the Server and does not drive the + Kubernetes API for session work, so this is minimal by construction. +- **priorityClass**: high enough that a Runner pod is not preempted by ordinary + workload, consistent with the eviction reasoning above. + +### Relationship to the elastic-session-runtime record + +This record does not amend the runtime record's D3/D6/D7. It *consumes* them +and adds the container/pod layer beneath: D3's "openable by the Runner uid" +becomes a device-plugin grant plus a gid; D6's passt backend is unchanged; D7's +quota assignment stays outside the Runner, satisfied by operator provisioning +and checked by preflight. + +## Alternatives considered + +### Host-level systemd service — rejected + +Satisfies the runtime record and avoids the privilege question entirely, but +loses all four properties in §Container vs host process: no atomic digest, no +declarative fleet object, node-resident userland, and node mutation on every +upgrade. Rejected for operators who are already running Kubernetes; it remains +the right shape for a bare-metal single-node deployment, which this record does +not address. + +### `privileged: true` DaemonSet — rejected + +Trivially works and is what most VM-on-Kubernetes stacks ship. Rejected: a +privileged pod holds every capability and effectively owns the node, which +defeats the isolation the microVM boundary exists to provide. The whole point +of the microVM is that a session cannot reach the host; a privileged Runner +re-opens that path from the other side. + +### `RuntimeDefault` seccomp with no custom profile — rejected + +Preferred if it worked, since it needs no on-node artifact. Rejected because +the composition needs `unshare`/`mount`/`pivot_root`, which `RuntimeDefault` +restricts — this is precisely the host-rootless affordance a pod removes, and +the reason §Privilege shape carries a custom `Localhost` profile as a new +deliverable. The cost is real: `Localhost` requires the profile be staged on +the node before the pod starts, which couples the DaemonSet to operator node +provisioning. + +### Static pod / runner-in-node-image hybrid — rejected + +Bakes the Runner into the node image and manages it as a static pod. Gets +atomic delivery of a sort, but the unit of rollback becomes the node image, so +a Runner version bump is a node reprovision — the exact cost containerizing is +meant to remove. + +### Kata / pod-sandbox runtime class — rejected upstream + +Already rejected by the runtime record: the Runner runs the VMM as its own +children, not as a CRI-owned pod sandbox. Restated here only because it is the +first thing a Kubernetes reader reaches for. + +## Global Constraints + +- **Zero privilege, or the ruling reopens.** No `privileged`, no added + capability. If R8 shows the composition cannot run under the §Privilege shape + contract, this record's central decision is reopened rather than patched with + a capability grant. +- **The image is the unit of version.** Anything in the KVM userland or guest + asset set ships in the image; nothing is expected on the node except the two + hostPath trees and the seccomp profile. +- **A rollout is session-affecting.** Any change to the pod template + terminates sessions on each replaced node, so every delivery mechanism must + rate-limit it. +- **Operator-supplied values stay operator-supplied.** This record fixes object + shape, not node labels, taint values, capacity numbers, or a GitOps path. +- **Deployment-plane concerns are out of scope.** Which cluster, which node + provisioner, and how the manifests reach a cluster are operator concerns; + they are named and deferred here, never described. + +## Plan + +### R1 — `runner-image/` project + +A nix-built container image project in this repo, mirroring the existing +agent-image lane: Runner binary, cloud-hypervisor, virtiofsd, passt, guest +kernel/rootfs/initrd. Test cycle: the image builds reproducibly and the +resulting layer set contains each expected binary and guest asset. + +### R2 — publish lane + +Extend the release workflow to build and publish the runner image by digest. +Test cycle: a tagged run publishes a manifest whose digest is recorded in the +run output; a second build of the same input yields the same digest. + +### R3 — DaemonSet + RBAC manifests as config-as-data + +Author the object contract from §Kubernetes object contract as plain +manifests, parameterized where §Global Constraints says the operator supplies +values. Test cycle: the rendered objects assert the full §Privilege shape +(no `privileged`, `drop: ["ALL"]`, `runAsNonRoot`, the device resource, the +`Localhost` profile reference, exactly two hostPath mounts); requests and +limits are present and sized per the capacity model; `maxUnavailable` is +bounded. + +### R4 — `/dev/kvm` device-plugin delivery + +Document and encode the device-plugin requirement, including the resource-name +parameterization and the `supplementalGroups` gid — both contingent on R8. +Test cycle: rendered pod spec requests the device resource and carries the gid; +a spec that omits either fails the assertion. + +### R5 — entrypoint fix for `--backend microvm` + +The image entrypoint must pass agent-image semantics correctly under +`--backend microvm`. Test cycle: the container started with `--backend microvm` +resolves its agent image from the documented flag/env precedence, asserted +without launching a VM. + +### R6 — ledger row + +Mint DL-357 recording the containerize ruling and the zero-privilege +constraint. + +### R7 — privilege-shape spike on real hardware (gates the §Privilege shape freeze) + +The verification the contract is contingent on. Findings land in +`spike-findings.md` beside this record. It must answer, each with a negative +control: + +1. Does an unprivileged pod with the §Privilege shape grants boot + cloud-hypervisor + virtiofsd + passt end to end? +2. Does the hostPath char-device route actually fail `open()` without + device-plugin injection? (If it succeeds, the option set reopens.) +3. Does a non-root uid without the kvm gid fail to open `/dev/kvm`? (If it + opens, the `supplementalGroups` grant drops.) +4. Is the custom `Localhost` profile actually required — does + `RuntimeDefault` fail, and on which syscall? +5. Does guest RAM appear in the pod's memory cgroup as §Pod resources claims? +6. Does a container restart leave zero stranded VMM/virtiofsd/passt processes? + +### Task ordering + +R7 gates R3 and R4 (it freezes the contract they encode). R1 gates R2. R5 is +independent. R6 lands with the freeze. + +## Tasks + +| Task | Deliverable | Depends on | +| ------ | ----------- | ---------- | +| R1 | `runner-image/` nix project | — | +| R2 | publish lane by digest | R1 | +| R3 | DaemonSet + RBAC manifests + render tests | R7 | +| R4 | device-plugin resource + gid wiring | R7 | +| R5 | entrypoint agent-image fix under `--backend microvm` | — | +| R6 | DL-357 ledger row | R7 | +| R7 | real-hardware privilege spike + `spike-findings.md` | R1 | + +## Open Questions + +### OQ-1 [load-bearing] — is the zero-privilege pod sufficient? + +The record's central contingency. R7 answers it. If the composition needs a +capability, the container-vs-host ruling reopens rather than the constraint +bending. + +### OQ-2 [load-bearing] — is the custom seccomp profile avoidable? + +If `RuntimeDefault` turns out to suffice, the `Localhost` profile and its +on-node staging requirement both drop, which materially simplifies the +operator's job. R7 item 4 settles it. + +### OQ-3 [non-load-bearing] — device-plugin implementation pick + +The contract names a resource, not a plugin. Which plugin an operator runs is +an implementation choice; the community options are thinly maintained and want +a digest pin. + +### OQ-4 [non-load-bearing] — runner uid and kvm gid values + +Fixed values chosen at implementation and single-sourced between the image, the +manifests, and operator node provisioning. + +### OQ-5 [non-load-bearing] — session-volume host path and filesystem + +The path is operator-supplied; the filesystem must support project quotas for +D7. Default path chosen at implementation. + +## Resolved decisions + +- **Containerize the Runner** as the Kubernetes delivery unit, rather than a + host systemd service — contingent on OQ-1 (§Container vs host process). +- **Never `privileged: true`** — the microVM isolation boundary is the reason + the Runner exists, and a privileged Runner re-opens the host path + (§Alternatives considered). +- **`/dev/kvm` by device plugin**, not hostPath char device, not privileged + mode (§Privilege shape). +- **Guest RAM is pod RAM**, so DaemonSet requests must be sized to node + session capacity, not to the Runner process (§Pod resources). +- **Container restart implies full session teardown** — forced by the pid + namespace, not chosen (§Privilege shape). +- **The image carries the whole KVM userland and guest assets**, keeping + hostPath to exactly two mounts (§Where the KVM userland and guest assets + live). diff --git a/docs/designs/platform/compass-runner-containerization/spike-findings.md b/docs/designs/platform/compass-runner-containerization/spike-findings.md new file mode 100644 index 000000000..23876cb41 --- /dev/null +++ b/docs/designs/platform/compass-runner-containerization/spike-findings.md @@ -0,0 +1,104 @@ +# Privilege-shape spike findings + +Findings for R7 of +[runner containerization](design.md). **Not yet run** — this file is the +recorded shape of the verification so the questions and their negative controls +are fixed before anyone runs them, rather than being chosen after seeing +results. + +The design record's §Privilege shape is a contract *contingent on this spike*. +Two of its grants rest on `[INFERENCE]` claims about upstream +Kubernetes/CRI/cgroup-v2 and node-image behaviour, grounded in no artifact in +this repo. This spike is what converts them to evidence or reopens the option +set. + +## Why each item carries a negative control + +A positive result alone cannot distinguish "the grant is necessary" from "the +grant is inert and something else made it work". Every item below therefore +states the control that must FAIL, and a control that unexpectedly passes is +the finding — it shrinks the contract. + +## Items + +### S1 — does a zero-privilege pod boot the composition end to end? + +Run the §Privilege shape pod spec on a real node and boot a session microVM: +cloud-hypervisor + virtiofsd + passt, through to an agent session that +executes. + +- **Expected:** boots. +- **Negative control:** none needed; this is the load-bearing positive. +- **If it fails:** OQ-1 resolves against containerization and the + container-vs-host tradeoff reopens. Do **not** grant a capability to make it + pass — that is the decision this record forbids. +- **Record:** the failing syscall and the component that needed it. + +### S2 — does the hostPath char-device route actually fail? + +Mount `/dev/kvm` as a hostPath char device, with no device plugin, in an +unprivileged pod, and attempt `open()`. + +- **Expected:** `open()` fails (the cgroup device controller denies it without + CRI `Devices` injection). +- **This IS the negative control** for the design record's claim that hostPath + is non-functional rather than merely worse posture. +- **If it succeeds:** the `[INFERENCE]` is wrong, hostPath returns to the + option set, and §Privilege shape's device-plugin requirement must be + re-argued on posture grounds alone. + +### S3 — is the `kvm` gid grant necessary? + +With the device plugin injecting the device, run as the non-root runner uid +**without** `supplementalGroups`, and attempt `open()`. + +- **Expected:** fails `EPERM` (device injection grants a cgroup allowance, not + filesystem permission). +- **Negative control is the point of the item.** If it *opens* without the gid, + the `supplementalGroups` grant is unnecessary and drops from the contract, + along with the gid-value question. +- **Also record:** the node's actual `/dev/kvm` mode and owner, since the + inference rests on it. + +### S4 — is the custom `Localhost` seccomp profile required? + +Run the same pod under `RuntimeDefault` instead of the custom profile. + +- **Expected:** fails on `unshare`, `mount`, or `pivot_root`. +- **Record which syscall and which component** — that is what justifies each + permit in the profile, and an over-broad profile is a real cost. +- **If `RuntimeDefault` suffices:** OQ-2 resolves, the custom profile drops, + and with it the on-node staging requirement that couples the DaemonSet to + operator node provisioning. This is the most valuable possible outcome of the + spike and must not be assumed away. + +### S5 — does guest RAM appear in the pod's memory cgroup? + +Boot a session with a known guest memory size and read the pod's +`memory.current`. + +- **Expected:** pod memory rises by approximately the guest size, confirming + §Pod resources' "guest RAM is pod RAM". +- **Negative control:** a pod with no session running shows only the Runner's + own footprint. +- **If guest RAM is NOT charged to the pod:** the requests/limits sizing model + is wrong and R3's assertion changes; the eviction risk argument weakens. + +### S6 — does a container restart strand any process? + +Kill the Runner (pid 1) mid-session and enumerate node processes. + +- **Expected:** zero surviving cloud-hypervisor, virtiofsd, or passt + processes — the pid-namespace teardown reaps them. +- **Negative control:** the same enumeration *before* the kill must find them, + or the check proves nothing. +- **Also confirm:** the restarted Runner finds only stale pidfiles in the + hostPath runtime dir and reaps them by its existing pidfile plus + process-liveness path. + +## Reporting + +Each item records: what ran, the observed result, the control's result, and +whether the contract changed. An item whose control was not run is reported as +**not verified**, never as passing — a positive without its control is the +failure mode this file exists to prevent. From 6fc047c0959aa98a8e2579e0c296d8f39d958dec Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 11 Sep 2026 21:56:39 -0400 Subject: [PATCH 2/8] docs(design): fold review findings into the containerization record (RIG-3336) Six findings from the review loop, all real. - The hostPath-over-emptyDir justification was wrong. emptyDir is pod-scoped and DOES survive a container restart, so the axis I contrasted did not distinguish them. Rejustified on outliving the POD: pod recreation and node reboot are when a previous pod's stale pidfile must still be reachable for orphan-reaping. - A DAC permission denial is EACCES, not EPERM (verified: errno 13). This is load-bearing rather than cosmetic, because the errno is what separates the two layers the record is at pains to distinguish -- cgroup device-controller denial is EPERM, filesystem denial is EACCES. Spike S3 now records the errno verbatim and states that seeing EPERM would mean the cgroup layer denied, confounding S3 with S2. - Guest RAM is charged to the pod's cgroup because the VMM is an ordinary child in that cgroup, NOT because of `--memory shared=on`. shared=on's consequence here is an accounting one: guest memory is shared mappings, so summing RSS across cloud-hypervisor, virtiofsd and the Runner double-counts it. Size against PSS or the configured guest total. - Fully-absent requests make the pod BestEffort, not Burstable. The eviction conclusion is unchanged and slightly stronger. - A probe-driven restart is the same node-wide session teardown as a rollout, but nothing rate-limits it the way maxUnavailable bounds a rollout. The contract now requires a conservative or absent liveness probe, and a terminationGracePeriodSeconds sized to the reap budget. - A node drain is equally session-affecting and was covered by nothing, since it is not a delivery mechanism. Named in the constraints, with the note that a fleet PDB bounds concurrent drains but cannot protect a pod on its own draining node. Gates on this tree: root:markdownlint rc=0, orion-ref-gate rc=0, ledger gate rc=0 (OK -- 319 rows, 133 headers). Co-authored-by: Matt Wilkinson --- .../compass-runner-containerization/design.md | 70 ++++++++++++++----- .../spike-findings.md | 8 ++- 2 files changed, 59 insertions(+), 19 deletions(-) diff --git a/docs/designs/platform/compass-runner-containerization/design.md b/docs/designs/platform/compass-runner-containerization/design.md index d860d96e6..3a7d11921 100644 --- a/docs/designs/platform/compass-runner-containerization/design.md +++ b/docs/designs/platform/compass-runner-containerization/design.md @@ -37,7 +37,7 @@ never `privileged: true`.** The apparent conflict with the runtime record's "launched rootless as an ordinary host process per session" (`docs/designs/infra/runtime/compass-elastic-session-runtime/microvm-runner.md`, -§D6) dissolves once the two layers are separated. *Rootless* is a property of +§Approach (a) “VMM shape”) dissolves once the two layers are separated. *Rootless* is a property of the uid the Runner and its VMM children run as. *Ordinary host process* distinguishes the VMM from a CRI-owned pod-sandbox runtime. A container is a namespaced process tree: cloud-hypervisor, virtiofsd, and passt run as @@ -138,7 +138,9 @@ verification. `root:kvm 0660`, and device injection grants a *cgroup allowance*, not filesystem permission — so the non-root runner uid needs the `kvm` gid via `securityContext.supplementalGroups` (or a node-provisioning chmod), or the - first real `open()` fails `EPERM`. Marked inference because the device node's + first real `open()` fails `EACCES`. The errno is the observable that separates + the two layers: a DAC permission denial is `EACCES`, while the cgroup + device-controller denial in the `/dev/kvm` bullet above is `EPERM`. Marked inference because the device node's mode and ownership are properties of the node image's udev rules, not of anything in this repo, and because it is the sole justification for the `supplementalGroups` grant. **R8 verifies it directly**, and the negative @@ -167,8 +169,11 @@ verification. 2. the Runner runtime dir (`--runtime-dir`, default `/run/compass`) plus the microVM runroot (`--microvm-runroot`) — a single host tree, read-write. - Host-visible rather than `emptyDir` so pidfiles survive a container restart - for orphan-reaping. Nothing else: the VMM/virtiofsd binaries and the guest + Host-visible rather than `emptyDir` because the tree must outlive the **pod**, + not just the container. An `emptyDir` is pod-scoped and does survive a + container restart, so that axis does not distinguish them; what it does not + survive is pod recreation or a node reboot, which is exactly when a stale + pidfile from a previous pod must still be visible for orphan-reaping. Nothing else: the VMM/virtiofsd binaries and the guest kernel/rootfs/initrd ship **in the image**, so no hostPath reaches them. - **Explicitly absent:** `privileged`, every capability, `hostNetwork`, @@ -177,16 +182,24 @@ verification. ### Pod resources, QoS, and eviction: guest RAM is pod RAM -cloud-hypervisor's virtio-fs path requires `--memory shared=on` (the argv -construction in `go/internal/runtime/microvm/launch.go`; `BootConfig.MemoryMB` -in `go/internal/runtime/microvm/config.go` is documented as always launched -with `shared=on`). Guest memory is therefore memfd/shared mappings charged to -the pod's memory cgroup: **every session microVM's RAM counts against the -Runner pod.** - -With absent or low memory requests the pod is Burstable with a large -usage-over-requests overage — precisely the pod the kubelet's node-pressure -eviction ranks first — and evicting it kills every session on the node. A +The VMM is an ordinary child of the Runner inside the pod's cgroup, so guest +memory is the VMM process's own memory and is charged to the pod's memory +cgroup: **every session microVM's RAM counts against the Runner pod.** This +holds regardless of the virtio-fs memory mode. + +`--memory shared=on` is nonetheless set unconditionally (the argv construction +in `go/internal/runtime/microvm/launch.go`; `BootConfig.MemoryMB` in +`go/internal/runtime/microvm/config.go` is documented as always launched with +`shared=on`), because virtio-fs requires it. Its consequence here is an +*accounting* one, not a charging one: guest memory is memfd/shared mappings +visible in more than one process, so naively summing RSS across +cloud-hypervisor, virtiofsd, and the Runner double-counts it. Size against PSS +or against the configured guest total, never a sum of RSS. + +With absent or low memory requests the pod is BestEffort (requests and limits +fully absent) or Burstable with a large usage-over-requests overage — precisely +the pod the kubelet's node-pressure eviction ranks first — and evicting it +kills every session on the node. A priorityClass helps preemption and eviction ranking only if requests are honest. The DaemonSet's requests and limits MUST account for the aggregate guest RAM of the node's session capacity, not just the Runner process itself. @@ -219,6 +232,21 @@ capacity sizing); this record fixes the *shape*. - `spec.nodeName` via `fieldRef` into the environment, so the Runner can identify its node. - A scrape annotation for the metrics endpoint. + - **Liveness probe: conservative, or absent.** A probe-driven container + restart is the same full-session teardown as a rollout (§Privilege shape, + restart semantics) — pid 1 dies and every session on the node dies with + it — but nothing rate-limits it the way `maxUnavailable` bounds a rollout, + so a transient health blip costs the whole node's sessions. Prefer a + readiness probe for traffic gating and either omit liveness or give it a + failure threshold well past any transient stall. A startup probe is + unobjectionable. + - **`terminationGracePeriodSeconds` sized to the reap budget.** On + termination the kubelet SIGTERMs pid 1 and SIGKILLs after the grace + period; that window is what lets the Runner shut its VMM children down in + order and notify the Server. Too short and teardown is hard — still no + stranded processes (the pid namespace guarantees that), but no graceful + session drain. Size it to the per-session reap budget times the node's + session capacity. - **RBAC**: a dedicated ServiceAccount with the narrowest role the Runner actually needs. The Runner dials out to the Server and does not drive the Kubernetes API for session work, so this is minimal by construction. @@ -284,9 +312,17 @@ first thing a Kubernetes reader reaches for. - **The image is the unit of version.** Anything in the KVM userland or guest asset set ships in the image; nothing is expected on the node except the two hostPath trees and the seccomp profile. -- **A rollout is session-affecting.** Any change to the pod template - terminates sessions on each replaced node, so every delivery mechanism must - rate-limit it. +- **A rollout is session-affecting — and so is anything else that replaces or + restarts the pod.** Any change to the pod template terminates sessions on + each replaced node, so every delivery mechanism must rate-limit it. The same + blast radius applies to a probe-driven restart and to a **node drain** + (autoscaler, node upgrade, manual drain), neither of which is a delivery + mechanism and so neither is covered by that rate limit. A fleet + PodDisruptionBudget can bound *concurrent* drains — it applies to + eviction-API disruption, though not to the DaemonSet controller's own + rollout — but it cannot keep a pod alive while its own node drains, so + draining a Runner node is inherently session-terminating and must be + scheduled as such. - **Operator-supplied values stay operator-supplied.** This record fixes object shape, not node labels, taint values, capacity numbers, or a GitOps path. - **Deployment-plane concerns are out of scope.** Which cluster, which node diff --git a/docs/designs/platform/compass-runner-containerization/spike-findings.md b/docs/designs/platform/compass-runner-containerization/spike-findings.md index 23876cb41..37ae83985 100644 --- a/docs/designs/platform/compass-runner-containerization/spike-findings.md +++ b/docs/designs/platform/compass-runner-containerization/spike-findings.md @@ -52,8 +52,12 @@ unprivileged pod, and attempt `open()`. With the device plugin injecting the device, run as the non-root runner uid **without** `supplementalGroups`, and attempt `open()`. -- **Expected:** fails `EPERM` (device injection grants a cgroup allowance, not - filesystem permission). +- **Expected:** fails `EACCES` (device injection grants a cgroup allowance, not + filesystem permission, and a DAC denial is `EACCES`). **The errno + discriminates the layer**: seeing `EPERM` instead would mean the cgroup + device controller denied the open, not the filesystem — which confounds this + item with S2 and means the gid question is still unanswered. Record the errno + verbatim, not just pass/fail. - **Negative control is the point of the item.** If it *opens* without the gid, the `supplementalGroups` grant is unnecessary and drops from the contract, along with the gid-value question. From 5eb7a99fbef5ca04eb2b1484a090b4525c26cfee Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 13:46:34 -0400 Subject: [PATCH 3/8] docs(design): reframe R8 as grant-narrowing, not a feasibility contingency (RIG-3336) The record read as though containerizing the Runner might not work. It does not not work. The composition already boots on Linux with /dev/kvm -- the frozen microVM CI/dev enablement record runs KVM-backed boot tests as a required leg on GitHub Actions ubuntu-latest. What a pod adds over that is confinement: a cgroup device controller, a seccomp filter, a memory cgroup. So R8 asks which grants the confinement makes necessary, not whether microVMs run in containers. Every plausible outcome costs at most a wider pod spec this record already specifies -- a seccomp profile, a device plugin, a supplemental gid. The valuable outcome is a smaller contract, since a grant that proves inert drops out. The one result that would reopen the container-vs-host ruling is a requirement for a Linux capability or privileged: true, and no identified mechanism in the composition needs one. That is now stated as the low-probability tail rather than the expected case. Changes: - OQ-1 reframed from "is the zero-privilege pod sufficient" (load-bearing feasibility) to "which grants does confinement make necessary" (non-load-bearing spec narrowing). OQ-2 stays load-bearing -- the seccomp profile is the only item with a real cost attached -- but a needed profile is now named as a bounded cost, not a setback. - Approach, Global Constraints, Privilege shape, R8 and the spike header all cite the established KVM baseline instead of treating the boot as unproven. - S1 rescoped from "does the composition boot" to "does the confinement hold". - All 8 "contingent on R8" phrasings removed. Also supersedes a stale measurement: the frozen enablement record reads /dev/kvm as crw-rw-rw-, but this box measures crw-rw---- root:kvm (2026-09-12). The mode decides whether the supplementalGroups grant is required or inert, so R8 must record the node's own mode. Gates: design-ledger-gate rc=0 (320 rows, 133 headers), orion-ref-gate rc=0, root:ci rc=0. Co-authored-by: Matt Wilkinson --- .../compass-runner-containerization/design.md | 87 +++++++++++++------ .../spike-findings.md | 42 ++++++--- 2 files changed, 87 insertions(+), 42 deletions(-) diff --git a/docs/designs/platform/compass-runner-containerization/design.md b/docs/designs/platform/compass-runner-containerization/design.md index 1c55e6ecb..f0e863f20 100644 --- a/docs/designs/platform/compass-runner-containerization/design.md +++ b/docs/designs/platform/compass-runner-containerization/design.md @@ -1,6 +1,6 @@ # Containerizing the Compass Runner -Status: Draft — freezes on merge. The §Privilege shape contract is contingent on R8's real-hardware verification. +Status: Draft — freezes on merge. §Privilege shape names the grants R8 must confirm are *necessary*; R8 narrows the grant set, it does not decide whether containerization works. Ledger-impact: mints DL-358 @@ -49,10 +49,20 @@ anything needing a capability the rootless Runner lacks out to operator provisioning plus preflight verification. That sentence rules out capabilities as a *requirement*; it does not establish that a `drop: ["ALL"]` + `RuntimeDefault` pod grants *enough*, because a pod denies affordances the -host-rootless model silently assumed. So the containerize ruling is explicitly -**contingent on the privilege shape verifying on real hardware (R8)**: if a -zero-privilege pod cannot run the frozen virtiofsd/passt composition, the -container-vs-host tradeoff reopens. +host-rootless model silently assumed. + +**This is a grant-tuning question, not a feasibility one.** The composition is +already verified to boot on Linux with `/dev/kvm`: the frozen +[microVM CI/dev enablement](../../infra/runtime/compass-elastic-session-runtime/microvm-ci-dev-enablement.md) +record runs KVM-backed boot tests as a required leg on GitHub Actions' +`ubuntu-latest`. What a pod adds over that environment is confinement — a +cgroup device controller, a seccomp filter, and a memory cgroup — so R8 asks +**which grants the confinement makes necessary**, not whether microVMs run in +containers. Each plausible R8 outcome costs a wider pod spec (a seccomp +profile, a device plugin, a supplemental gid), all of which this record already +specifies; none of them reopens the container-vs-host ruling. The one result +that *would* reopen it is a requirement for a Linux **capability** or +`privileged: true`, which no identified mechanism in the composition needs. ### Container vs host process: why the container wins @@ -85,12 +95,11 @@ in-guest; the Runner dials **out** to the Server over gRPC with its per-Runner token (`go/cmd/compass-runner/main.go`), so the pod needs no host port and no inbound service. -### Privilege shape (the pod spec — contract contingent on verification) +### Privilege shape (the pod spec — the grant set R8 narrows) -Every grant is justified; anything not listed is denied. This is a -contract-contingent-on-verification: the ruled privilege shape and the ruled -`/dev/kvm` grant must still verify on real hardware (R8) before R3 encodes -them. The rulings pick what R8 verifies first; they do not remove the +Every grant is justified; anything not listed is denied. R8 confirms which +grants are *necessary* rather than inert, on real hardware, before R3 encodes +them — a grant that proves inert drops out, which is the outcome to hope for. The rulings pick what R8 verifies first; they do not remove the verification. - **`/dev/kvm` via a device plugin — NOT `privileged: true`, NOT a raw @@ -110,14 +119,18 @@ verification. under privileged mode, which is banned here. Marked inference because this is upstream Kubernetes/CRI/cgroup-v2 behaviour, grounded in no artifact in this repo, and because it is what demotes hostPath from *worse-posture* to *not a - candidate*. If it is wrong, the option set reopens. **R8 verifies it**: + candidate*. If it is wrong, hostPath returns to the option set and the + device-plugin requirement must be re-argued on posture grounds — a change to + *which* mechanism delivers the device, not to whether containerization works. + **R8 verifies it**: attempt the hostPath route on a real node and confirm `open()` actually fails without device-plugin injection. The device-plugin implementation is an operator pick, not frozen here; any community plugin image an operator pins is thinly maintained and should be - pinned by digest. The contract this record carries, contingent on R8, is - *scoped device node, zero capabilities, no privileged mode*. + pinned by digest. The contract this record carries is *scoped device node, + zero capabilities, no privileged mode*; R8 confirms the device-plugin + mechanism is the one that delivers it. - **`securityContext`:** `runAsNonRoot: true`, `runAsUser`/`runAsGroup` fixed to a dedicated runner uid, `allowPrivilegeEscalation: false`, @@ -135,7 +148,11 @@ verification. preflight (the runtime record's D7). Networking needs nothing (passt). One filesystem grant IS required. **[INFERENCE]** `/dev/kvm` is typically - `root:kvm 0660`, and device injection grants a *cgroup allowance*, not + `root:kvm 0660` — **measured `crw-rw---- root:kvm` on the compass dev box + (2026-09-12)**, which supersedes the `crw-rw-rw-` reading in the frozen + microVM CI/dev enablement record; the grant is inert under a world-readable + mode and required under `0660`, so R8 must record the node's actual mode — + and device injection grants a *cgroup allowance*, not filesystem permission — so the non-root runner uid needs the `kvm` gid via `securityContext.supplementalGroups` (or a node-provisioning chmod), or the first real `open()` fails `EACCES`. The errno is the observable that separates @@ -305,10 +322,12 @@ first thing a Kubernetes reader reaches for. ## Global Constraints -- **Zero privilege, or the ruling reopens.** No `privileged`, no added - capability. If R8 shows the composition cannot run under the §Privilege shape - contract, this record's central decision is reopened rather than patched with - a capability grant. +- **No capability, no `privileged`.** The grant set may widen along the axes + §Privilege shape already names (seccomp profile, device plugin, supplemental + gid) — that is R8 narrowing or confirming a spec, and is expected. What is + banned is patching a shortfall with a Linux capability or `privileged: true`: + if the composition genuinely needed one, the container-vs-host ruling is + reopened instead. No mechanism in the composition is known to need one. - **The image is the unit of version.** Anything in the KVM userland or guest asset set ships in the image; nothing is expected on the node except the two hostPath trees and the seccomp profile. @@ -357,7 +376,8 @@ bounded. ### R4 — `/dev/kvm` device-plugin delivery Document and encode the device-plugin requirement, including the resource-name -parameterization and the `supplementalGroups` gid — both contingent on R8. +parameterization and the `supplementalGroups` gid — both confirmed necessary +(or dropped as inert) by R8. Test cycle: rendered pod spec requests the device resource and carries the gid; a spec that omits either fails the assertion. @@ -375,7 +395,7 @@ constraint. ### R7 — privilege-shape spike on real hardware (gates the §Privilege shape freeze) -The verification the contract is contingent on. Findings land in +Confirms which grants the confinement makes necessary. Findings land in `spike-findings.md` beside this record. It must answer, each with a negative control: @@ -409,17 +429,27 @@ independent. R6 lands with the freeze. ## Open Questions -### OQ-1 [load-bearing] — is the zero-privilege pod sufficient? +### OQ-1 [non-load-bearing] — which grants does confinement make necessary? -The record's central contingency. R7 answers it. If the composition needs a -capability, the container-vs-host ruling reopens rather than the constraint -bending. +Not a feasibility question. The composition boots on KVM today (see §Approach); +R8 determines which of the specified grants — seccomp profile, device plugin, +`supplementalGroups` — are load-bearing rather than inert, so the pod spec can +be narrowed to the minimum that works. Every outcome is a pod-spec edit this +record already anticipates. + +The ruling *would* reopen only if some component required a Linux **capability** +or `privileged: true`. No mechanism in the composition is known to: cloud- +hypervisor, virtiofsd and passt are ordinary user binaries by the frozen +runtime record's Global Constraint. Treat that as the low-probability tail, not +the expected case. ### OQ-2 [load-bearing] — is the custom seccomp profile avoidable? -If `RuntimeDefault` turns out to suffice, the `Localhost` profile and its -on-node staging requirement both drop, which materially simplifies the -operator's job. R7 item 4 settles it. +The genuinely open question, and the only one with a real cost attached. If +`RuntimeDefault` suffices, the `Localhost` profile and its on-node staging +requirement both drop, which materially simplifies the operator's job. If it +does not, we ship the profile — a known, bounded cost this record already +specifies, not a setback. R8 item 4 settles it. ### OQ-3 [non-load-bearing] — device-plugin implementation pick @@ -440,7 +470,8 @@ D7. Default path chosen at implementation. ## Resolved decisions - **Containerize the Runner** as the Kubernetes delivery unit, rather than a - host systemd service — contingent on OQ-1 (§Container vs host process). + host systemd service — only if R8 surfaced a capability requirement, which + no known mechanism in the composition needs (§Container vs host process). - **Never `privileged: true`** — the microVM isolation boundary is the reason the Runner exists, and a privileged Runner re-opens the host path (§Alternatives considered). diff --git a/docs/designs/platform/compass-runner-containerization/spike-findings.md b/docs/designs/platform/compass-runner-containerization/spike-findings.md index 37ae83985..06088b39e 100644 --- a/docs/designs/platform/compass-runner-containerization/spike-findings.md +++ b/docs/designs/platform/compass-runner-containerization/spike-findings.md @@ -6,11 +6,19 @@ recorded shape of the verification so the questions and their negative controls are fixed before anyone runs them, rather than being chosen after seeing results. -The design record's §Privilege shape is a contract *contingent on this spike*. -Two of its grants rest on `[INFERENCE]` claims about upstream -Kubernetes/CRI/cgroup-v2 and node-image behaviour, grounded in no artifact in -this repo. This spike is what converts them to evidence or reopens the option -set. +**This spike narrows a pod spec; it does not ask whether containerization +works.** The composition already boots on Linux with `/dev/kvm` — the frozen +[microVM CI/dev enablement](../../infra/runtime/compass-elastic-session-runtime/microvm-ci-dev-enablement.md) +record runs KVM-backed boot tests as a required leg on GitHub Actions' +`ubuntu-latest`. What a pod adds is confinement: a cgroup device controller, a +seccomp filter, and a memory cgroup. So every item below asks **which grant the +confinement makes necessary**, and each answer costs at most a wider pod spec +the design record already specifies. + +Two grants rest on `[INFERENCE]` claims about upstream Kubernetes/CRI/cgroup-v2 +and node-image behaviour, grounded in no artifact in this repo. Converting them +to evidence is the point: the valuable outcome is a *smaller* contract, since a +grant that proves inert drops out. ## Why each item carries a negative control @@ -21,17 +29,21 @@ the finding — it shrinks the contract. ## Items -### S1 — does a zero-privilege pod boot the composition end to end? +### S1 — does the confinement hold, given the composition already boots? -Run the §Privilege shape pod spec on a real node and boot a session microVM: -cloud-hypervisor + virtiofsd + passt, through to an agent session that -executes. +The KVM baseline is established (see above), so this item is scoped to what +confinement changes. Run the §Privilege shape pod spec on a real node and boot +a session microVM: cloud-hypervisor + virtiofsd + passt, through to an agent +session that executes. -- **Expected:** boots. +- **Expected:** boots. S2-S4 attribute any shortfall to a specific layer, so + this item's job is to say whether the assembled spec is sufficient, not to + establish feasibility. - **Negative control:** none needed; this is the load-bearing positive. -- **If it fails:** OQ-1 resolves against containerization and the - container-vs-host tradeoff reopens. Do **not** grant a capability to make it - pass — that is the decision this record forbids. +- **If it fails:** expect the cause to be one of the named grants, and widen + that grant — a pod-spec edit, not a design change. Only a requirement for a + Linux **capability** or `privileged: true` reopens the container-vs-host + ruling, and no known mechanism needs one. - **Record:** the failing syscall and the component that needed it. ### S2 — does the hostPath char-device route actually fail? @@ -62,7 +74,9 @@ With the device plugin injecting the device, run as the non-root runner uid the `supplementalGroups` grant is unnecessary and drops from the contract, along with the gid-value question. - **Also record:** the node's actual `/dev/kvm` mode and owner, since the - inference rests on it. + inference rests on it. The compass dev box measures `crw-rw---- root:kvm` + (2026-09-12), under which the grant is required; a world-readable mode would + make it inert, so the node's own mode is the thing that decides it. ### S4 — is the custom `Localhost` seccomp profile required? From 7d14e206d9261db2e72b9988dfeee15e6e647d4a Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 14:31:02 -0400 Subject: [PATCH 4/8] docs(design): fix DL-358 framing and the R8 -> R7 task refs (RIG-3336) The reframe commit corrected design.md but left two things stale. The DL-358 ledger row still said the ruling was "explicitly CONTINGENT" on the spike and that the container-vs-host tradeoff "reopens" if it disappoints. That is the framing the reframe removed, sitting in the durable one-line summary a reader hits before the record. Rewrite it as a grant-set question, cite the KVM CI leg as the reason feasibility is settled, and state the capability/privileged tail as the only reopening outcome. Also restate the seccomp profile as specified-but-open (OQ-2) rather than flatly required. Prose referred to the privilege spike as R8 in 17 places; the task is R7 and no R8 exists, so every "R8 verifies it" pointed at nothing. --- docs/designs/DECISIONS.md | 2 +- .../compass-runner-containerization/design.md | 32 +++++++++---------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index 7d109d220..09a23a2a0 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -471,4 +471,4 @@ check enforces the mechanical half. Full rationale: | DL-347 | Multi-actor comms coverage is tiered: cross-agent conversation, fan-out/isolation, and offline redelivery are asserted at the podman e2e tier with agent authorship carried by the real Runner session→account binding (no per-agent credential — `relay_comms.go:7-15`), and a non-admin observer client proves the credentialed transport path; the D9 visibility/leak matrix stays at the pgtest tier with e2e proving only one transport positive+negative. every agent-reachable relay arm is covered by a unit test at the runnerhub tier (podman is spent on the tool→relay→server JOIN, never on arm dispatch), and each native agent tool has at least one real-server execution. No new build tag or CI tier — new legs ride the existing e2e skip-guard | Active (Matt, 2026-09-08) | [multi-actor comms coverage](server/compass-comms-multi-actor-test-coverage/design.md#resolved-decisions) | | DL-348 | The UI emits the PostHog session id on outbound Connect requests via a `sessionIdInterceptor` in `@compass/client` (sibling to `traceResponseInterceptor`), completing the J1 correlation-key seam whose inbound half stamps semconv `session.id` on backend spans. `X-POSTHOG-SESSION-ID` ONLY — `X-POSTHOG-DISTINCT-ID` is permanently excluded because it identifies a person and backend spans land in Grafana/Tempo, the plane J1 keeps identity out of; identity resolution stays PostHog-side where `identify()` holds it. Supersedes the 2026-09-05 HOLD (RIG-3233), whose two reasons — no inbound consumer, and a lazy-getter-over-nothing boot workaround — are discharged by the server merge and by the boot reorder respectively. posthog-js's own `TracingHeaders` extension is rejected: it monkey-patches global fetch/XHR and also sends the distinct-id header | Active (Matt, 2026-09-07) | [outbound session header](ui/compass-outbound-session-header/design.md#the-outbound-interceptor) | | DL-349 | Boot constructs analytics BEFORE the live clients (`createAnalytics` → `createLiveClients` → `bootCaller` → `identify`), so the transport's session-id getter closes over a real `Analytics` rather than a mutable ref slot or a forward `let`. Rejected alternatives: mirroring the `clients.traceId` sink (a workaround for a construction-order problem the reorder deletes) and a forward `let analytics` (a TDZ-shaped `undefined` window every reader must guard). Accepted trade, stated not silent: on the WhoAmI-failure early return an analytics-enabled deployment now emits an anonymous PostHog session (init-time remote-config egress) where it previously emitted nothing — the reorder is unavoidable while the transport needs the getter at construction, and the egress itself is declined-not-absent (`advanced_disable_flags` would suppress it, at the cost of remote config), with nothing captured either way. The sender-side guard is printable ASCII + `.length ≤ 200`, STRICTER than the server's `≤200 bytes` + valid-UTF-8 pair and NARROWER than `Headers.set` itself, because `Headers.set` is a WebIDL ByteString: a well-formed id above U+00FF THROWS inside the interceptor and would fail the RPC, and U+0080–U+00FF is accepted by `set` but a browser then emits it as a SINGLE RAW HIGH BYTE on the wire, which fails Go's `utf8.ValidString`, so `sessionIDFromHeader` returns `""` and the id is DROPPED — silent loss, the same failure class as every other rejected value, NOT a wrong correlation key the server accepts (measured: raw-TCP wire bytes `736573732de9` from Node/undici, which serializes like a browser; and real Chromium → real Go `net/http` running a verbatim `sessionIDFromHeader` copy — `len` 6, `utf8.ValidString` false, result `""`. A Bun-client-to-`Bun.serve` round-trip measures Bun's own encode/decode pair, not the wire, and is NOT valid evidence here) | Active (Matt, 2026-09-07) | [outbound session header](ui/compass-outbound-session-header/design.md#the-boot-reorder-chosen--approved-by-matt) | -| DL-358 | The Compass Runner is containerized as the Kubernetes delivery and lifecycle unit — a DaemonSet pod running rootless with a scoped `/dev/kvm` device, NEVER `privileged: true`. The container wins over a host systemd service on four operator properties (atomic digest-named userland, one declarative fleet object, userland isolated from the node, no node mutation on upgrade); the privilege question is the price, and the ruling is explicitly CONTINGENT on a real-hardware spike (R7) proving a zero-privilege pod boots cloud-hypervisor + virtiofsd + passt — if it cannot, the container-vs-host tradeoff reopens rather than a capability being granted. `/dev/kvm` arrives by device plugin, not a hostPath char device: [INFERENCE] the cgroup device controller denies `open()` without CRI `Devices` injection, so hostPath functions only under privileged mode — R7 verifies it with a negative control. Guest RAM is POD RAM (cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup), therefore DaemonSet requests must be sized to the node's aggregate session capacity or the kubelet ranks the Runner first for node-pressure eviction and kills every session on the node. Container restart implies full session teardown — FORCED by the pid namespace (the Runner is pid 1, so its death reaps every VMM descendant), not elected. The image carries the whole KVM userland and guest assets, holding hostPath to exactly two mounts. A custom `Localhost` seccomp profile is required because the composition needs `unshare`/`mount`/`pivot_root` that `RuntimeDefault` restricts — the host-rootless affordance a pod removes; if R7 shows `RuntimeDefault` suffices, the profile and its on-node staging both drop. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred | Active (Matt, 2026-09-12) | [runner containerization](platform/compass-runner-containerization/design.md#resolved-decisions) | +| DL-358 | The Compass Runner is containerized as the Kubernetes delivery and lifecycle unit — a DaemonSet pod running rootless with a scoped `/dev/kvm` device, NEVER `privileged: true`. The container wins over a host systemd service on four operator properties (atomic digest-named userland, one declarative fleet object, userland isolated from the node, no node mutation on upgrade); the privilege question is the price, and it is a grant-set question, NOT a feasibility one — the composition already boots on KVM as a required CI leg (see the frozen microVM CI/dev enablement record), so a pod adds only confinement (cgroup device controller, seccomp filter, memory cgroup). R7 narrows which listed grants are necessary rather than inert; every outcome is a pod-spec edit this record already specifies. The ruling would reopen only if some component required a Linux capability or `privileged: true`, which no mechanism in the composition is known to need. `/dev/kvm` arrives by device plugin, not a hostPath char device: [INFERENCE] the cgroup device controller denies `open()` without CRI `Devices` injection, so hostPath functions only under privileged mode — R7 verifies it with a negative control. Guest RAM is POD RAM (cloud-hypervisor's virtio-fs path requires `--memory shared=on`, so guest memory is charged to the pod's memory cgroup), therefore DaemonSet requests must be sized to the node's aggregate session capacity or the kubelet ranks the Runner first for node-pressure eviction and kills every session on the node. Container restart implies full session teardown — FORCED by the pid namespace (the Runner is pid 1, so its death reaps every VMM descendant), not elected. The image carries the whole KVM userland and guest assets, holding hostPath to exactly two mounts. A custom `Localhost` seccomp profile is specified because the composition needs `unshare`/`mount`/`pivot_root` that `RuntimeDefault` restricts — the host-rootless affordance a pod removes. Whether it is avoidable is the one load-bearing open question (OQ-2): if R7 shows `RuntimeDefault` suffices, the profile and its on-node staging both drop; if not, shipping it is a known bounded cost, not a setback. Cluster choice, node provisioning, and manifest delivery are operator concerns, named and deferred | Active (Matt, 2026-09-12) | [runner containerization](platform/compass-runner-containerization/design.md#resolved-decisions) | diff --git a/docs/designs/platform/compass-runner-containerization/design.md b/docs/designs/platform/compass-runner-containerization/design.md index f0e863f20..7c9ec1ff6 100644 --- a/docs/designs/platform/compass-runner-containerization/design.md +++ b/docs/designs/platform/compass-runner-containerization/design.md @@ -1,6 +1,6 @@ # Containerizing the Compass Runner -Status: Draft — freezes on merge. §Privilege shape names the grants R8 must confirm are *necessary*; R8 narrows the grant set, it does not decide whether containerization works. +Status: Draft — freezes on merge. §Privilege shape names the grants R7 must confirm are *necessary*; R7 narrows the grant set, it does not decide whether containerization works. Ledger-impact: mints DL-358 @@ -56,9 +56,9 @@ already verified to boot on Linux with `/dev/kvm`: the frozen [microVM CI/dev enablement](../../infra/runtime/compass-elastic-session-runtime/microvm-ci-dev-enablement.md) record runs KVM-backed boot tests as a required leg on GitHub Actions' `ubuntu-latest`. What a pod adds over that environment is confinement — a -cgroup device controller, a seccomp filter, and a memory cgroup — so R8 asks +cgroup device controller, a seccomp filter, and a memory cgroup — so R7 asks **which grants the confinement makes necessary**, not whether microVMs run in -containers. Each plausible R8 outcome costs a wider pod spec (a seccomp +containers. Each plausible R7 outcome costs a wider pod spec (a seccomp profile, a device plugin, a supplemental gid), all of which this record already specifies; none of them reopens the container-vs-host ruling. The one result that *would* reopen it is a requirement for a Linux **capability** or @@ -84,7 +84,7 @@ provide: template change, not a node reprovision. The cost is the privilege question, which is the subject of §Privilege shape -and R8. +and R7. ### Net backend: passt, implemented, not an open pick @@ -95,11 +95,11 @@ in-guest; the Runner dials **out** to the Server over gRPC with its per-Runner token (`go/cmd/compass-runner/main.go`), so the pod needs no host port and no inbound service. -### Privilege shape (the pod spec — the grant set R8 narrows) +### Privilege shape (the pod spec — the grant set R7 narrows) -Every grant is justified; anything not listed is denied. R8 confirms which +Every grant is justified; anything not listed is denied. R7 confirms which grants are *necessary* rather than inert, on real hardware, before R3 encodes -them — a grant that proves inert drops out, which is the outcome to hope for. The rulings pick what R8 verifies first; they do not remove the +them — a grant that proves inert drops out, which is the outcome to hope for. The rulings pick what R7 verifies first; they do not remove the verification. - **`/dev/kvm` via a device plugin — NOT `privileged: true`, NOT a raw @@ -122,14 +122,14 @@ verification. candidate*. If it is wrong, hostPath returns to the option set and the device-plugin requirement must be re-argued on posture grounds — a change to *which* mechanism delivers the device, not to whether containerization works. - **R8 verifies it**: + **R7 verifies it**: attempt the hostPath route on a real node and confirm `open()` actually fails without device-plugin injection. The device-plugin implementation is an operator pick, not frozen here; any community plugin image an operator pins is thinly maintained and should be pinned by digest. The contract this record carries is *scoped device node, - zero capabilities, no privileged mode*; R8 confirms the device-plugin + zero capabilities, no privileged mode*; R7 confirms the device-plugin mechanism is the one that delivers it. - **`securityContext`:** `runAsNonRoot: true`, `runAsUser`/`runAsGroup` fixed @@ -151,7 +151,7 @@ verification. `root:kvm 0660` — **measured `crw-rw---- root:kvm` on the compass dev box (2026-09-12)**, which supersedes the `crw-rw-rw-` reading in the frozen microVM CI/dev enablement record; the grant is inert under a world-readable - mode and required under `0660`, so R8 must record the node's actual mode — + mode and required under `0660`, so R7 must record the node's actual mode — and device injection grants a *cgroup allowance*, not filesystem permission — so the non-root runner uid needs the `kvm` gid via `securityContext.supplementalGroups` (or a node-provisioning chmod), or the @@ -160,7 +160,7 @@ verification. device-controller denial in the `/dev/kvm` bullet above is `EPERM`. Marked inference because the device node's mode and ownership are properties of the node image's udev rules, not of anything in this repo, and because it is the sole justification for the - `supplementalGroups` grant. **R8 verifies it directly**, and the negative + `supplementalGroups` grant. **R7 verifies it directly**, and the negative control is the one that matters: confirm a non-root uid *without* the kvm gid actually fails to open the device. If it opens without the gid, the grant is unnecessary and drops out of the contract. @@ -324,7 +324,7 @@ first thing a Kubernetes reader reaches for. - **No capability, no `privileged`.** The grant set may widen along the axes §Privilege shape already names (seccomp profile, device plugin, supplemental - gid) — that is R8 narrowing or confirming a spec, and is expected. What is + gid) — that is R7 narrowing or confirming a spec, and is expected. What is banned is patching a shortfall with a Linux capability or `privileged: true`: if the composition genuinely needed one, the container-vs-host ruling is reopened instead. No mechanism in the composition is known to need one. @@ -377,7 +377,7 @@ bounded. Document and encode the device-plugin requirement, including the resource-name parameterization and the `supplementalGroups` gid — both confirmed necessary -(or dropped as inert) by R8. +(or dropped as inert) by R7. Test cycle: rendered pod spec requests the device resource and carries the gid; a spec that omits either fails the assertion. @@ -432,7 +432,7 @@ independent. R6 lands with the freeze. ### OQ-1 [non-load-bearing] — which grants does confinement make necessary? Not a feasibility question. The composition boots on KVM today (see §Approach); -R8 determines which of the specified grants — seccomp profile, device plugin, +R7 determines which of the specified grants — seccomp profile, device plugin, `supplementalGroups` — are load-bearing rather than inert, so the pod spec can be narrowed to the minimum that works. Every outcome is a pod-spec edit this record already anticipates. @@ -449,7 +449,7 @@ The genuinely open question, and the only one with a real cost attached. If `RuntimeDefault` suffices, the `Localhost` profile and its on-node staging requirement both drop, which materially simplifies the operator's job. If it does not, we ship the profile — a known, bounded cost this record already -specifies, not a setback. R8 item 4 settles it. +specifies, not a setback. R7 item 4 settles it. ### OQ-3 [non-load-bearing] — device-plugin implementation pick @@ -470,7 +470,7 @@ D7. Default path chosen at implementation. ## Resolved decisions - **Containerize the Runner** as the Kubernetes delivery unit, rather than a - host systemd service — only if R8 surfaced a capability requirement, which + host systemd service — only if R7 surfaced a capability requirement, which no known mechanism in the composition needs (§Container vs host process). - **Never `privileged: true`** — the microVM isolation boundary is the reason the Runner exists, and a privileged Runner re-opens the host path From d37f1b7fcc8046c1cf7dd1dffc337e0ca6396bd2 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 14:40:49 -0400 Subject: [PATCH 5/8] docs(design): R1 builds a Dockerfile on a hardened base, not a nix image (RIG-3336) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R1 said "a nix-built container image project ... mirroring the existing agent-image lane". That contradicts the private monorepo's frozen first-party-image-builds spec, which splits the two mechanisms by what the image's RUNTIME is rather than by what built the artifact: an image that IS a Nix environment (a CI step image, a dev/agent shell) earns nix2container, while a prebuilt application on a minimal base is a Dockerfile built by rootless BuildKit. The spec forecloses the exact inference R1 made — a first-party image does not earn the nix path "merely because Nix built it or it ships a compiled binary" — and rejects nix2container for app images on cost. The Runner is a prebuilt application: a static CGO_ENABLED=0 binary that exec's three userland binaries and runs no toolchain at runtime. agent-image sits on the other row by name, because that container's job is to be a toolchain; sharing the *-image/ directory shape is not sharing the mechanism. Distroless suits a managed multi-tenant cluster, but the KVM userland comes from nixpkgs and carries store-path interpreter/rpath references, so R1 must settle how those binaries run on a minimal base. R1's test cycle now asserts each carried binary EXECUTES, which a layer-contents check alone would miss. R2 gains the build mechanism and the digest-pinning contract: GHCR has no server-side tag immutability, so R3 pins repo@sha256, never a tag. --- .../compass-runner-containerization/design.md | 44 ++++++++++++++++--- 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/docs/designs/platform/compass-runner-containerization/design.md b/docs/designs/platform/compass-runner-containerization/design.md index 7c9ec1ff6..97145cb15 100644 --- a/docs/designs/platform/compass-runner-containerization/design.md +++ b/docs/designs/platform/compass-runner-containerization/design.md @@ -352,14 +352,46 @@ first thing a Kubernetes reader reaches for. ### R1 — `runner-image/` project -A nix-built container image project in this repo, mirroring the existing -agent-image lane: Runner binary, cloud-hypervisor, virtiofsd, passt, guest -kernel/rootfs/initrd. Test cycle: the image builds reproducibly and the -resulting layer set contains each expected binary and guest asset. +A `Dockerfile` built on a **minimal hardened base** (distroless, or Alpine +where a shell is genuinely needed), carrying the Runner binary, +cloud-hypervisor, virtiofsd, passt, and the guest kernel/rootfs/initrd. + +**Not a nix image.** The private monorepo's frozen first-party-image-builds +spec splits the two mechanisms by what the image's *runtime* is, not by what +built the artifact: an image that **is** a Nix environment (a CI step image, a +dev/agent shell) earns `nix2container`; a **prebuilt application** on a minimal +base is a Dockerfile built by rootless BuildKit. Its words: a first-party image +does not earn the nix path "merely because Nix built it or it ships a compiled +binary — build tool and runtime base are independent choices", and the +prescribed shape for a nix-built artifact is to `nix build` it and `COPY` the +result onto the base. `nix2container` for app images is rejected there on cost +— a per-app regeneration tax plus a maintained fork. + +The Runner is squarely the prebuilt-application row: a static `CGO_ENABLED=0` +Go binary that exec's three userland binaries and runs no package manager, +toolchain, or `nix` at runtime. The agent-image lane sits on the *other* row by +name, because that container's job is to be a toolchain. Sharing the +`*-image/` directory shape is not sharing the mechanism. + +Distroless is the stronger posture for a managed multi-tenant cluster and fits +the Runner's shape — but the KVM userland binaries come from nixpkgs and carry +store-path interpreter and rpath references, so R1 must settle how they are +made runnable on a minimal base (a static or patchelf'd copy, or an Alpine base +with the loader present). That is R1's one real engineering question. + +Test cycle: the image builds reproducibly; the resulting image contains each +expected binary and guest asset; every carried binary actually executes on the +chosen base (the store-path-reference check above, which a layer-contents +assertion alone would miss). ### R2 — publish lane -Extend the release workflow to build and publish the runner image by digest. +Extend the release workflow to build and publish the runner image by digest, +built with rootless BuildKit (`buildkitd`/`buildctl`) per the spec cited in R1 +— never `docker build`, never a host docker socket. The deployed contract is +the resolved immutable digest (`repo@sha256:…`), not a tag: GHCR has no +server-side tag immutability, so R3's DaemonSet pins the digest. + Test cycle: a tagged run publishes a manifest whose digest is recorded in the run output; a second build of the same input yields the same digest. @@ -419,7 +451,7 @@ independent. R6 lands with the freeze. | Task | Deliverable | Depends on | | ------ | ----------- | ---------- | -| R1 | `runner-image/` nix project | — | +| R1 | `runner-image/` Dockerfile on a minimal hardened base | — | | R2 | publish lane by digest | R1 | | R3 | DaemonSet + RBAC manifests + render tests | R7 | | R4 | device-plugin resource + gid wiring | R7 | From b9301ecaec80cbdf209fa12cf9d7711bc783e411 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 15:06:05 -0400 Subject: [PATCH 6/8] feat(runner-image): carry the Runner, KVM userland and guest assets in a container image (RIG-3720) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R1 of the Runner containerization record. A Dockerfile on distroless, built by rootless BuildKit — the ruled mechanism for a prebuilt application. Nix still builds every carried artifact; it is the builder, never the runtime. The image carries the compass-runner binary, the KVM userland it exec's (cloud-hypervisor, virtiofsd, passt) and the three guest assets the microVM backend direct-boots, so a Runner pod needs no host toolchain. WHY DISTROLESS STATIC AND NOT :base. The carried binaries are dynamically linked but nix-closed: each names an absolute /nix/store interpreter and resolves every NEEDED library through its own RPATH, so the closure supplies its own ld.so and glibc and the base is never consulted for a library. That makes the base's libc dead weight. Measured, not assumed: with the closure staged all four binaries execute on distroless-static; with only cloud-hypervisor's own store path copied and its glibc omitted, the same exec fails "missing dynamic library" — the control proving the loader comes from the closure, not the base. Two consequences that are easy to get wrong, both encoded in the lane: - The staged context is the TRANSITIVE closure, not the five roots. A context missing the shared glibc yields an image whose binaries cannot exec at all. - The entrypoint is a RELATIVE symlink inside store/. An absolute symlink at the context root dangles on the build host, and BuildKit checksums context entries before any COPY runs, so it fails the build outright rather than resolving later inside the image. The realise-and-stage logic is TypeScript (the no-bash-gate task), split pure-core/thin-runner like the microvm-boot-test lane: build-core.ts holds the out-path parsing, the artifact->build-arg mapping and the closure-root selection, covered by 13 unit tests. The image build itself is runInCI:false — it needs a buildkitd, which no compass CI step carries until R2 — so the gate coverage is that suite, which is where a mapping drift is actually caught. Verified end-to-end on a rootless buildkitd: the lane builds the real image; its entrypoint runs as uid 65532; all four binaries execute in-image; passt resolves by name off PATH; all three guest assets match their host sizes byte for byte. --- .gitignore | 6 + .moon/workspace.yml | 16 ++ bun.lock | 12 ++ runner-image/.dockerignore | 6 + runner-image/Dockerfile | 102 +++++++++++ runner-image/moon.yml | 67 +++++++ tools/runner-image/biome.json | 4 + tools/runner-image/build-core.test.ts | 141 +++++++++++++++ tools/runner-image/build-core.ts | 141 +++++++++++++++ tools/runner-image/build.ts | 246 ++++++++++++++++++++++++++ tools/runner-image/moon.yml | 34 ++++ tools/runner-image/package.json | 14 ++ tools/runner-image/tsconfig.json | 11 ++ 13 files changed, 800 insertions(+) create mode 100644 runner-image/.dockerignore create mode 100644 runner-image/Dockerfile create mode 100644 runner-image/moon.yml create mode 100644 tools/runner-image/biome.json create mode 100644 tools/runner-image/build-core.test.ts create mode 100644 tools/runner-image/build-core.ts create mode 100755 tools/runner-image/build.ts create mode 100644 tools/runner-image/moon.yml create mode 100644 tools/runner-image/package.json create mode 100644 tools/runner-image/tsconfig.json diff --git a/.gitignore b/.gitignore index 82bd83e1e..211db5d14 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,9 @@ result-* # NOTE: generated clients (go/gen, packages/compass-client/src/gen) # are intentionally committed and CI drift-gated — do not ignore them. + +# runner-image build outputs: the staged nix closure, the entrypoint symlink and +# the OCI layout are realised by tools/runner-image/build.ts, never committed. +/runner-image/store/ +/runner-image/out/ +/runner-image/compass-runner diff --git a/.moon/workspace.yml b/.moon/workspace.yml index a718eec44..5b06d8578 100644 --- a/.moon/workspace.yml +++ b/.moon/workspace.yml @@ -39,12 +39,28 @@ projects: # cache lane or V2a boot bring-up. Same affected-detection posture as # compass-agent-image (see guest-image/moon.yml). compass-guest-image: 'guest-image' + # The Runner container image (R1): the compass-runner binary, the KVM userland + # and the guest assets on a minimal hardened base. A Dockerfile project, not a + # nix one — the fleet image-builds spec routes a prebuilt application on a + # minimal base to rootless BuildKit, and nix2container only to images that ARE + # a Nix environment. Its `build` is runInCI:false (it needs a buildkitd and a + # multi-GB realise), so registration here is for graph membership and + # affected-detection rather than a gate leg; the gate coverage is the + # runner-image tool's pure-core suite below. See runner-image/moon.yml. + compass-runner-image: 'runner-image' # The local microVM boot-test lane (RIG-2591): realises the guest image + VMM # stack from nix and execs the KVM-gated `go test -tags microvm` suite the # untagged compass-go:test lane never builds. Registered so its typecheck + # unit tests ride the moon-driven CI sweep; the boot lane itself lives on # compass-go:test-microvm (runInCI:false — it needs KVM + a nix build). microvm-boot-test: 'tools/microvm-boot-test' + # The runner-image build lane (R1): realises the nix closure, stages it, and + # drives the rootless-BuildKit Dockerfile build. TypeScript rather than a + # shell script (the no-bash-gate CI task). Registered so its typecheck + the + # pure-core unit tests ride the moon-driven CI sweep — that suite is what + # catches a build-arg or closure-root drift, since the image build itself + # cannot run in CI until R2 stands up a buildkitd. + runner-image: 'tools/runner-image' # The Compass native-app release bundle: a heavy nix build (realises the # WebKitGTK cc/pkg-config closure) that stages the versioned tarball. Same # affected-detection posture as compass-agent-image — registered here so the diff --git a/bun.lock b/bun.lock index 7e4f2f793..786a61944 100644 --- a/bun.lock +++ b/bun.lock @@ -236,6 +236,16 @@ "typescript": "catalog:", }, }, + "tools/runner-image": { + "name": "@compass/runner-image", + "bin": { + "runner-image-build": "./build.ts", + }, + "devDependencies": { + "@types/bun": "catalog:", + "typescript": "catalog:", + }, + }, "tools/sea-ref-gate": { "name": "@compass/sea-ref-gate", "bin": { @@ -491,6 +501,8 @@ "@compass/renovate-preflight": ["@compass/renovate-preflight@workspace:tools/renovate-preflight"], + "@compass/runner-image": ["@compass/runner-image@workspace:tools/runner-image"], + "@compass/sea-ref-gate": ["@compass/sea-ref-gate@workspace:tools/sea-ref-gate"], "@compass/sql-migration-gate": ["@compass/sql-migration-gate@workspace:tools/sql-migration-gate"], diff --git a/runner-image/.dockerignore b/runner-image/.dockerignore new file mode 100644 index 000000000..4080d8467 --- /dev/null +++ b/runner-image/.dockerignore @@ -0,0 +1,6 @@ +# Context prune for the runner image build. The context is runner-image/ itself, +# which holds only the Dockerfile, the staged store/ closure and the +# compass-runner entrypoint symlink — so this excludes the build OUTPUT (out/), +# which would otherwise be re-transferred into every rebuild's context and grow +# it without bound. +out diff --git a/runner-image/Dockerfile b/runner-image/Dockerfile new file mode 100644 index 000000000..337faf9a6 --- /dev/null +++ b/runner-image/Dockerfile @@ -0,0 +1,102 @@ +# syntax=docker/dockerfile:1.7-labs +# The compass-runner container image (R1 of the Compass Runner containerization +# record). Carries the Runner binary, the KVM userland it exec's +# (cloud-hypervisor, virtiofsd, passt) and the three guest assets the microVM +# backend direct-boots, so a Runner pod needs no host toolchain — only /dev/kvm +# and the session-volume mount. +# +# WHY A DOCKERFILE AND NOT nix2container. The fleet's first-party image-builds +# spec splits the two mechanisms by what the image's RUNTIME is, not by what +# built the artifact: an image that IS a Nix environment (a CI step image, a +# dev/agent shell) earns nix2container; a PREBUILT APPLICATION on a minimal base +# is a Dockerfile built by rootless BuildKit. It forecloses the tempting +# inference in terms — an image does not earn the nix path "merely because Nix +# built it or it ships a compiled binary" — and rejects nix2container for app +# images on cost. The Runner is the prebuilt-application row: it runs no +# toolchain, it exec's four binaries. The sibling agent-image/ sits on the other +# row BY NAME, because that container's job is to be a toolchain; sharing the +# `*-image/` directory shape is not sharing the mechanism. +# +# The prescribed shape for a nix-built artifact is therefore `nix build` it and +# COPY the result onto the base — which is exactly what build.sh stages into +# `store/` and what the single COPY below installs. + +# Distroless STATIC, not :base and not Alpine. The carried binaries are +# dynamically linked but nix-closed: each names an ABSOLUTE /nix/store +# interpreter and resolves every NEEDED library through its own RPATH, so the +# closure supplies its own ld.so and glibc and the base is never consulted for a +# library. That makes the base's own libc dead weight, and `static` is the +# smallest base that still provides what IS needed from it: /etc/passwd + +# /etc/group (the nonroot uid), CA certificates (the Runner dials the Server +# over TLS), and /tmp. Verified: with the closure present all four binaries +# execute; with ONLY cloud-hypervisor's own store path copied — its glibc +# omitted — the same exec fails "missing dynamic library", which is the control +# proving the loader comes from the closure rather than the base. +# +# :nonroot selects uid/gid 65532. The Runner does not need root: R4 grants +# /dev/kvm access by supplementary group, not by uid. Pinned by digest, never by +# tag, for the same reason R2 pins this image's own digest downstream. +FROM gcr.io/distroless/static-debian12@sha256:c0f429e16b13e583da7e5a6ec20dd656d325d88e6819cafe0adb0828976529dc + +# The realised nix closure: the Runner binary, the KVM userland and the guest +# assets, each at its own absolute /nix/store path. ONE COPY of a +# build.ts-staged directory rather than a COPY per component, because the store +# paths reference each other by absolute path — splitting them across layers by +# component would interleave a binary and its glibc into different layers for no +# cache benefit, since a Go rebuild moves the Runner path and nothing else. +# +# Ownership is left at the default (root-owned, world-readable) and the image +# runs as nonroot: the closure is read-only at runtime, so a non-root process +# needs no write access to it. This also keeps the layer bit-identical to the +# staged tree, which is what makes the digest reproducible across rebuilds. +COPY store /nix/store + +# The stable entrypoint name is /nix/store/.compass-runner, a RELATIVE symlink +# build.ts stages inside store/ (so it arrives with the COPY above — there is no +# second COPY). ENTRYPOINT is exec-form and cannot expand a build ARG, while the +# Runner's own store path carries a hash that moves on every Go rebuild; the +# symlink gives the image a fixed entrypoint whose target moves with the build. +# +# It must be relative and inside the staged tree. An absolute symlink at the +# context root dangles on the BUILD HOST, and BuildKit checksums context entries +# before any COPY runs, so such a link fails the build outright with "not found" +# rather than resolving later inside the image. + +# Absolute store paths, resolved at build time from the staged closure by +# build.ts and passed in as build args. They are baked into ENV rather than +# hardcoded because every path carries a content hash that moves on any rebuild +# of its input. +# +# The Runner reads each of these as the documented env fallback for the matching +# --microvm-* flag (main.go registerBackendFlags), so an operator can still +# override any one on the command line without rebuilding the image. Setting +# them here is what makes the image self-describing: the microVM backend's +# preflight is fail-closed (it requires both a static support probe and a real +# boot canary), so a missing or wrong path fails at startup, loudly, rather than +# at first session. +ARG RUNNER_BIN +ARG VMM_BIN +ARG VIRTIOFSD_BIN +ARG GUEST_KERNEL +ARG GUEST_ROOTFS +ARG GUEST_INITRD + +ENV COMPASS_RUNTIME_BACKEND=microvm \ + COMPASS_MICROVM_VMM=${VMM_BIN} \ + COMPASS_MICROVM_VIRTIOFSD=${VIRTIOFSD_BIN} \ + COMPASS_MICROVM_KERNEL=${GUEST_KERNEL} \ + COMPASS_MICROVM_ROOTFS=${GUEST_ROOTFS} \ + COMPASS_MICROVM_INITRD=${GUEST_INITRD} + +# passt is exec'd by name, not by configured path (unlike the VMM and virtiofsd, +# which have --microvm-* flags), so its bin dir must be on PATH. Prepended to +# the base PATH rather than replacing it. +ARG STACK_BIN_DIR +ENV PATH=${STACK_BIN_DIR}:/usr/local/bin:/usr/bin:/bin + +# ENTRYPOINT, not CMD: the image runs exactly one program, and an ENTRYPOINT +# lets a pod spec's `args` add flags without re-stating the binary path. Exec +# form, so the Runner is pid 1 and receives SIGTERM directly at pod shutdown — +# a shell wrapper would swallow it and leave the graceful drain to the kill +# timeout. +ENTRYPOINT ["/nix/store/.compass-runner"] diff --git a/runner-image/moon.yml b/runner-image/moon.yml new file mode 100644 index 000000000..8f123844a --- /dev/null +++ b/runner-image/moon.yml @@ -0,0 +1,67 @@ +# yaml-language-server: $schema=https://moonrepo.dev/schemas/project.json +# +# compass-runner-image (runner-image/): the Runner container image — the +# compass-runner binary, the KVM userland it exec's (cloud-hypervisor, +# virtiofsd, passt) and the three guest assets the microVM backend direct-boots. +# This project carries the Dockerfile and the staged build context; the +# realise-and-stage logic is tools/runner-image (TypeScript, not a shell script). +# +# WHY THIS IS A DOCKERFILE PROJECT AND NOT A NIX ONE. The sibling image projects +# (agent-image/, guest-image/) are `language: nix` because their artifacts ARE +# nix derivations. This one is not: the fleet's first-party image-builds spec +# routes a prebuilt application on a minimal base to a Dockerfile built by +# rootless BuildKit, and routes nix2container only to images that ARE a Nix +# environment. Nix still BUILDS every carried artifact — the spec's prescribed +# shape is `nix build` then COPY — but the image's runtime is a distroless base, +# not a Nix environment, and the mechanism follows the runtime. +# +# WHY THERE IS NO `ci` TASK. Unlike agent-image/ and guest-image/, this project +# deliberately does NOT ride the pre-merge gate: the build needs a reachable +# buildkitd (which no compass CI step carries today — R2 stands that up) and it +# stages a multi-GB closure dominated by the guest rootfs. Registering a `ci` +# task now would add an unrunnable task to the gate, so the gate coverage this +# project has today is the pure-core suite in tools/runner-image, which is where +# a mapping drift would actually be caught. R2 adds the publish lane and with it +# the first CI leg that builds this image. +layer: 'application' +language: 'bash' +tags: [] + +workspace: + inheritedTasks: + # Not a bun package — no package.json, no bun.lock — so the tag-bun `install` + # must never run here; `lint` and `format` are whole-repo tasks on the root + # project, never inherited. The same defensive guard the sibling image + # projects carry. + exclude: ['install', 'lint', 'format'] + +tasks: + build: + # Realise the closure and build the image. runInCI:false — the build needs a + # buildkitd and a multi-GB nix realise, so this is the dev-box/publish entry + # point, not a gate task. cache:false — nix owns the store-path caching and + # buildkit owns the layer caching, so moon must not false-green a build it + # did not re-run (the posture the sibling image projects take). + command: 'bun tools/runner-image/build.ts' + options: + cache: false + runInCI: false + runFromWorkspaceRoot: true + # The image's true build closure. A leading `/` is workspace-root-relative + # (the convention the sibling image projects use); the bare glob is this + # project's own tree. Each entry is an input whose change alters the built + # image: the Dockerfile and the build script directly, the Go module + + # runner command for the binary, and the guest-image/VMM-env closures for + # the carried assets. A glob that misses one would leave a stale image + # passing as current. + inputs: + - '**/*' + - '/tools/runner-image/**/*' + - '/flake.nix' + - '/flake.lock' + - '/go/go.mod' + - '/go/go.sum' + - '/go/cmd/compass-runner/**' + - '/go/internal/**' + - '/guest-image/**' + - '/tools/toolchain/microvm-vmm-env.nix' diff --git a/tools/runner-image/biome.json b/tools/runner-image/biome.json new file mode 100644 index 000000000..ece11dd9b --- /dev/null +++ b/tools/runner-image/biome.json @@ -0,0 +1,4 @@ +{ + "extends": "//", + "linter": { "rules": { "suspicious": { "noConsole": "off" } } } +} diff --git a/tools/runner-image/build-core.test.ts b/tools/runner-image/build-core.test.ts new file mode 100644 index 000000000..d87da1905 --- /dev/null +++ b/tools/runner-image/build-core.test.ts @@ -0,0 +1,141 @@ +// Unit tests for the runner-image build lane's pure core. Each case pins a +// behaviour the image's correctness depends on, and each would fail on a +// plausible bug — not on a restatement of the implementation. + +import { describe, expect, test } from "bun:test"; +import { + buildArgs, + buildctlArgs, + closureRoots, + kernelImagePath, + outputSpec, + parseOutPaths, + type RunnerImageOutputs, +} from "./build-core.ts"; + +const outputs: RunnerImageOutputs = { + runner: "/nix/store/aaa-compass-runner", + stack: "/nix/store/bbb-compass-stack-env", + kernelDir: "/nix/store/ccc-linux", + rootfs: "/nix/store/ddd-rootfs.erofs", + initrd: "/nix/store/eee-initrd", +}; + +describe("parseOutPaths", () => { + test("drops the blank trailing line nix emits", () => { + expect(parseOutPaths("/nix/store/a\n/nix/store/b\n")).toEqual([ + "/nix/store/a", + "/nix/store/b", + ]); + }); + + test("yields nothing for empty stdout, so a silent build failure cannot read as one path", () => { + expect(parseOutPaths("")).toEqual([]); + expect(parseOutPaths("\n \n")).toEqual([]); + }); +}); + +describe("kernelImagePath", () => { + // The kernel derivation is a directory (bzImage + System.map); the rootfs and + // initrd derivations ARE files. Booting the directory would fail at the VMM. + test("points at the bzImage inside the kernel derivation directory", () => { + expect(kernelImagePath("/nix/store/ccc-linux")).toBe( + "/nix/store/ccc-linux/bzImage", + ); + }); +}); + +describe("buildArgs", () => { + test("maps each artifact to the env the Runner reads, kernel suffixed and the file assets bare", () => { + expect(buildArgs(outputs)).toEqual({ + RUNNER_BIN: "/nix/store/aaa-compass-runner/bin/compass-runner", + VMM_BIN: "/nix/store/bbb-compass-stack-env/bin/cloud-hypervisor", + VIRTIOFSD_BIN: "/nix/store/bbb-compass-stack-env/bin/virtiofsd", + STACK_BIN_DIR: "/nix/store/bbb-compass-stack-env/bin", + GUEST_KERNEL: "/nix/store/ccc-linux/bzImage", + GUEST_ROOTFS: "/nix/store/ddd-rootfs.erofs", + GUEST_INITRD: "/nix/store/eee-initrd", + }); + }); + + test("every value is absolute, since the image bakes them as env the Runner resolves without a cwd", () => { + for (const value of Object.values(buildArgs(outputs))) { + expect(value.startsWith("/nix/store/")).toBe(true); + } + }); +}); + +describe("closureRoots", () => { + // The roots are derivation out-paths: `nix path-info -r` takes store paths, + // and passing the bzImage FILE instead of its directory would stage a partial + // closure whose binaries fail to exec. + test("uses the kernel DIRECTORY, not the bzImage inside it", () => { + const roots = closureRoots(outputs); + expect(roots).toContain("/nix/store/ccc-linux"); + expect(roots).not.toContain("/nix/store/ccc-linux/bzImage"); + }); + + test("covers all five artifacts, so no carried binary is staged without its dependencies", () => { + expect(closureRoots(outputs).sort()).toEqual( + [ + outputs.runner, + outputs.stack, + outputs.kernelDir, + outputs.rootfs, + outputs.initrd, + ].sort(), + ); + }); +}); + +describe("outputSpec", () => { + test("oci writes a browsable layout, which is what the publish lane scans before pushing", () => { + expect(outputSpec("oci", "ignored:tag", "/tmp/out")).toBe( + "type=oci,dest=/tmp/out,tar=false", + ); + }); + + test("image names the tag for a local load", () => { + expect(outputSpec("image", "compass-runner:dev", "/tmp/out")).toBe( + "type=image,name=compass-runner:dev", + ); + }); + + test("oci mode never names the tag, so a dev tag cannot leak into a layout build", () => { + expect(outputSpec("oci", "compass-runner:dev", "/tmp/out")).not.toContain( + "compass-runner:dev", + ); + }); +}); + +describe("buildctlArgs", () => { + test("passes every build-arg the Dockerfile declares", () => { + const args = buildctlArgs( + "/repo/runner-image", + outputs, + "linux/amd64", + "type=oci,dest=/repo/runner-image/out,tar=false", + ); + const joined = args.join(" "); + for (const name of Object.keys(buildArgs(outputs))) { + expect(joined).toContain(`build-arg:${name}=`); + } + }); + + test("build-arg order is deterministic, so two identical builds produce identical argv", () => { + const once = buildctlArgs("/ctx", outputs, "linux/amd64", "type=oci"); + const twice = buildctlArgs("/ctx", outputs, "linux/amd64", "type=oci"); + expect(once).toEqual(twice); + const names = once + .filter((a) => a.startsWith("build-arg:")) + .map((a) => a.slice("build-arg:".length).split("=")[0]); + expect(names).toEqual([...names].sort()); + }); + + test("uses the dockerfile.v0 frontend against the staged context", () => { + const args = buildctlArgs("/ctx", outputs, "linux/amd64", "type=oci"); + expect(args.slice(0, 3)).toEqual(["build", "--frontend", "dockerfile.v0"]); + expect(args).toContain("context=/ctx"); + expect(args).toContain("dockerfile=/ctx"); + }); +}); diff --git a/tools/runner-image/build-core.ts b/tools/runner-image/build-core.ts new file mode 100644 index 000000000..de211e6c8 --- /dev/null +++ b/tools/runner-image/build-core.ts @@ -0,0 +1,141 @@ +// The pure core of the runner-image build lane (R1). Every function here is a +// total map over its inputs with no I/O, so build-core.test.ts can drive each +// mapping — and each fail-closed edge — without nix, buildkit, or a subprocess. +// +// The lane's job is to turn realised nix store paths into the two things the +// container build needs: the set of paths to stage into the build context, and +// the build-args that bake those absolute paths into the image's env. + +/** The six artifacts the image carries, as realised store paths. */ +export interface RunnerImageOutputs { + /** The `compass-runner` package out-path (the binary is at `bin/compass-runner`). */ + runner: string; + /** The `compass-stack-env` symlinkJoin (cloud-hypervisor, virtiofsd, passt under `bin/`). */ + stack: string; + /** The guest kernel DERIVATION dir; the bootable artifact is `/bzImage`. */ + kernelDir: string; + /** The guest rootfs image — the derivation IS the file. */ + rootfs: string; + /** The guest initramfs — the derivation IS the file. */ + initrd: string; +} + +/** Split `nix build --print-out-paths` stdout into trimmed, non-empty store + * paths, one per line. Mirrors the microvm-boot-test lane's parser: the same + * command shape deserves the same reader, not a second convention. */ +export function parseOutPaths(stdout: string): string[] { + return stdout + .split("\n") + .map((line) => line.trim()) + .filter((line) => line !== ""); +} + +/** + * The absolute in-image path of the bootable kernel. The kernel derivation is a + * DIRECTORY (it also carries System.map); cloud-hypervisor direct-boots the + * bzImage inside it, with no bootloader. The rootfs and initrd derivations are + * files, so only this one gains a suffix — the asymmetry is the derivations', + * not a convention. + */ +export function kernelImagePath(kernelDir: string): string { + return `${kernelDir}/bzImage`; +} + +/** + * The `--opt build-arg:` values the Dockerfile bakes into ENV. Every value is an + * absolute /nix/store path resolved at build time, because each carries a + * content hash that moves whenever its input is rebuilt — hardcoding any of them + * in the Dockerfile would pin a stale artifact that the staged closure no longer + * contains. + * + * These land on the documented `COMPASS_MICROVM_*` env fallbacks for the + * matching `--microvm-*` flags, so an operator can still override any one at + * runtime without rebuilding. + */ +export function buildArgs(outputs: RunnerImageOutputs): Record { + return { + RUNNER_BIN: `${outputs.runner}/bin/compass-runner`, + VMM_BIN: `${outputs.stack}/bin/cloud-hypervisor`, + VIRTIOFSD_BIN: `${outputs.stack}/bin/virtiofsd`, + // passt is exec'd by NAME rather than by a configured path (it has no + // --microvm-* flag), so the image needs its bin dir on PATH. + STACK_BIN_DIR: `${outputs.stack}/bin`, + GUEST_KERNEL: kernelImagePath(outputs.kernelDir), + GUEST_ROOTFS: outputs.rootfs, + GUEST_INITRD: outputs.initrd, + }; +} + +/** + * The closure ROOTS whose transitive dependencies must be staged into the build + * context. + * + * The roots are the derivation out-paths, NOT the inner file paths: `nix + * path-info -r` takes store paths, and the kernel's root is its directory even + * though the image references the bzImage inside it. + * + * Staging the transitive closure — rather than just these five — is + * load-bearing. Each carried binary names an absolute /nix/store interpreter and + * resolves every NEEDED library through its own RPATH, so a context missing the + * transitive glibc produces an image whose binaries fail to exec with "missing + * dynamic library". That is the measured negative control behind the base-image + * choice, not a hypothetical. + */ +export function closureRoots(outputs: RunnerImageOutputs): string[] { + return [ + outputs.runner, + outputs.stack, + outputs.kernelDir, + outputs.rootfs, + outputs.initrd, + ]; +} + +/** The buildctl `--output` spec for each supported output mode. `oci` writes a + * browsable local layout (what R2's publish lane scans BEFORE deciding to push); + * `image` names a tagged image for a local dogfood load. */ +export function outputSpec( + mode: "oci" | "image", + tag: string, + ociDir: string, +): string { + return mode === "oci" + ? `type=oci,dest=${ociDir},tar=false` + : `type=image,name=${tag}`; +} + +/** + * The full `buildctl` argv (after the binary). Rootless BuildKit is the ruled + * mechanism for a prebuilt-application image; this lane never shells out to + * `docker build` and never mounts a host docker socket, which would hand the + * build the daemon's blast radius. + */ +export function buildctlArgs( + contextDir: string, + outputs: RunnerImageOutputs, + platform: string, + output: string, +): string[] { + const args = [ + "build", + "--frontend", + "dockerfile.v0", + "--local", + `context=${contextDir}`, + "--local", + `dockerfile=${contextDir}`, + "--opt", + "filename=Dockerfile", + "--opt", + `platform=${platform}`, + ]; + // Sorted, so the argv is deterministic across runs: an unstable arg order + // would make two otherwise-identical builds diff in logs for no reason. + for (const [name, value] of Object.entries(buildArgs(outputs)).sort( + ([a], [b]) => a.localeCompare(b), + )) { + args.push("--opt", `build-arg:${name}=${value}`); + } + args.push("--output", output); + return args; +} diff --git a/tools/runner-image/build.ts b/tools/runner-image/build.ts new file mode 100755 index 000000000..3bdd637b0 --- /dev/null +++ b/tools/runner-image/build.ts @@ -0,0 +1,246 @@ +#!/usr/bin/env bun +// Build the compass-runner container image (R1 of the Compass Runner +// containerization record): realise the nix closure, stage it into a build +// context, and hand that plain directory to a rootless BuildKit Dockerfile +// build. +// +// THE MECHANISM IS THE RULED ONE, and it is part of the reviewed surface rather +// than an implementation detail. The fleet's first-party image-builds spec +// splits mechanisms by what the image's RUNTIME is, not by what built the +// artifact: an image that IS a Nix environment earns nix2container; a PREBUILT +// APPLICATION on a minimal base is a Dockerfile built by rootless BuildKit. The +// Runner is the latter — it runs no toolchain, it exec's four binaries — so nix +// is the BUILDER here and never the runtime. The prescribed shape is `nix +// build` then COPY, which is exactly the two halves below. +// +// TypeScript, not bash: this has real logic — four nix builds whose out-paths +// are parsed and mapped to distinct build-args, a transitive closure staged +// path-by-path, and a fail-fast on every missing output — so the parsing and +// mapping core is pure and unit-tested (./build-core.test.ts) while this file is +// the thin I/O shell. +// +// Usage: +// bun tools/runner-image/build.ts [--tag ] [--output oci|image] + +import { spawnSync } from "node:child_process"; +import { + chmodSync, + cpSync, + existsSync, + mkdirSync, + readdirSync, + rmSync, + symlinkSync, +} from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { + buildctlArgs, + closureRoots, + outputSpec, + parseOutPaths, + type RunnerImageOutputs, +} from "./build-core.ts"; + +// This file is tools/runner-image/build.ts, so `../..` is the workspace root. +const workspaceRoot = join(dirname(fileURLToPath(import.meta.url)), "..", ".."); +const imageDir = join(workspaceRoot, "runner-image"); +const stageDir = join(imageDir, "store"); +const ociDir = join(imageDir, "out"); + +// Single-arch linux/amd64: the Runner's microVM backend needs KVM on the node, +// and the cluster nodes are amd64. A second leg would need its own KVM-capable +// builder, so it is added when a node arch is, not speculatively. +const IMAGE_PLATFORM = "linux/amd64"; + +function parseArgs(argv: readonly string[]): { + tag: string; + mode: "oci" | "image"; +} { + let tag = "compass-runner:dev"; + let mode: "oci" | "image" = "oci"; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + const value = argv[i + 1]; + if (arg === "--tag" && value !== undefined) { + tag = value; + i += 1; + } else if (arg === "--output" && value !== undefined) { + if (value !== "oci" && value !== "image") { + console.error(`--output must be 'oci' or 'image', got: ${value}`); + process.exit(2); + } + mode = value; + i += 1; + } else { + console.error(`unknown argument: ${arg}`); + process.exit(2); + } + } + return { tag, mode }; +} + +/** Run a nix build and return its realised out-paths in argument order. Exits on + * failure or on an out-path count that does not match what was asked for, so a + * drifted attr set is a named, fail-closed abort rather than a silently + * incomplete image. */ +function nixBuild( + args: readonly string[], + expected: number, + cwd: string, +): string[] { + const result = spawnSync( + "nix", + ["build", "--no-link", "--print-out-paths", ...args], + { + cwd, + encoding: "utf8", + stdio: ["ignore", "pipe", "inherit"], + }, + ); + if (result.status !== 0) { + console.error(`nix build ${args.join(" ")} failed (exit ${result.status})`); + process.exit(1); + } + const paths = parseOutPaths(result.stdout); + if (paths.length !== expected) { + console.error( + `nix build ${args.join(" ")} produced ${paths.length} out-paths, expected ${expected}`, + ); + process.exit(1); + } + return paths; +} + +/** Restore owner-write on a staged tree so it can be removed. Copying from the + * nix store preserves its read-only directory modes, which would otherwise make + * the stage dir undeletable on the next run. A no-op when the path is absent. */ +function makeWritable(dir: string): void { + if (!existsSync(dir)) return; + chmodSync(dir, 0o755); + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.isDirectory() && !entry.isSymbolicLink()) { + makeWritable(join(dir, entry.name)); + } + } +} + +const { tag, mode } = parseArgs(process.argv.slice(2)); + +// buildctl is a CLIENT; it needs a reachable buildkitd. Checked BEFORE the +// expensive nix builds so a missing daemon fails in a second rather than after a +// multi-minute realise. There is deliberately no `docker build` fallback. +const buildkitHost = process.env.BUILDKIT_HOST; +if (buildkitHost === undefined || buildkitHost === "") { + console.error( + "BUILDKIT_HOST is unset — start a buildkitd and point it here.\n" + + " This lane never falls back to `docker build`: rootless BuildKit is the\n" + + " ruled mechanism for a prebuilt-application image, not a preference.", + ); + process.exit(1); +} + +// --------------------------------------------------------------------------- +// 1. Realise every carried artifact. +// --------------------------------------------------------------------------- +// The Runner and the VMM env are flake outputs; the three guest assets come from +// guest-image/default.nix, which is a bare nix file and NOT a flake, so it takes +// the `-f` form (no `#attr` flake-ref exists for it) and runs from guest-image/. +console.error("runner-image: realising closure…"); +const [runner] = nixBuild([".#compass-runner"], 1, workspaceRoot) as [string]; +const [stack] = nixBuild([".#compass-stack-env"], 1, workspaceRoot) as [string]; +const guest = nixBuild( + [ + "-f", + "default.nix", + "compass-guest-kernel", + "compass-guest-rootfs", + "compass-guest-initrd", + ], + 3, + join(workspaceRoot, "guest-image"), +); +const [kernelDir, rootfs, initrd] = guest as [string, string, string]; +const outputs: RunnerImageOutputs = { + runner, + stack, + kernelDir, + rootfs, + initrd, +}; + +// --------------------------------------------------------------------------- +// 2. Stage the transitive closure. +// --------------------------------------------------------------------------- +// `nix path-info -r` over all five roots emits the DEDUPED union, so a path +// shared by several roots (glibc, most obviously) is staged once. The transitive +// set — not just the five roots — is what makes the image runnable: see +// closureRoots' note on the measured "missing dynamic library" control. +console.error("runner-image: staging closure…"); +// Nix store paths are read-only, and a recursive copy preserves that — so a +// previously staged tree cannot be removed until its directories are made +// writable again. Without this, the SECOND run of this lane fails EACCES on its +// own leftovers. +makeWritable(stageDir); +rmSync(stageDir, { recursive: true, force: true }); +rmSync(ociDir, { recursive: true, force: true }); +mkdirSync(stageDir, { recursive: true }); + +const closure = spawnSync( + "nix", + ["path-info", "-r", ...closureRoots(outputs)], + { + cwd: workspaceRoot, + encoding: "utf8", + stdio: ["ignore", "pipe", "inherit"], + }, +); +if (closure.status !== 0) { + console.error(`nix path-info -r failed (exit ${closure.status})`); + process.exit(1); +} +const closurePaths = parseOutPaths(closure.stdout); +for (const path of closurePaths) { + // Each basename is unique by construction (it carries a content hash), and + // the store's symlinks are preserved verbatim so the symlinkJoin'd stack env + // still resolves inside the image. + cpSync(path, join(stageDir, path.replace(/^\/nix\/store\//, "")), { + recursive: true, + verbatimSymlinks: true, + }); +} +console.error(`runner-image: staged ${closurePaths.length} store paths`); + +// The Dockerfile's ENTRYPOINT is exec-form and so cannot expand a build ARG, +// while the Runner's own store path carries a hash that moves on every Go +// rebuild. A stable relative symlink INSIDE the staged tree resolves both: the +// entrypoint is always /nix/store/.compass-runner, and what it points at moves +// with the build. +// +// It must live inside store/ and be RELATIVE. An absolute symlink at the +// context root dangles on the host (nothing is mounted at the target yet), and +// BuildKit checksums context entries before any COPY runs, so it fails the +// build with "not found" rather than deferring to runtime. Relative-and-inside +// keeps the link resolvable in both places. +rmSync(join(stageDir, ".compass-runner"), { force: true }); +symlinkSync( + join(runner.replace(/^\/nix\/store\//, ""), "bin", "compass-runner"), + join(stageDir, ".compass-runner"), +); + +// --------------------------------------------------------------------------- +// 3. Build the image from the staged directory. +// --------------------------------------------------------------------------- +mkdirSync(ociDir, { recursive: true }); +console.error(`runner-image: building ${tag}…`); +const build = spawnSync( + "buildctl", + buildctlArgs( + imageDir, + outputs, + IMAGE_PLATFORM, + outputSpec(mode, tag, ociDir), + ), + { cwd: workspaceRoot, stdio: "inherit" }, +); +process.exit(build.status ?? 1); diff --git a/tools/runner-image/moon.yml b/tools/runner-image/moon.yml new file mode 100644 index 000000000..a0d3e09c6 --- /dev/null +++ b/tools/runner-image/moon.yml @@ -0,0 +1,34 @@ +# yaml-language-server: $schema=https://moonrepo.dev/schemas/project.json +# +# runner-image (R1): the build lane for the compass-runner container image. The +# image is a Dockerfile on a minimal hardened base, built by rootless BuildKit — +# the ruled mechanism for a prebuilt application, as against the nix2container +# path the sibling agent-image/ takes because THAT container's job is to be a +# toolchain. The realise-and-stage logic lives here in TypeScript rather than a +# shell script (the no-bash-gate CI task); runner-image/ carries the Dockerfile +# and the staged context. +# +# The BUILD itself needs a reachable buildkitd and realises a multi-GB nix +# closure, so it is not a CI gate task (see runner-image/moon.yml for the build +# task and why it is runInCI:false). The typecheck/test tasks here ARE ordinary +# bun gates that ride CI: they cover the pure core — the out-path parsing, the +# build-arg mapping, and the closure-root selection whose drift would silently +# produce an unrunnable image. +# +# A bun/TypeScript CLI, hoisted root-workspace member (`bun` tag): install is +# inherited via .moon/tasks/tag-bun.yml and lint/format are whole-repo root +# tasks, so this leaf carries no own bun.lock. +layer: 'tool' +language: 'typescript' +tags: ['bun', 'ci-group.bun'] + +tasks: + typecheck: + command: 'bunx tsc --noEmit' + deps: ['install'] + inputs: ['*.ts', 'tsconfig.json', '/tsconfig.base.json', 'package.json', '/bun.lock'] + test: + # The pure core (build-core.ts) over fixtures: the out-path parsing, the + # artifact→build-arg mapping (incl. the kernel's bzImage suffix, which the + # file-shaped rootfs/initrd must NOT gain), and the closure-root selection. + inputs: ['*.ts', 'tsconfig.json', '/tsconfig.base.json', 'package.json', '/bun.lock'] diff --git a/tools/runner-image/package.json b/tools/runner-image/package.json new file mode 100644 index 000000000..33cc0cc07 --- /dev/null +++ b/tools/runner-image/package.json @@ -0,0 +1,14 @@ +{ + "name": "@compass/runner-image", + "private": true, + "type": "module", + "description": "Build the compass-runner container image (R1): realise the nix closure, stage it into a build context, and build a Dockerfile on a minimal hardened base with rootless BuildKit. Nix is the builder, never the runtime.", + "module": "build.ts", + "bin": { + "runner-image-build": "./build.ts" + }, + "devDependencies": { + "@types/bun": "catalog:", + "typescript": "catalog:" + } +} diff --git a/tools/runner-image/tsconfig.json b/tools/runner-image/tsconfig.json new file mode 100644 index 000000000..d40cc9e50 --- /dev/null +++ b/tools/runner-image/tsconfig.json @@ -0,0 +1,11 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2022"], + "moduleDetection": "force", + "allowJs": true, + "allowImportingTsExtensions": true, + "noUncheckedIndexedAccess": true, + "types": ["bun"] + } +} From a8f2f796022717a15a9b17dbc56f0d9411e35f31 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 16:12:36 -0400 Subject: [PATCH 7/8] fix(runner-image): make the image digest reproducible and pin the nonroot uid (RIG-3720) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review findings on the R1 lane. The load-bearing one: the image was NOT reproducible, contradicting R1's acceptance and a Dockerfile comment that claimed the COPY layer was "bit-identical to the staged tree". Two builds of identical inputs produced different manifest digests. Two independent causes, both measured rather than reasoned about: - cpSync defaults preserveTimestamps:false, so every staged file got a wall-clock mtime instead of the 1s nix normalises its store to. - BuildKit carries the context's mtimes into the layer tar regardless. Fixing only the first was NOT sufficient: the staged tree hashed bit-identical across runs while the `COPY store` layer digest still moved. The export needs rewrite-timestamp=true plus a fixed SOURCE_DATE_EPOCH. With both, two consecutive builds yield the same digest. The publish lane asserts exactly this, so the property is now real instead of asserted. Also from review: - Drop the RUNNER_BIN build-arg. It was declared and supplied but never consumed — the entrypoint became a staged symlink and the arg was left behind. The new drift test is what catches this class: it parses the Dockerfile's own ARG lines and asserts set-equality with what build-core supplies. Confirmed it fails when an ARG is renamed and passes when restored. - Add an explicit USER 65532:65532 rather than inheriting nonroot from an opaque base digest, so a future digest bump onto the root variant cannot silently produce a root-running container. - Assert the build host is x64. The platform is a manifest LABEL BuildKit applies without checking the staged files; today only flake.nix's systems list keeps them honest, which this lane never reads. - Narrow the build task's inputs from `**/*` to named tracked files: moon's native walker hashes contents directly and does not skip gitignored paths, so the glob would hash the multi-GB closure this task itself writes. - Cut the mechanism rationale from five restatements to one plus pointers, drop forward task references from source, and delete a tautological test that could not fail. --- .gitignore | 5 ++- .moon/workspace.yml | 21 +++++------- runner-image/Dockerfile | 37 ++++++++++---------- runner-image/moon.yml | 24 ++++++------- tools/runner-image/build-core.test.ts | 49 ++++++++++++++++++++------- tools/runner-image/build-core.ts | 23 ++++++++++--- tools/runner-image/build.ts | 30 +++++++++++++--- tools/runner-image/moon.yml | 11 +++--- 8 files changed, 125 insertions(+), 75 deletions(-) diff --git a/.gitignore b/.gitignore index 211db5d14..982c5a8e9 100644 --- a/.gitignore +++ b/.gitignore @@ -44,8 +44,7 @@ result-* # NOTE: generated clients (go/gen, packages/compass-client/src/gen) # are intentionally committed and CI drift-gated — do not ignore them. -# runner-image build outputs: the staged nix closure, the entrypoint symlink and -# the OCI layout are realised by tools/runner-image/build.ts, never committed. +# runner-image build outputs: the staged nix closure (which carries the +# entrypoint symlink) and the OCI layout, realised by tools/runner-image/build.ts. /runner-image/store/ /runner-image/out/ -/runner-image/compass-runner diff --git a/.moon/workspace.yml b/.moon/workspace.yml index 5b06d8578..773d9d18a 100644 --- a/.moon/workspace.yml +++ b/.moon/workspace.yml @@ -39,13 +39,10 @@ projects: # cache lane or V2a boot bring-up. Same affected-detection posture as # compass-agent-image (see guest-image/moon.yml). compass-guest-image: 'guest-image' - # The Runner container image (R1): the compass-runner binary, the KVM userland - # and the guest assets on a minimal hardened base. A Dockerfile project, not a - # nix one — the fleet image-builds spec routes a prebuilt application on a - # minimal base to rootless BuildKit, and nix2container only to images that ARE - # a Nix environment. Its `build` is runInCI:false (it needs a buildkitd and a - # multi-GB realise), so registration here is for graph membership and - # affected-detection rather than a gate leg; the gate coverage is the + # The Runner container image: the compass-runner binary, the KVM userland and + # the guest assets on a minimal hardened base. Its `build` is runInCI:false (it + # needs a buildkitd and a multi-GB realise), so registration here is for graph + # membership and affected-detection, not a gate leg; the gate coverage is the # runner-image tool's pure-core suite below. See runner-image/moon.yml. compass-runner-image: 'runner-image' # The local microVM boot-test lane (RIG-2591): realises the guest image + VMM @@ -54,12 +51,10 @@ projects: # unit tests ride the moon-driven CI sweep; the boot lane itself lives on # compass-go:test-microvm (runInCI:false — it needs KVM + a nix build). microvm-boot-test: 'tools/microvm-boot-test' - # The runner-image build lane (R1): realises the nix closure, stages it, and - # drives the rootless-BuildKit Dockerfile build. TypeScript rather than a - # shell script (the no-bash-gate CI task). Registered so its typecheck + the - # pure-core unit tests ride the moon-driven CI sweep — that suite is what - # catches a build-arg or closure-root drift, since the image build itself - # cannot run in CI until R2 stands up a buildkitd. + # The runner-image build lane: realises the nix closure, stages it, and drives + # the Dockerfile build. Registered so its typecheck + pure-core unit tests ride + # the CI sweep — that suite is what catches a build-arg or closure-root drift, + # since no compass CI step can build the image itself today. runner-image: 'tools/runner-image' # The Compass native-app release bundle: a heavy nix build (realises the # WebKitGTK cc/pkg-config closure) that stages the versioned tarball. Same diff --git a/runner-image/Dockerfile b/runner-image/Dockerfile index 337faf9a6..166872001 100644 --- a/runner-image/Dockerfile +++ b/runner-image/Dockerfile @@ -5,21 +5,10 @@ # backend direct-boots, so a Runner pod needs no host toolchain — only /dev/kvm # and the session-volume mount. # -# WHY A DOCKERFILE AND NOT nix2container. The fleet's first-party image-builds -# spec splits the two mechanisms by what the image's RUNTIME is, not by what -# built the artifact: an image that IS a Nix environment (a CI step image, a -# dev/agent shell) earns nix2container; a PREBUILT APPLICATION on a minimal base -# is a Dockerfile built by rootless BuildKit. It forecloses the tempting -# inference in terms — an image does not earn the nix path "merely because Nix -# built it or it ships a compiled binary" — and rejects nix2container for app -# images on cost. The Runner is the prebuilt-application row: it runs no -# toolchain, it exec's four binaries. The sibling agent-image/ sits on the other -# row BY NAME, because that container's job is to be a toolchain; sharing the -# `*-image/` directory shape is not sharing the mechanism. -# -# The prescribed shape for a nix-built artifact is therefore `nix build` it and -# COPY the result onto the base — which is exactly what build.sh stages into -# `store/` and what the single COPY below installs. +# A Dockerfile on a minimal base, not a nix2container image: the mechanism +# follows the image's RUNTIME, and the Runner is a prebuilt application that +# runs no toolchain. Full rationale in the design record's R1 task; nix still +# builds every carried artifact, and build.ts COPYs the result. # Distroless STATIC, not :base and not Alpine. The carried binaries are # dynamically linked but nix-closed: each names an ABSOLUTE /nix/store @@ -33,9 +22,10 @@ # omitted — the same exec fails "missing dynamic library", which is the control # proving the loader comes from the closure rather than the base. # -# :nonroot selects uid/gid 65532. The Runner does not need root: R4 grants -# /dev/kvm access by supplementary group, not by uid. Pinned by digest, never by -# tag, for the same reason R2 pins this image's own digest downstream. +# The pinned digest is the nonroot variant (uid/gid 65532). The Runner needs no +# root: /dev/kvm access is granted by supplementary group at the pod layer. +# Pinned by digest, never by tag — GHCR-style tag mutability gives no +# immutability guarantee. FROM gcr.io/distroless/static-debian12@sha256:c0f429e16b13e583da7e5a6ec20dd656d325d88e6819cafe0adb0828976529dc # The realised nix closure: the Runner binary, the KVM userland and the guest @@ -74,7 +64,6 @@ COPY store /nix/store # preflight is fail-closed (it requires both a static support probe and a real # boot canary), so a missing or wrong path fails at startup, loudly, rather than # at first session. -ARG RUNNER_BIN ARG VMM_BIN ARG VIRTIOFSD_BIN ARG GUEST_KERNEL @@ -91,6 +80,11 @@ ENV COMPASS_RUNTIME_BACKEND=microvm \ # passt is exec'd by name, not by configured path (unlike the VMM and virtiofsd, # which have --microvm-* flags), so its bin dir must be on PATH. Prepended to # the base PATH rather than replacing it. +# Consumed by the dockerfile frontend (paired with the export's +# rewrite-timestamp) to fix the config `created` field and the layer mtimes, so +# two builds of the same inputs yield the same digest. +ARG SOURCE_DATE_EPOCH + ARG STACK_BIN_DIR ENV PATH=${STACK_BIN_DIR}:/usr/local/bin:/usr/bin:/bin @@ -99,4 +93,9 @@ ENV PATH=${STACK_BIN_DIR}:/usr/local/bin:/usr/bin:/bin # form, so the Runner is pid 1 and receives SIGTERM directly at pod shutdown — # a shell wrapper would swallow it and leave the graceful drain to the kill # timeout. +# Explicit, rather than inherited from the base's own config: a future digest +# bump that landed on the root `static` variant would otherwise silently produce +# a root-running container, and nothing here would catch it. +USER 65532:65532 + ENTRYPOINT ["/nix/store/.compass-runner"] diff --git a/runner-image/moon.yml b/runner-image/moon.yml index 8f123844a..160368119 100644 --- a/runner-image/moon.yml +++ b/runner-image/moon.yml @@ -6,23 +6,17 @@ # This project carries the Dockerfile and the staged build context; the # realise-and-stage logic is tools/runner-image (TypeScript, not a shell script). # -# WHY THIS IS A DOCKERFILE PROJECT AND NOT A NIX ONE. The sibling image projects -# (agent-image/, guest-image/) are `language: nix` because their artifacts ARE -# nix derivations. This one is not: the fleet's first-party image-builds spec -# routes a prebuilt application on a minimal base to a Dockerfile built by -# rootless BuildKit, and routes nix2container only to images that ARE a Nix -# environment. Nix still BUILDS every carried artifact — the spec's prescribed -# shape is `nix build` then COPY — but the image's runtime is a distroless base, -# not a Nix environment, and the mechanism follows the runtime. +# A Dockerfile project, not a nix one like the sibling agent-image/ and +# guest-image/: their artifacts ARE nix derivations, this image's runtime is a +# distroless base. See the design record's R1 task. # # WHY THERE IS NO `ci` TASK. Unlike agent-image/ and guest-image/, this project # deliberately does NOT ride the pre-merge gate: the build needs a reachable -# buildkitd (which no compass CI step carries today — R2 stands that up) and it +# buildkitd (which no compass CI step carries today) and it # stages a multi-GB closure dominated by the guest rootfs. Registering a `ci` # task now would add an unrunnable task to the gate, so the gate coverage this # project has today is the pure-core suite in tools/runner-image, which is where -# a mapping drift would actually be caught. R2 adds the publish lane and with it -# the first CI leg that builds this image. +# a mapping drift would actually be caught. The publish lane adds the first CI leg that builds this image. layer: 'application' language: 'bash' tags: [] @@ -54,8 +48,14 @@ tasks: # runner command for the binary, and the guest-image/VMM-env closures for # the carried assets. A glob that misses one would leave a stale image # passing as current. + # Named tracked files, NOT a `**/*` glob. moon hashes with its native walker, + # which reads contents directly and does not skip gitignored paths — a bare + # glob here would hash the multi-GB staged closure this task itself writes + # into store/, on every affected-computation. inputs: - - '**/*' + - 'Dockerfile' + - '.dockerignore' + - 'moon.yml' - '/tools/runner-image/**/*' - '/flake.nix' - '/flake.lock' diff --git a/tools/runner-image/build-core.test.ts b/tools/runner-image/build-core.test.ts index d87da1905..32fe265d0 100644 --- a/tools/runner-image/build-core.test.ts +++ b/tools/runner-image/build-core.test.ts @@ -3,6 +3,8 @@ // plausible bug — not on a restatement of the implementation. import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; import { buildArgs, buildctlArgs, @@ -48,7 +50,6 @@ describe("kernelImagePath", () => { describe("buildArgs", () => { test("maps each artifact to the env the Runner reads, kernel suffixed and the file assets bare", () => { expect(buildArgs(outputs)).toEqual({ - RUNNER_BIN: "/nix/store/aaa-compass-runner/bin/compass-runner", VMM_BIN: "/nix/store/bbb-compass-stack-env/bin/cloud-hypervisor", VIRTIOFSD_BIN: "/nix/store/bbb-compass-stack-env/bin/virtiofsd", STACK_BIN_DIR: "/nix/store/bbb-compass-stack-env/bin", @@ -57,12 +58,6 @@ describe("buildArgs", () => { GUEST_INITRD: "/nix/store/eee-initrd", }); }); - - test("every value is absolute, since the image bakes them as env the Runner resolves without a cwd", () => { - for (const value of Object.values(buildArgs(outputs))) { - expect(value.startsWith("/nix/store/")).toBe(true); - } - }); }); describe("closureRoots", () => { @@ -91,13 +86,25 @@ describe("closureRoots", () => { describe("outputSpec", () => { test("oci writes a browsable layout, which is what the publish lane scans before pushing", () => { expect(outputSpec("oci", "ignored:tag", "/tmp/out")).toBe( - "type=oci,dest=/tmp/out,tar=false", + "type=oci,dest=/tmp/out,tar=false,rewrite-timestamp=true", ); }); test("image names the tag for a local load", () => { expect(outputSpec("image", "compass-runner:dev", "/tmp/out")).toBe( - "type=image,name=compass-runner:dev", + "type=image,name=compass-runner:dev,rewrite-timestamp=true", + ); + }); + + // The digest-stability property the publish lane depends on: without this, + // two builds of a bit-identical staged tree still export different layer + // digests, because the context's mtimes ride into the layer tar. + test("both modes rewrite layer timestamps, so a rebuild is digest-stable", () => { + expect(outputSpec("oci", "t", "/tmp/out")).toContain( + "rewrite-timestamp=true", + ); + expect(outputSpec("image", "t", "/tmp/out")).toContain( + "rewrite-timestamp=true", ); }); @@ -126,10 +133,26 @@ describe("buildctlArgs", () => { const once = buildctlArgs("/ctx", outputs, "linux/amd64", "type=oci"); const twice = buildctlArgs("/ctx", outputs, "linux/amd64", "type=oci"); expect(once).toEqual(twice); - const names = once - .filter((a) => a.startsWith("build-arg:")) - .map((a) => a.slice("build-arg:".length).split("=")[0]); - expect(names).toEqual([...names].sort()); + }); + + // The load-bearing drift test: buildArgs is only correct RELATIVE to the + // Dockerfile's own ARG declarations, and nothing else compares the two. It is + // what caught a RUNNER_BIN that build-core supplied and the Dockerfile had + // stopped consuming. + test("supplies exactly the build-args the Dockerfile declares", () => { + const dockerfile = readFileSync( + join(import.meta.dir, "..", "..", "runner-image", "Dockerfile"), + "utf8", + ); + const declared = new Set( + [...dockerfile.matchAll(/^ARG\s+([A-Z_][A-Z0-9_]*)/gm)].map( + (m) => m[1] as string, + ), + ); + // SOURCE_DATE_EPOCH is consumed by the frontend itself and supplied by + // buildctlArgs directly, not through the artifact mapping. + declared.delete("SOURCE_DATE_EPOCH"); + expect(new Set(Object.keys(buildArgs(outputs)))).toEqual(declared); }); test("uses the dockerfile.v0 frontend against the staged context", () => { diff --git a/tools/runner-image/build-core.ts b/tools/runner-image/build-core.ts index de211e6c8..67fd4f838 100644 --- a/tools/runner-image/build-core.ts +++ b/tools/runner-image/build-core.ts @@ -6,6 +6,11 @@ // container build needs: the set of paths to stage into the build context, and // the build-args that bake those absolute paths into the image's env. +/** The fixed build epoch. Nix normalises every store mtime to 1, so using the + * same value keeps the rewritten layer timestamps equal to what the staged tree + * already carries. */ +export const SOURCE_DATE_EPOCH = 1; + /** The six artifacts the image carries, as realised store paths. */ export interface RunnerImageOutputs { /** The `compass-runner` package out-path (the binary is at `bin/compass-runner`). */ @@ -54,7 +59,6 @@ export function kernelImagePath(kernelDir: string): string { */ export function buildArgs(outputs: RunnerImageOutputs): Record { return { - RUNNER_BIN: `${outputs.runner}/bin/compass-runner`, VMM_BIN: `${outputs.stack}/bin/cloud-hypervisor`, VIRTIOFSD_BIN: `${outputs.stack}/bin/virtiofsd`, // passt is exec'd by NAME rather than by a configured path (it has no @@ -92,16 +96,22 @@ export function closureRoots(outputs: RunnerImageOutputs): string[] { } /** The buildctl `--output` spec for each supported output mode. `oci` writes a - * browsable local layout (what R2's publish lane scans BEFORE deciding to push); + * browsable local layout (what the publish lane scans BEFORE deciding to push); * `image` names a tagged image for a local dogfood load. */ export function outputSpec( mode: "oci" | "image", tag: string, ociDir: string, ): string { + // rewrite-timestamp normalises every layer entry's mtime to SOURCE_DATE_EPOCH. + // Without it two builds of a BIT-IDENTICAL staged tree still produce different + // layer digests, because BuildKit carries the context's mtimes into the layer + // tar. Measured: only the `COPY store` layer differed between runs; with this + // set, two builds yield the same manifest digest. + const rewrite = "rewrite-timestamp=true"; return mode === "oci" - ? `type=oci,dest=${ociDir},tar=false` - : `type=image,name=${tag}`; + ? `type=oci,dest=${ociDir},tar=false,${rewrite}` + : `type=image,name=${tag},${rewrite}`; } /** @@ -128,6 +138,11 @@ export function buildctlArgs( "filename=Dockerfile", "--opt", `platform=${platform}`, + // Pairs with rewrite-timestamp: this fixes the image config's `created` + // field and is the epoch every layer mtime is rewritten to. 1, not 0, + // matching the mtime nix normalises its store paths to. + "--opt", + `build-arg:SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}`, ]; // Sorted, so the argv is deterministic across runs: an unstable arg order // would make two otherwise-identical builds diff in logs for no reason. diff --git a/tools/runner-image/build.ts b/tools/runner-image/build.ts index 3bdd637b0..658e5edd3 100755 --- a/tools/runner-image/build.ts +++ b/tools/runner-image/build.ts @@ -119,7 +119,9 @@ function makeWritable(dir: string): void { if (!existsSync(dir)) return; chmodSync(dir, 0o755); for (const entry of readdirSync(dir, { withFileTypes: true })) { - if (entry.isDirectory() && !entry.isSymbolicLink()) { + // readdirSync withFileTypes uses lstat semantics, so a symlink already + // reports isDirectory() === false and is never followed here. + if (entry.isDirectory()) { makeWritable(join(dir, entry.name)); } } @@ -127,9 +129,22 @@ function makeWritable(dir: string): void { const { tag, mode } = parseArgs(process.argv.slice(2)); -// buildctl is a CLIENT; it needs a reachable buildkitd. Checked BEFORE the -// expensive nix builds so a missing daemon fails in a second rather than after a -// multi-minute realise. There is deliberately no `docker build` fallback. +// The platform below is a manifest LABEL; BuildKit applies it without checking +// what the COPY'd files actually are. Today only flake.nix's systems list keeps +// the two honest — a separate file this lane never reads — so assert it here +// rather than inherit an unchecked invariant. +if (process.arch !== "x64") { + console.error( + `runner-image targets ${IMAGE_PLATFORM}, but this host is ${process.arch}.\n` + + " Building here would label the image amd64 while staging this host's binaries.", + ); + process.exit(1); +} + +// buildctl is a CLIENT; it needs a reachable buildkitd. This only checks the +// var is SET, so an unreachable daemon still fails at the build step after the +// realise — it catches the common "forgot to start one" case early, not a dead +// socket. There is deliberately no `docker build` fallback. const buildkitHost = process.env.BUILDKIT_HOST; if (buildkitHost === undefined || buildkitHost === "") { console.error( @@ -204,9 +219,16 @@ for (const path of closurePaths) { // Each basename is unique by construction (it carries a content hash), and // the store's symlinks are preserved verbatim so the symlinkJoin'd stack env // still resolves inside the image. + // + // preserveTimestamps is load-bearing for the DIGEST, not just tidiness. Nix + // normalises every store mtime to 1; without this, cpSync stamps wall-clock + // mtimes into the staged tree, they land in the layer tar, and two builds of + // identical inputs produce different digests. The publish lane asserts a + // stable digest, so this is part of what makes that assertion meaningful. cpSync(path, join(stageDir, path.replace(/^\/nix\/store\//, "")), { recursive: true, verbatimSymlinks: true, + preserveTimestamps: true, }); } console.error(`runner-image: staged ${closurePaths.length} store paths`); diff --git a/tools/runner-image/moon.yml b/tools/runner-image/moon.yml index a0d3e09c6..8e6826d14 100644 --- a/tools/runner-image/moon.yml +++ b/tools/runner-image/moon.yml @@ -1,12 +1,9 @@ # yaml-language-server: $schema=https://moonrepo.dev/schemas/project.json # -# runner-image (R1): the build lane for the compass-runner container image. The -# image is a Dockerfile on a minimal hardened base, built by rootless BuildKit — -# the ruled mechanism for a prebuilt application, as against the nix2container -# path the sibling agent-image/ takes because THAT container's job is to be a -# toolchain. The realise-and-stage logic lives here in TypeScript rather than a -# shell script (the no-bash-gate CI task); runner-image/ carries the Dockerfile -# and the staged context. +# runner-image (R1): the build lane for the compass-runner container image. +# Realises the nix closure, stages it, and drives the rootless-BuildKit build. +# TypeScript rather than a shell script (the no-bash-gate CI task); +# runner-image/ carries the Dockerfile and the staged context. # # The BUILD itself needs a reachable buildkitd and realises a multi-GB nix # closure, so it is not a CI gate task (see runner-image/moon.yml for the build From c724a9ee03a8f7ff284c633643d5c3a39c7ebe17 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 12 Sep 2026 16:33:57 -0400 Subject: [PATCH 8/8] fix(runner-image): give the image project a ci-group tag (RIG-3720) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI `setup` job failed at "Generate the concern matrix", blocking every downstream leg. tools/ci-matrix requires each moon project to carry exactly one `ci-group.*` tag and fails loud otherwise — the fail-loud posture that lets moon own discovery instead of a stale YAML enumeration — and the new compass-runner-image project shipped with `tags: []`. ci-group.nix, with the sibling image projects: the real build closure is the nix one those legs already provision. The project contributes no CI task today (`build` is runInCI:false), so the tag places it in the graph rather than adding a leg. `moon ci` does not run the matrix generator, so a green local sweep said nothing about this. Run `bun run tools/ci-matrix/index.ts` when adding a project. --- runner-image/moon.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/runner-image/moon.yml b/runner-image/moon.yml index 160368119..a96169df3 100644 --- a/runner-image/moon.yml +++ b/runner-image/moon.yml @@ -19,7 +19,12 @@ # a mapping drift would actually be caught. The publish lane adds the first CI leg that builds this image. layer: 'application' language: 'bash' -tags: [] +# ci-group.nix with the sibling image projects: the CI matrix requires every +# project to carry exactly one group tag, and this project's real build closure +# is the nix one those legs already provision. It contributes no CI task today +# (`build` is runInCI:false), so the tag places it in the graph rather than +# adding a leg. +tags: ['ci-group.nix'] workspace: inheritedTasks: