Skip to content

feat(ci): add a local Linux validation task so non-Linux contributors can exercise Linux-gated Rust code #3039

Description

@letv1nnn

User Story

As a contributor developing OpenShell on macOS or Windows, I want a mise task that compiles, lints, and tests the workspace for a Linux target locally, so that I catch failures in #[cfg(target_os = "linux")] code before pushing — instead of discovering them only in the branch checks or the merge queue.

Problem Statement

mise run ci / mise run test run against the host target. The workspace contains 382 target_os = "linux" gated lines across 31 files and 10 crates (heaviest: openshell-supervisor-process, openshell-supervisor-network, openshell-driver-vm, openshell-sandbox). On a macOS or Windows host, cargo's clippy/check/test never see those blocks. Windows contributors do run Rust checks locally (the rust:lint / test:rust tasks have run_windows -> windows-msvc.ps1 for the native MSVC target), but that target is windows, so the Linux-gated code is still never compiled. There is no host-runnable path that compiles the Linux-gated code on a non-Linux machine.

CI does exercise it, but through a path contributors cannot run locally. In .github/workflows/branch-checks.yml the Rust lane runs inside a Nix devShell on a matrix of x86_64-linux | aarch64-linux | aarch64-darwin:

shell: nix develop .#devShells.${{ matrix.system }}.default -c bash ...
# Format: cargo fmt --all -- --check
# Lint:   cargo clippy --workspace --all-targets -- -D warnings
# Test:   cargo nextest run --profile ci --workspace --features openshell-server/test-support

The aarch64-darwin lane mirrors a non-Linux host and passes; the Linux-gated code compiles only on the two Linux runners. Locally, nix develop .#devShells.x86_64-linux... cannot realize a Linux devShell on a non-Linux host without a Linux builder. A Linux CI image exists (deploy/docker/Dockerfile.ci -> ghcr.io/nvidia/openshell/ci, built by ci-image.yml, used by the go/python/license/markdown/cargo-deny branch-checks jobs), but no task runs the Rust checks — or any checks — inside a Linux container. This is a concrete slice of the local<->CI parity gap tracked in #2204.

Impact / Why This Matters

Without a local Linux path, non-Linux contributors get a green local run, then a red Rust branch-check or merge-queue failure — feedback latency jumps from seconds to a full CI round-trip, and merge-queue failures block other PRs. Today the only workarounds are (a) push and wait for CI, or (b) hand-craft a docker run ... cargo ... command that bypasses the repo's docker/podman engine abstraction and mishandles file ownership and build caches. Neither is reproducible or discoverable, so Linux-gated regressions keep reaching CI. This matters most for the four Linux-heavy supervisor/driver/sandbox crates, where the gated surface is largest.

Proposed Design

New mise tasks, discoverable in mise tasks:

mise run ci:linux     # the ci checks, run for a Linux target in a Linux environment
mise run test:linux   # the test checks, run for a Linux target in a Linux environment

Each realizes a Linux environment, runs the existing ci / test task graph against the host working tree, and exits with the in-environment status. On a Linux host the tasks still work.

Two candidate mechanisms (workflow-level choice for maintainers; see Alternatives):

  • (A) Container-based (recommended default). Run the existing task graph inside a container built from the same Dockerfile.ci CI uses. Reproduces target_os = "linux" compile/clippy/test faithfully (identical pinned Rust 1.95.0). It approximates the CI Rust lane rather than being bit-identical — CI's Rust lane uses the Nix devShell, and the mise test:rust task uses cargo test where the CI lane uses cargo nextest run --profile ci — but it catches the class of failure this issue targets. Portable across macOS/Windows/Linux hosts.
  • (B) Nix-based. Reproduce the CI lane exactly via nix develop .#devShells.x86_64-linux.default. Bit-faithful, but on a non-Linux host requires a Linux Nix builder (remote builder, or a Linux container running the Nix daemon) — heavier prerequisites.

Observable behavior (either mechanism):

  • Runs the repo's standard Linux checks so a local pass predicts the CI Rust lane. mise run ci's rust:format:check and rust:lint already match the CI cargo fmt/cargo clippy invocations verbatim; the test step reuses test:rust (implementers may instead invoke cargo nextest --profile ci to match the CI lane exactly).
  • For the container mechanism, works with docker and podman alike (repo convention).
  • Reuses build caches across runs (cargo registry, target/, sccache); repeat runs are incremental.
  • Host working tree is the source of truth (bind-mounted); artifacts written back do not break subsequent host cargo/git use.
  • Image/toolchain source is explicit, version-pinned, and consistent with mise.toml / Dockerfile.ci.

Essential constraints (existing conventions):

  • Thin task in tasks/ci.toml; logic in a new tasks/scripts/*.sh, mirroring docker-build-ci.sh.
  • For (A): source tasks/scripts/container-engine.sh and use ce run — not bare docker run — so docker and podman both work; keep run flags portable (ce_build normalizes flags but there is no ce_run equivalent).
  • SPDX headers on all new files (enforced by license:check).
  • Handle file ownership for both rootful docker (UID mapping) and rootless podman (userns).
  • Mount cache volumes; account for the global RUSTC_WRAPPER=sccache / SCCACHE_DIR env from mise.toml.
  • ghcr.io/nvidia/openshell/ci is not anonymously pullable (CI authenticates with GITHUB_TOKEN); the task must authenticate to ghcr or fall back to a local build:docker:ci, and document which.
  • Scope: lint + compile + test. Excludes e2e (needs docker-in-docker / a live gateway).

Acceptance Criteria

  • mise run ci:linux runs the ci task graph (format check, clippy -D warnings, tests) for a Linux target in a Linux environment and propagates the exit code.
  • mise run test:linux runs the test task graph for a Linux target in a Linux environment.
  • A deliberately introduced compile error inside a #[cfg(target_os = "linux")] block is caught by mise run ci:linux on a macOS host.
  • For the container mechanism: works with docker and with podman via ce (no bare docker/podman calls).
  • Repeat runs reuse caches (second run substantially faster; no cold rebuild).
  • Artifacts written to the host tree are not left root-owned in a way that breaks subsequent host cargo/git operations.
  • Image/toolchain source is pinned and documented; ghcr-auth vs. local-build behavior is explicit; version-sync with mise.toml/Dockerfile.ci is maintained.
  • New files carry SPDX headers; mise run lint passes.
  • CONTRIBUTING testing docs and the AGENTS.md "Testing" list are updated; CI-referencing skills reviewed per the Skill Maintenance rule / sync-agent-infra.
  • Related issue Use a unified mechanism for build, test, packaging, and release #2204 linked.

Alternatives Considered

  1. Nix Linux devShell as the only path (design B). Exact parity but needs a Linux Nix builder on non-Linux hosts — a real prerequisite barrier. Kept as the "exact parity" alternative, not the default.
  2. Cross-compile with cargo-zigbuild (already a dep). Covers compile/clippy but not test execution, and Linux-only runtime behavior (namespaces, Landlock, process supervision) still can't run. Partial.
  3. Document a raw docker run snippet in CONTRIBUTING. No engine abstraction, no cache/ownership handling, drifts, undiscoverable. Rejected.
  4. Rely on remote CI feedback only. Status quo; the round-trip latency and merge-queue blocking are the reported problem.
  5. Fold into Use a unified mechanism for build, test, packaging, and release #2204's unified build mechanism. Use a unified mechanism for build, test, packaging, and release #2204 is a broad roadmap item; this is a small, independently shippable step. Link as parent rather than block on it.

Agent Investigation

Investigated from a source read of main (working tree clean at the start of this session).

  • Counted the Linux-gated surface: grep -rn 'target_os = "linux"' crates/ -> 382 lines, 31 files, 10 crates; heaviest in openshell-supervisor-process, openshell-supervisor-network, openshell-driver-vm, openshell-sandbox.
  • Traced task graph: mise run ci -> lint (rust:format:check, rust:lint), check, test (test:rust = cargo test --workspace --exclude openshell-server + cargo test -p openshell-server --features test-support), go:ci, rust:deny:policy. All host-target.
  • Confirmed CI path: branch-checks.yml Rust lane runs in a Nix devShell (x86_64-linux, aarch64-linux, aarch64-darwin) with cargo fmt/cargo clippy/cargo nextest --profile ci; go/python/license/markdown/cargo-deny run in ghcr.io/nvidia/openshell/ci:latest. build:docker:ci only builds that image; no task runs checks inside a container (grep 'docker run' tasks/ -> none).
  • Verified conventions: tasks/scripts/container-engine.sh provides the required docker/podman ce/ce_build abstraction (bare docker run would violate it); SPDX headers enforced via license:check; ghcr.io/nvidia/openshell/ci is not anonymously pullable.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions