Standardize on colima for macOS; retire Docker Desktop as a supported engine

README.md

@@ -158,7 +158,7 @@ Missing files are empty layers; a key set in the user file wins over the system
 plans_repo: d3mlabs/plans              # org-wide plans repo (dev plan --org)
 knowledge_repo: d3mlabs/knowledge      # org learnings sync source
 deployment_formula: d3mlabs/d3mlabs/dev  # the formula `dev up` self-updates (the deployment names itself)
-container_engine: colima               # per-user container engine record ("docker" or "colima"; unset = bare docker)
+container_engine: docker               # per-user opt-out from the host's engine ("docker" = bare dockerd; unset = colima on macOS, bare dockerd elsewhere)
 ```
 
 Leaving a nilable key unset turns its feature off (`plans_repo` is only required by `dev plan --org`). Manage the user file with `dev config` instead of hand-editing YAML: `list` shows every known key with its resolved value and source layer (`env` / `user` / `system` / unset) — the settings debugging tool; `get <key>` prints the resolved value (exit 1 when unset); `set <key> <value>` writes the user file, creating it if missing. Known keys only; global, works without a `dev.yml`. The tool ships as two kinds of formula (the Debian core-package/config-package split, applied to a tap):
@@ -473,9 +473,17 @@ For repos that declare a `build.container`, dev builds and runs commands inside
 
 ### The container engine (per-user)
 
-*Which daemon serves a build* is a per-user provisioning decision, not a repo-shape detail: every docker invocation rides a resolved `Dev::ContainerEngine` (argv prefix + env + capabilities). Resolution is config-first, per invoking user: an explicit `DOCKER_HOST` in the environment wins; otherwise the user's `container_engine` settings record (`docker` or `colima`); otherwise bare docker. The human rides Docker Desktop; a no-GUI account (the agent user) records `colima` and gets `DOCKER_HOST` pointed at its **own** `~/.colima/default/docker.sock` — nothing crosses the sudo boundary, and both engines coexist on one machine. The engine's one capability flag, `local_mounts?`, names the single remote-poisoned assumption (bind-mounting local paths); both shipped engines answer true, and a future remote engine joins as config with its own sync strategy rather than an architecture fork.
+*Which daemon serves a build* is a per-user provisioning decision, not a repo-shape detail: every docker invocation rides a resolved `Dev::ContainerEngine` (argv prefix + env + capabilities). There is **one supported engine per host OS**, and dev owns its lifecycle end to end — that is what lets `dev up` leave a machine where `docker build` just works, for a human and for the no-GUI agent account alike:
 
-Provisioning is engine-shaped: `DockerDesktopProvisioner` is verify-only (`docker info` — dev never starts the GUI app), while `ColimaProvisioner` idempotently starts the user's VM (`colima start --vm-type vz --vz-rosetta`, so amd64 build images run on Apple silicon), sized from the repo's optional `build.container.resources` hint (`cpus`, `memory_gib`; defaults 4 / 8 GiB — colima applies sizing at VM creation).
+| Host | Engine | What `dev up` does |
+|---|---|---|
+| macOS | **colima** (per-user VM, `vz` + Rosetta so amd64 build images run on Apple silicon) | registers brew's `docker-buildx` with the brew `docker` CLI (`cliPluginsExtraDirs` in `~/.docker/config.json`, merged, never clobbered); starts the VM if it isn't running, sized from the repo's `build.container.resources` hint (`cpus`, `memory_gib`; defaults 4 / 8 GiB — colima applies sizing at VM creation) |
+| Linux | **bare dockerd** (the distro's docker packages, a system service) | nothing — there is no VM to own |
+| Windows | **bare dockerd inside the WSL2 distro** dev runs in | nothing — same as Linux; the distro's `docker-ce` + `docker-buildx-plugin`, enabled under systemd |
+
+Resolution is per invoking user: an explicit `DOCKER_HOST` in the environment wins and is left entirely alone (your engine, your problem); otherwise the `container_engine` settings record; otherwise the host OS's engine above. The only record worth writing is `docker` — the opt-out to bare docker with no env, reaching whatever daemon the CLI's own context does. That is where a Docker Desktop user lands: **unsupported but not blocked**. Two colima users on one Mac (a human and the agent account) each get `DOCKER_HOST` pointed at their **own** `~/.colima/default/docker.sock` — nothing crosses the sudo boundary. The engine's one capability flag, `local_mounts?`, names the single remote-poisoned assumption (bind-mounting local paths); every local engine answers true, and a future remote engine joins as config with its own sync strategy rather than an architecture fork.
+
+**Migrating a Mac off Docker Desktop.** Quit Docker Desktop (and stop it launching at login); `brew upgrade d3mlabs/d3mlabs/dev` brings `colima`, `docker` and `docker-buildx` in as formula dependencies; `dev up` in a containerized repo wires the CLI and starts the VM. Images are re-pulled/rebuilt once into the new engine's store, and a `persist: true` warm container is recreated on first use. Uninstall Desktop whenever you like — dev never touches it. An agent host ends up with two colima VMs (the human's and the agent's), each sized from the repo hint; stopping an idle one is engine-lifecycle work tracked in #187.
 
 ### Content-addressed image tag
 

lib/dev/agent_bootstrap.rb

@@ -191,8 +191,8 @@ def converge!
 
     # Step 6, invoked by the label contract only when a served repo declares
     # `build.container` (the only place local-vs-remote is expressed):
-    # converge the agent's own engine — colima installed (Docker Desktop
-    # cannot serve a no-GUI user), the agent's `container_engine: colima`
+    # converge the agent's own engine — colima installed (the macOS engine,
+    # per-user by nature), the agent's `container_engine: colima`
     # record written into its own config (resolution never crosses the sudo
     # boundary), and its VM provisioned, sized from the repo's resources
     # hint. Every colima invocation crosses to the agent via sudo.

lib/dev/build_container.rb

@@ -356,7 +356,7 @@ def ensure_image!(config, project_root:, push: true, publish: false,
     # build-context? BuildKit *streams* a build-context from the client on demand;
     # for a large, randomly-read dependency (e.g. a ~30GB engine read during
     # compilation) that transport stalls/deadlocks, especially under emulation. A
-    # plain `-v` volume (virtiofs on Docker Desktop) is the robust path the runtime
+    # plain `-v` volume (virtiofs on colima) is the robust path the runtime
     # already uses, so the prewarm reuses it.
     #
     # @param tag          [String] final content-addressed tag to commit

lib/dev/colima_provisioner.rb

@@ -5,10 +5,11 @@
 
 module Dev
   # Per-user provisioning for the colima engine: idempotently ensure the
-  # invoking user's own colima VM is running. This is what gives a no-GUI
-  # user (the agent account) a docker daemon of its own — Docker Desktop
-  # cannot serve it. The VM is vz-virtualized with Rosetta so amd64 build
-  # images (e.g. the linux cross-compile image) run on Apple silicon.
+  # invoking user's own colima VM is running. colima is the one macOS
+  # engine: it serves a human's `dev up` and a no-GUI agent account alike,
+  # and dev owns its whole lifecycle. The VM is vz-virtualized with Rosetta
+  # so amd64 build images (e.g. the linux cross-compile image) run on Apple
+  # silicon.
   #
   # Sizing comes from the repo's build.container.resources hint when given
   # (a UE compile wants more than the shipped defaults); colima applies
@@ -73,7 +74,7 @@ def provision!(cpus: nil, memory_gib: nil)
       ]
       return if T.unsafe(@executor).run(*args)
 
-      raise StartFailedError, "colima start failed — the agent's engine VM could not be brought up."
+      raise StartFailedError, "colima start failed — the container engine VM could not be brought up."
     end
 
     private

lib/dev/container_engine.rb

@@ -3,6 +3,7 @@
 
 require "open3"
 
+require "dev/deps"
 require "dev/settings"
 
 module Dev
@@ -11,13 +12,15 @@ module Dev
   # detail; *which daemon serves it* is a per-user provisioning decision, so
   # the engine is resolved from the invoking user's own config and injected
   # into everything that composes docker argv (BuildContainer, BuildWatcher,
-  # CacheGc). The human rides Docker Desktop; the agent user rides its own
-  # colima VM (Docker Desktop cannot serve a no-GUI user); a remote engine
-  # later joins as config, never an architecture fork.
+  # CacheGc). One engine per host OS: macOS users (human and agent alike)
+  # ride their own colima VM — the only macOS engine whose whole lifecycle
+  # dev can own (start, idle-stop, resize); Linux and WSL2 ride the bare
+  # dockerd already running on the host, where there is no VM to own. A
+  # remote engine later joins as config, never an architecture fork.
   #
   # Resolution order (see .resolve): explicit DOCKER_HOST in the caller's
-  # environment → the per-user `container_engine` settings record → the
-  # bare-docker default.
+  # environment → the per-user `container_engine` settings record → the host
+  # OS's default engine.
   #
   # The load-bearing capability predicate is #local_mounts?: bind-mounting
   # local paths (`-v project_root:/project`) is the single remote-poisoned
@@ -32,7 +35,8 @@ class UnknownEngineError < RuntimeError; end
     # The colima profile dev provisions and points at (colima's own default).
     COLIMA_PROFILE = "default"
 
-    # @return [Symbol] :docker_desktop, :colima, or :explicit (DOCKER_HOST)
+    # @return [Symbol] :colima (the user's own VM), :docker (bare dockerd —
+    #   whatever daemon the CLI's own context reaches), or :explicit (DOCKER_HOST)
     sig { returns(Symbol) }
     attr_reader :kind
 
@@ -50,25 +54,33 @@ class << self
 
       # Resolve the invoking user's engine: an explicit DOCKER_HOST wins (the
       # docker CLI reads it from the inherited environment, so the engine adds
-      # nothing) → the per-user settings record → the bare-docker default.
-      # Empty strings count as unset, matching Settings' layer semantics.
+      # nothing) → the per-user settings record → the host OS's default
+      # (colima on darwin, bare dockerd elsewhere). Empty strings count as
+      # unset, matching Settings' layer semantics. A `docker` record is the
+      # opt-out from the macOS default: bare docker with no env, which reaches
+      # whatever daemon the CLI's own context does — unsupported but not
+      # blocked (Docker Desktop users land here).
       #
       # @param settings [Dev::Settings] the invoking user's settings
       # @param env [Hash{String => String}] environment to consult (tests inject)
+      # @param host_os [String] "darwin" / "linux" / "windows" (tests inject)
       # @return [Dev::ContainerEngine]
       # @raise [UnknownEngineError] when the record names an unshipped engine
-      sig { params(settings: Dev::Settings, env: T::Hash[String, String]).returns(ContainerEngine) }
-      def resolve(settings: Dev::Settings.new, env: ENV.to_h)
+      sig do
+        params(settings: Dev::Settings, env: T::Hash[String, String], host_os: String).returns(ContainerEngine)
+      end
+      def resolve(settings: Dev::Settings.new, env: ENV.to_h, host_os: Dev::Deps.detect_host)
         docker_host = env["DOCKER_HOST"]
         return new(kind: :explicit) if docker_host && !docker_host.empty?
 
         record = settings.container_engine
         case record
-        when nil, "docker" then new(kind: :docker_desktop)
+        when nil then host_os == "darwin" ? colima : new(kind: :docker)
+        when "docker" then new(kind: :docker)
         when "colima" then colima
         else
           raise UnknownEngineError,
-            "unknown container_engine #{record.inspect} — dev ships \"docker\" and \"colima\"."
+            "unknown container_engine #{record.inspect} — dev ships \"colima\" and \"docker\"."
         end
       end
 
@@ -128,7 +140,7 @@ def capture(args, env: {})
     end
 
     # Whether local paths bind-mounted into containers reach this engine's
-    # daemon. True for every shipped engine (Docker Desktop, colima, explicit
+    # daemon. True for every shipped engine (colima, bare dockerd, explicit
     # local DOCKER_HOST); the future remote engine answers false and brings
     # its sync strategy with it — mount call sites guard on this rather than
     # assume it.

lib/dev/docker_cli_plugins.rb

@@ -0,0 +1,84 @@
+# typed: strict
+# frozen_string_literal: true
+
+require "fileutils"
+require "json"
+
+module Dev
+  # Points the Homebrew `docker` CLI at Homebrew's CLI plugins so `docker
+  # buildx` (and therefore `docker build` with --secret / --build-context)
+  # works without Docker Desktop.
+  #
+  # Docker Desktop ships the CLI plugins in ~/.docker/cli-plugins itself;
+  # brew's `docker-buildx` formula instead drops them under
+  # $(brew --prefix)/lib/docker/cli-plugins and documents a one-line
+  # `cliPluginsExtraDirs` entry in ~/.docker/config.json as the way to
+  # register them. This class owns that one line: it merges into whatever
+  # config.json already holds (auths, contexts, the user's own dirs) and
+  # never rewrites a file it cannot parse.
+  class DockerCliPlugins
+    extend T::Sig
+
+    # ~/.docker/config.json exists but is not JSON — dev will not clobber a
+    # file it cannot read back; the user fixes or removes it.
+    class UnreadableConfigError < RuntimeError; end
+
+    # Where brew's docker-* formulas link their CLI plugins, under the prefix.
+    PLUGIN_DIR = "lib/docker/cli-plugins"
+    KEY = "cliPluginsExtraDirs"
+
+    # @param config_path [String] the docker CLI config file (~/.docker/config.json)
+    # @param brew_prefix [String] the Homebrew prefix the plugins live under
+    sig { params(config_path: String, brew_prefix: String).void }
+    def initialize(config_path: File.join(Dir.home, ".docker", "config.json"), brew_prefix: self.class.default_brew_prefix)
+      @config_path = config_path
+      @brew_prefix = brew_prefix
+    end
+
+    # Ensure the brew plugin dir is listed in cliPluginsExtraDirs.
+    #
+    # @return [Symbol] :added when the entry was written, :already_present otherwise
+    # @raise [UnreadableConfigError] when an existing config.json is not JSON
+    sig { returns(Symbol) }
+    def ensure!
+      plugin_dir = File.join(@brew_prefix, PLUGIN_DIR)
+      config = read_config
+      dirs = Array(config[KEY])
+      return :already_present if dirs.include?(plugin_dir)
+
+      config[KEY] = dirs + [plugin_dir]
+      FileUtils.mkdir_p(File.dirname(@config_path))
+      File.write(@config_path, "#{JSON.pretty_generate(config)}\n")
+      :added
+    end
+
+    class << self
+      extend T::Sig
+
+      # The Homebrew prefix: the shellenv export when present, else the
+      # platform default (Apple silicon vs Intel).
+      #
+      # @return [String]
+      sig { returns(String) }
+      def default_brew_prefix
+        ENV.fetch("HOMEBREW_PREFIX") { RUBY_PLATFORM.include?("arm64") ? "/opt/homebrew" : "/usr/local" }
+      end
+    end
+
+    private
+
+    # @return [Hash] the parsed config, {} when the file does not exist
+    # @raise [UnreadableConfigError]
+    sig { returns(T::Hash[String, T.untyped]) }
+    def read_config
+      return {} unless File.exist?(@config_path)
+
+      parsed = JSON.parse(File.read(@config_path))
+      raise UnreadableConfigError, "#{@config_path} is not a JSON object" unless parsed.is_a?(Hash)
+
+      parsed
+    rescue JSON::ParserError => e
+      raise UnreadableConfigError, "#{@config_path} is not valid JSON (#{e.message.lines.first&.strip}); fix or remove it"
+    end
+  end
+end

lib/dev/engine_provisioner.rb

@@ -0,0 +1,66 @@
+# typed: strict
+# frozen_string_literal: true
+
+require "dev/build_container_config"
+require "dev/colima_provisioner"
+require "dev/container_engine"
+require "dev/deps"
+require "dev/docker_cli_plugins"
+require "dev/settings"
+
+module Dev
+  # `dev up`'s engine half: bring the resolved container engine to a state
+  # where `docker build` works for this project.
+  #
+  # What that takes depends on the host OS — one engine per OS, see
+  # ContainerEngine:
+  # - macOS: the brew docker CLI needs its buildx plugin registered, and the
+  #   colima VM needs to be running (started sized from the repo's
+  #   build.container.resources hint; colima sizes only at creation).
+  # - Linux / WSL2: bare dockerd is a system service, nothing to start; the
+  #   distro's docker packages ship buildx in place.
+  # - An explicit DOCKER_HOST is the user's own engine and is left alone.
+  class EngineProvisioner
+    extend T::Sig
+
+    # @param settings [Dev::Settings] carries the optional container_engine record
+    # @param colima [Dev::ColimaProvisioner] starts the macOS VM
+    # @param cli_plugins [Dev::DockerCliPlugins] wires brew's buildx into the CLI
+    # @param host_os [String] "darwin" / "linux" / "windows"
+    # @param env [Hash{String => String}] the process env (DOCKER_HOST wins)
+    sig do
+      params(
+        settings: Dev::Settings,
+        colima: Dev::ColimaProvisioner,
+        cli_plugins: Dev::DockerCliPlugins,
+        host_os: String,
+        env: T::Hash[String, String],
+      ).void
+    end
+    def initialize(settings: Dev::Settings.new, colima: Dev::ColimaProvisioner.new,
+      cli_plugins: Dev::DockerCliPlugins.new, host_os: Dev::Deps.detect_host, env: ENV.to_h)
+      @settings = settings
+      @colima = colima
+      @cli_plugins = cli_plugins
+      @host_os = host_os
+      @env = env
+    end
+
+    # Idempotently bring the engine up for a containerized project.
+    #
+    # @param resources [BuildContainerConfig::Resources, nil] the repo's VM sizing hint
+    # @return [void]
+    # @raise [ContainerEngine::UnknownEngineError] on an unrecognized container_engine record
+    # @raise [ColimaProvisioner::StartFailedError] when the VM will not start
+    sig { params(resources: T.nilable(BuildContainerConfig::Resources)).void }
+    def provision!(resources:)
+      engine = ContainerEngine.resolve(settings: @settings, env: @env, host_os: @host_os)
+      return if engine.kind == :explicit
+
+      @cli_plugins.ensure! if @host_os == "darwin"
+      return unless engine.kind == :colima
+
+      @colima.provision!(cpus: resources&.cpus, memory_gib: resources&.memory_gib)
+    end
+  end
+end

lib/dev/runner_status.rb

@@ -162,7 +162,7 @@ def report_agent_host(runner_dirs)
       line(File.directory?(@shared_root), "shared root present (#{@shared_root})")
       return unless @container_required
 
-      line(@executor.quiet?("brew", "list", "--formula", "colima"), "colima installed (agent engine)")
+      line(@executor.quiet?("brew", "list", "--formula", "colima"), "colima installed (container engine)")
     end
 
     # Group + setgid facts via stat, so inspection needs no privileges.