feat(harness): mount tool images read-only into the runtime

docs/architecture/configuration-and-compilation.md

@@ -4,7 +4,8 @@
 
 `Harness` describes how to run a class of agents. It selects exactly one runtime
 variant—kagent, Codex, Claude, or BYO—and contains workload image/command/args,
-literal environment values, WorkerPool configuration, and snapshot location.
+tool images, literal environment values, WorkerPool configuration, and snapshot
+location.
 
 `AgentTemplate` describes what the agent does. It contains model configuration,
 description and prompt, MCP tool bindings, skills, plugins, and Shared
@@ -85,6 +86,58 @@ spec:
 Command and argument changes participate in revision identity. They affect newly
 prepared revisions, not existing Sessions pinned to an older revision.
 
+### Tool images
+
+`spec.workload.toolImages` lists digest-pinned OCI images whose contents the
+controller mounts read-only next to the runtime image, for every runtime
+variant. Each entry is one Substrate image volume mounted at
+`/opt/kagent/tools/<name>`:
+
+```yaml
+spec:
+  claude: {}
+  workload:
+    image: registry.example.com/claude-harness@sha256:...
+    toolImages:
+      - name: go
+        oci: registry.example.com/tools/go@sha256:...
+      - name: kubectl
+        oci: registry.example.com/tools/kubectl@sha256:...
+```
+
+The controller sets `KAGENT_TOOL_PATHS` to the colon-joined
+`/opt/kagent/tools/<name>/bin` directories in list order. It never sets `PATH`,
+because a template `PATH` would replace the image's. Each managed runtime
+appends the listed directories to its own `PATH` at start, so the image's
+binaries always win: the Claude and Codex harnesses extend the environment of
+the CLI they spawn, the Go ADK extends its process environment before the
+`bash` tool and stdio MCP servers inherit it, and the Python ADK extends
+`os.environ` when `kagent-adk` starts (its `bash` tool still puts the venv
+first). `KAGENT_TOOL_PATHS` is reserved; a Harness `spec.env` entry of that
+name is rejected.
+
+An image may ship any number of binaries in its `bin/`. `PATH` resolves
+conflicts: a binary the runtime image already ships is never shadowed by a
+tool image, and when two tool images ship the same binary the earlier entry
+wins. Nothing reports a shadowed binary.
+
+Tool images follow the same rules as every other image in the API: the
+reference carries a sha256 digest and the controller resolves no tags. Keeping
+a digest current is the job of whatever writes the Harness. A change prepares a
+new revision; running Sessions keep theirs. Each entry costs one of Substrate's
+32 volume and 32 mount slots, and a non-empty list costs one of its 32
+environment entries. Substrate pulls tool images with the node's image
+credential provider, the same path as the runtime image.
+
+A tool image is a plain filesystem: it ships its files under one prefix with
+the executables in `bin/`, its entrypoint is never run, and the mount is
+read-only. Binaries must be static or self-contained for the runtime image
+they are mounted into: the built-in Claude, Codex, and Go ADK images are
+Alpine (musl), the Python ADK image is Debian (glibc).
+
+Tool images reach the runtime container only. A sandbox guest runs in its own
+tools image and does not see them.
+
 ## Prepared revision pipeline
 
 The controller compiles each Agent through one pipeline:
@@ -229,7 +282,9 @@ images, and compatible worker hardware.
   provide this integration. Other images must implement the same TaskStore
   contract; an upstream A2A server with only an in-memory store is insufficient.
   Optional model, prompt, tool, skill, and plugin configuration is
-  supplied in the ADK-shaped format when requested.
+  supplied in the ADK-shaped format when requested. Tool images are mounted
+  under `/opt/kagent/tools` and listed in `KAGENT_TOOL_PATHS`; a BYO image
+  extends its own `PATH` from that variable.
 
 The kagent and BYO compilers serialize the same ADK `AgentConfig` contract.
 `adkconfig.Builder.Build` assembles the complete payload and its model

docs/architecture/process-configuration.md

@@ -147,7 +147,8 @@ Python integrations load this document with `KAgentConfig.from_file()` or receiv
 an explicit `RuntimeConfig`. Managed processes still materialize the private
 `KAGENT_CONFIG_JSON` and `KAGENT_AGENT_CARD_JSON` payloads supplied by Substrate.
 Actor/session identity, credential projection, and private bootstrap paths keep
-their existing contracts. Substrate's limits remain 32 environment entries and
+their existing contracts. `KAGENT_TOOL_PATHS` is present only when the Harness
+lists tool images. Substrate's limits remain 32 environment entries and
 32,768 characters per value.
 
 ## OpenTelemetry

docs/architecture/sandboxes.md

@@ -182,7 +182,8 @@ separate and retains its own snapshot and execution-claim rules.
 The guest uses `agent-substrate/env/guest` for process and filesystem services.
 Kagent owns its main function, flags, logging, listener, readiness, and shutdown.
 The tools image supplies installed programs; an image volume supplies the managed
-guest binary and replaces the tools image's entrypoint.
+guest binary and replaces the tools image's entrypoint. Harness tool images
+(`spec.workload.toolImages`) use the same image-volume mechanism.
 
 The Actor mounts a durable directory at `/data`. The guest creates
 `/data/workspace` before readiness and stores process output in `/data/guest-logs`.

go/adk/cmd/main.go

@@ -22,6 +22,7 @@ import (
 	"github.com/kagent-dev/kagent/go/adk/pkg/session"
 	"github.com/kagent-dev/kagent/go/adk/pkg/telemetry"
 	"github.com/kagent-dev/kagent/go/pkg/logging"
+	"github.com/kagent-dev/kagent/go/pkg/toolpath"
 	"github.com/kagent-dev/kagent/go/pkg/tracing"
 )
 
@@ -45,6 +46,13 @@ func main() {
 }
 
 func run(logLevel, host, port, configDir string) error {
+	// The bash tool and stdio MCP servers inherit the process environment, so
+	// tool image binaries are exposed here, after the image's own PATH.
+	if path, ok := toolpath.ProcessPath(os.Environ()); ok {
+		if err := os.Setenv("PATH", path); err != nil {
+			return fmt.Errorf("expose tool image binaries on PATH: %w", err)
+		}
+	}
 	if err := config.MaterializeFromEnv(configDir); err != nil {
 		return fmt.Errorf("materialize agent config in %s: %w", configDir, err)
 	}

go/api/config/crd/bases/api.kagent.dev_harnesses.yaml

@@ -246,6 +246,40 @@ spec:
                       digest.
                     pattern: ^[^[:space:]@]+@sha256:[a-f0-9]{64}$
                     type: string
+                  toolImages:
+                    description: |-
+                      ToolImages lists digest-pinned OCI images the controller mounts read-only
+                      at /opt/kagent/tools/<name> in every runtime. KAGENT_TOOL_PATHS carries
+                      the colon-joined /opt/kagent/tools/<name>/bin directories in list order;
+                      each runtime appends them to PATH after the image's own binaries.
+                    items:
+                      description: HarnessToolImage is one read-only image volume
+                        added to the runtime container.
+                      properties:
+                        name:
+                          description: |-
+                            Name selects the mount directory /opt/kagent/tools/<name>. It must be a
+                            DNS label of at most 58 characters: the controller prefixes it with
+                            "tool-" to form the volume name, which is itself limited to 63.
+                          maxLength: 58
+                          minLength: 1
+                          pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
+                          type: string
+                        oci:
+                          description: OCI is the image reference pinned by sha256
+                            digest. Tags are not resolved.
+                          maxLength: 512
+                          pattern: ^[^[:space:]@]+@sha256:[a-f0-9]{64}$
+                          type: string
+                      required:
+                      - name
+                      - oci
+                      type: object
+                    maxItems: 16
+                    type: array
+                    x-kubernetes-list-map-keys:
+                    - name
+                    x-kubernetes-list-type: map
                 required:
                 - image
                 type: object

go/api/v1alpha3/configuration_crd_cel_test.go

@@ -95,6 +95,43 @@ func TestConfigurationCRDValidation(t *testing.T) {
 			}),
 			wantReject: "spec.workload.image",
 		},
+		{
+			name: "Harness rejects tag-only tool image",
+			object: validHarness(namespace, "harness-tagged-tool-image", HarnessSpec{
+				Kagent:   &KagentHarness{},
+				Workload: HarnessWorkload{ToolImages: []HarnessToolImage{{Name: "kubectl", OCI: "registry.example.com/kubectl:latest"}}},
+			}),
+			wantReject: "spec.workload.toolImages[0].oci",
+		},
+		{
+			name: "Harness rejects tool image name that is not a DNS label",
+			object: validHarness(namespace, "harness-tool-image-name", HarnessSpec{
+				Kagent:   &KagentHarness{},
+				Workload: HarnessWorkload{ToolImages: []HarnessToolImage{{Name: "Kube_ctl", OCI: "registry.example.com/kubectl@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}}},
+			}),
+			wantReject: "spec.workload.toolImages[0].name",
+		},
+		{
+			name: "Harness rejects duplicate tool image names",
+			object: validHarness(namespace, "harness-duplicate-tool-image", HarnessSpec{
+				Kagent: &KagentHarness{},
+				Workload: HarnessWorkload{ToolImages: []HarnessToolImage{
+					{Name: "kubectl", OCI: "registry.example.com/kubectl@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"},
+					{Name: "kubectl", OCI: "registry.example.com/kubectl@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"},
+				}},
+			}),
+			wantReject: "Duplicate value",
+		},
+		{
+			name: "Harness accepts digest-pinned tool images",
+			object: validHarness(namespace, "harness-tool-images", HarnessSpec{
+				Kagent: &KagentHarness{},
+				Workload: HarnessWorkload{ToolImages: []HarnessToolImage{
+					{Name: "kubectl", OCI: "registry.example.com/kubectl@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"},
+					{Name: "helm", OCI: "registry.example.com/helm@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"},
+				}},
+			}),
+		},
 		{
 			name: "Harness memory requires a model reference",
 			object: validHarness(namespace, "harness-empty-memory-model", HarnessSpec{

go/api/v1alpha3/harness_types.go

@@ -126,6 +126,34 @@ type HarnessWorkload struct {
 	// +kubebuilder:validation:MaxItems=64
 	// +optional
 	Args []string `json:"args,omitempty"`
+
+	// ToolImages lists digest-pinned OCI images the controller mounts read-only
+	// at /opt/kagent/tools/<name> in every runtime. KAGENT_TOOL_PATHS carries
+	// the colon-joined /opt/kagent/tools/<name>/bin directories in list order;
+	// each runtime appends them to PATH after the image's own binaries.
+	// +kubebuilder:validation:MaxItems=16
+	// +listType=map
+	// +listMapKey=name
+	// +optional
+	ToolImages []HarnessToolImage `json:"toolImages,omitempty"`
+}
+
+// HarnessToolImage is one read-only image volume added to the runtime container.
+type HarnessToolImage struct {
+	// Name selects the mount directory /opt/kagent/tools/<name>. It must be a
+	// DNS label of at most 58 characters: the controller prefixes it with
+	// "tool-" to form the volume name, which is itself limited to 63.
+	// +kubebuilder:validation:MinLength=1
+	// +kubebuilder:validation:MaxLength=58
+	// +kubebuilder:validation:Pattern=`^[a-z0-9]([-a-z0-9]*[a-z0-9])?$`
+	// +required
+	Name string `json:"name"`
+
+	// OCI is the image reference pinned by sha256 digest. Tags are not resolved.
+	// +kubebuilder:validation:MaxLength=512
+	// +kubebuilder:validation:Pattern=`^[^[:space:]@]+@sha256:[a-f0-9]{64}$`
+	// +required
+	OCI string `json:"oci"`
 }
 
 // HarnessSpec defines a reusable runtime and its infrastructure policy.

go/core/internal/substrate/actor_template.go

@@ -13,10 +13,12 @@ import (
 	"google.golang.org/protobuf/proto"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apimachinery/pkg/util/validation"
 
 	apia2a "github.com/kagent-dev/kagent/go/api/a2a"
 	"github.com/kagent-dev/kagent/go/core/internal/translator"
 	"github.com/kagent-dev/kagent/go/core/pkg/env"
+	"github.com/kagent-dev/kagent/go/pkg/toolpath"
 	"github.com/kagent-dev/kagent/go/pkg/tracing"
 )
 
@@ -36,6 +38,10 @@ const egressTrustMount = "/run/kagent/egress"
 const actorIdentityVolume = "actor-identity"
 const actorIdentityMount = "/run/kagent/identity"
 
+// Tool image volumes carry a prefix so a tool named like a built-in volume
+// cannot collide with it.
+const toolImageVolumePrefix = "tool-"
+
 var egressTrustEnvironment = map[string]struct{}{
 	"SSL_CERT_FILE": {}, "SSL_CERT_DIR": {}, "REQUESTS_CA_BUNDLE": {}, "AWS_CA_BUNDLE": {},
 	"NODE_EXTRA_CA_CERTS": {}, "CURL_CA_BUNDLE": {}, "GIT_SSL_CAINFO": {},
@@ -68,11 +74,21 @@ func ActorTemplateForRevision(spec *translator.Revision, revisionID translator.R
 		if _, owned := egressTrustEnvironment[variable.Name]; owned {
 			return nil, fmt.Errorf("runtime environment %q conflicts with gateway trust", variable.Name)
 		}
+		if variable.Name == toolpath.EnvName {
+			return nil, fmt.Errorf("runtime environment %q is reserved for tool images", variable.Name)
+		}
 	}
 	for _, name := range []string{"SSL_CERT_FILE", "REQUESTS_CA_BUNDLE", "AWS_CA_BUNDLE", "NODE_EXTRA_CA_CERTS", "CURL_CA_BUNDLE", "GIT_SSL_CAINFO"} {
 		environment = append(environment, corev1.EnvVar{Name: name, Value: egressTrustMount + "/trust-bundle.pem"})
 	}
 	environment = append(environment, corev1.EnvVar{Name: "SSL_CERT_DIR", Value: egressTrustMount})
+	toolVolumes, toolMounts, toolNames, err := toolImageVolumes(spec.ToolImages)
+	if err != nil {
+		return nil, err
+	}
+	if len(toolNames) > 0 {
+		environment = append(environment, corev1.EnvVar{Name: toolpath.EnvName, Value: toolpath.JoinBinDirs(toolNames)})
+	}
 	environment = append(environment,
 		corev1.EnvVar{Name: "KAGENT_CONFIG_JSON", Value: string(spec.ConfigJSON)},
 		corev1.EnvVar{Name: "KAGENT_AGENT_CARD_JSON", Value: string(cardJSON)},
@@ -102,11 +118,11 @@ func ActorTemplateForRevision(spec *translator.Revision, revisionID translator.R
 				Path: "/readyz",
 				Port: 8081,
 			}, TimeoutSeconds: 30},
-			VolumeMounts: []*ateapipb.VolumeMount{
+			VolumeMounts: append([]*ateapipb.VolumeMount{
 				{Name: durableDataVolume, MountPath: durableDataMount},
 				{Name: egressTrustVolume, MountPath: egressTrustMount},
 				{Name: actorIdentityVolume, MountPath: actorIdentityMount},
-			},
+			}, toolMounts...),
 		}},
 		WorkerSelector: workerSelectorForPool(workerKey),
 		SnapshotConfig: &ateapipb.SnapshotConfig{
@@ -115,7 +131,7 @@ func ActorTemplateForRevision(spec *translator.Revision, revisionID translator.R
 			// for both until lifecycle v2 separates them again.
 			OnCommit: ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL,
 		},
-		Volumes: []*ateapipb.Volume{
+		Volumes: append([]*ateapipb.Volume{
 			{Name: durableDataVolume, DurableDir: &ateapipb.DurableDirVolumeSource{}},
 			// Substrate regenerates this projection on Run and Restore. A fork
 			// therefore routes storage calls as its own actor, never its source.
@@ -129,11 +145,38 @@ func ActorTemplateForRevision(spec *translator.Revision, revisionID translator.R
 			{Name: egressTrustVolume, SystemInfo: &ateapipb.SystemInfoVolumeSource{DataSources: []*ateapipb.SystemInfoDataSource{
 				{TrustBundle: &ateapipb.TrustBundleDataSource{Names: []string{"egress-mitm.ate.dev"}, Path: "trust-bundle.pem"}},
 			}}},
-		},
+		}, toolVolumes...),
 	}
 	return template, nil
 }
 
+// toolImageVolumes projects each tool image as a read-only image volume mounted
+// under toolpath.MountRoot. The CRD validates the entries; the check here keeps
+// a hand-built revision on the same contract Substrate enforces.
+func toolImageVolumes(images []translator.ToolImage) ([]*ateapipb.Volume, []*ateapipb.VolumeMount, []string, error) {
+	var volumes []*ateapipb.Volume
+	var mounts []*ateapipb.VolumeMount
+	var names []string
+	seen := make(map[string]struct{}, len(images))
+	for _, image := range images {
+		if errs := validation.IsDNS1123Label(image.Name); len(errs) > 0 {
+			return nil, nil, nil, fmt.Errorf("tool image name %q: %s", image.Name, strings.Join(errs, "; "))
+		}
+		if _, duplicate := seen[image.Name]; duplicate {
+			return nil, nil, nil, fmt.Errorf("tool image name %q is listed twice", image.Name)
+		}
+		seen[image.Name] = struct{}{}
+		if !pinnedSandboxImage.MatchString(image.Reference) {
+			return nil, nil, nil, fmt.Errorf("tool image %q must be pinned by sha256 digest", image.Name)
+		}
+		volumeName := toolImageVolumePrefix + image.Name
+		volumes = append(volumes, &ateapipb.Volume{Name: volumeName, Image: &ateapipb.ImageVolumeSource{Reference: image.Reference}})
+		mounts = append(mounts, &ateapipb.VolumeMount{Name: volumeName, MountPath: toolpath.MountPath(image.Name)})
+		names = append(names, image.Name)
+	}
+	return volumes, mounts, names, nil
+}
+
 // ActorTemplateSpecEqual compares the client-owned immutable fields of two
 // templates, excluding server-owned metadata and golden-snapshot status.
 func ActorTemplateSpecEqual(left, right *ateapipb.ActorTemplate) bool {