Repository navigation
Add Loop Runtime on GKE with isolated Sandbox Service workers - #98
Alex R (alexr17) wants to merge 20 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c3b7cd9e53
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e8a133ecc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
Context The Helm chart had no Loop Runtime workload, so external EKS deployments could not run the Kubernetes-hosted runtime. Description Add an opt-in AWS EKS Loop Runtime Deployment, Service, ServiceAccount, and ConfigMap. Wire the runtime to the existing data-plane Secret, API URL, AWS MicroVM sandbox settings, and optional Istio route. Squashed from #98 and rebased onto main.
Run Loop Runtime on cloud: google as well as cloud: aws. Storage URIs follow the Brainstore ConfigMaps (s3:// or gs://), and the ServiceAccount and pod use IRSA or GKE Workload Identity. Add loopRuntime.sandbox.backend. awsLambdaMicrovm keeps the AWS Lambda MicroVM settings under sandbox.awsLambdaMicrovm. sandboxService points the runtime at Braintrust Sandbox Service workers over mTLS, which is the GKE default and also works on AWS. An optional sandbox egress gateway opens the runtime's forward proxy behind its own Service. Treat a missing loopRuntime map as disabled so helm upgrade --reuse-values from earlier chart releases keeps rendering. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1bc6147 to
b97b95f
Compare
Drop the AWS Lambda MicroVM backend. Loop Runtime on EKS and GKE now sends every sandbox to Braintrust Sandbox Service workers. With a Sandbox Service endpoint and no EXO_SANDBOX_PROVIDER, the runtime uses the Sandbox Service for all sandboxes instead of the MicroVM default plus a LaunchDarkly rollout, as the CloudFormation mixed mode does. Flatten loopRuntime.sandbox.sandboxService.* into loopRuntime.sandbox.* and expose the sandbox egress mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop checks the runtime already enforces (urls type, egress mode, egress port collision, TLS key names), inline the one-use egress Service name helper, and shorten the values comments and README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1e9322fa75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e459430706
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 922cc8a6ba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d40b5ed21b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Erik Weathers (erikdw)
left a comment
There was a problem hiding this comment.
Yes, this looks like the right direction. Loop Runtime in the Braintrust chart, the shared Sandbox Service chart as a separate release, and infrastructure outside Helm are sensible ownership boundaries.
Three integration fixes before it's ready (inline comments below):
- Shutdown time: the Deployment gets the default 30s grace period, but Loop takes up to 90s to drain (CFN uses
StopTimeout: 120). Rollouts can cut off turns still draining after 30 seconds. FUNCTION_SECRET_KEY: it's only mapped intoSERVICE_TOKEN_SECRET_KEY. The runtime also needsFUNCTION_SECRET_KEYitself, or MCP servers that use stored credentials break.- Empty
orgName: the chart allows it for multi-org installs, but with noallowedOrgIdsthe runtime refuses to start.
Resource/version contract:
- Give the runtime's disk cache an explicit budget:
BRAINSTORE_OBJECT_STORE_CACHE_FILE_SIZE, anemptyDirsizeLimit, and an ephemeral-storage request so scheduling accounts for it on the shared system nodes. - Pin a tested runtime/worker/sandbox image combination. The chart's runtime default (
latest-2.x) and the sandbox guest image currently float.
Then a real EKS lifecycle test: create a sandbox, use Git through the proxy, run a Loop turn, replace a runtime pod, and upgrade/drain/uninstall workers. That's the main evidence still missing. The GKE instructions also depend on unmerged Terraform work in terraform-google-braintrust-data-plane (draft braintrustdata/terraform-google-braintrust-data-plane#28, stacked on jeff/gke-standard), so make that dependency explicit.
All of this is integration and rollout work within the proposed design. The overall split makes sense.
…rkers Loop Runtime on AWS stays on ECS through the Terraform module, so the chart now rejects every cloud except google. Workers run in the isolated GKE cluster from the GCP Terraform module; the runtime reaches them through the discovery endpoint and exposes its Git forward proxy on an internal load balancer bound to an address Terraform reserves. - Drop AWS (IRSA, s3:// URIs), sandbox.urls, and the EKS example - Always mount the client TLS Secret and run the egress gateway - Match the ECS Sandbox Service environment and pinned images - Default runtime pods to 1 vCPU / 2 GiB with two replicas - Add a GKE example with a TLS Secret script and worker chart values Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker chart expects node-local storage at /var/lib/braintrust-sandboxes/1, /run/netns, and /dev/kvm. EKS node bootstrap provides these; GKE node pools have no bootstrap hook, so the example adds a node-init DaemonSet that formats and mounts the raw local SSDs (striping them when there are several) in the host mount namespace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Give pods 120s to stop and set the runtime's 90s drain explicitly - Pass FUNCTION_SECRET_KEY so stored MCP credentials decrypt - Render ORG_NAME as * when orgName is empty, as the API does - Bound the Brainstore file cache, size-limit the state and cache emptyDirs, and request matching ephemeral storage - Reject blank, null, padded, and non-CIDR egress source ranges - Node init: mount the SSD at the parent and create the worker directory on it, so workers cannot start before the mount - TLS script: check both clusters first and write the worker server Secret last, so a failed run can be retried - Note the dependency on terraform-google-braintrust-data-plane#28 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An empty, null, or missing repository or tag rendered an invalid image reference that only failed when pods started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
Model loopRuntime on the AI Gateway: the same value shapes, probes in values, and fixed defaults for settings with one correct value (state and cache paths and limits, security contexts, the 120s grace period, the TLS Secret name, the Git proxy port, in-cluster service URLs). Drop knobs no install needs and the tests that only covered them, and shorten the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a858ef5645
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 61788e2e30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2f725580eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
|
Codex (@codex) review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e22a347c29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| image: alpine:3.22 | ||
| command: [sh, -ec] | ||
| args: | ||
| - | | ||
| apk add --no-cache bash util-linux xfsprogs mdadm |
There was a problem hiding this comment.
This privileged container on the sandbox hosts uses alpine:3.22 with no digest and runs apk add from the internet on every start, with the host /dev mounted. Please use a prebuilt image pinned by digest.
It also needs one maintained owner. terraform-google #28's design doc says the worker release prepares the disks, but here it's an example manifest applied with kubectl. Whether that owner is the worker chart or node provisioning is a choice to settle across this PR and #28.
There was a problem hiding this comment.
Node preparation now belongs to the pinned worker release as an init container, using a prebuilt image pinned by digest. Removed the separate example DaemonSet. The worker starts only after the existing KVM/XFS checks complete; a GKE worker upgrade completed with the init container exiting successfully and the replacement worker becoming ready.
Erik Weathers (erikdw)
left a comment
There was a problem hiding this comment.
Note
This is a summary comment for the inline comments above
Keep this architecture. Loop Runtime in the Braintrust chart, the Sandbox Service workers as a separate release in an isolated GKE cluster, and infrastructure in Terraform are the right ownership boundaries.
Status: no confirmed P1 in the findings below. It isn't ready yet: it depends on terraform-google-braintrust-data-plane #30 → #27 → #28, which are still in progress (#28 is a draft), and nothing has run end to end on GKE.
Fixed since the last review: 120-second grace period with a 90-second drain, FUNCTION_SECRET_KEY, empty orgName rendered as *, and an explicit disk-cache budget. Thanks!
Still to decide (inline comments): TLS Secret writes, runtime sizing, node-prep ownership and pinning, and the sandbox admin default. On images: the runtime and worker use commit-specific tags, and the sandbox guest is pinned by digest. A commit-named tag can be re-pushed unless the registry enforces immutability, so moving to release tags (or digest pins, if the contents must stay fixed) is optional.
GKE validation still needed: create a sandbox, use Git through the egress proxy, run a real Loop turn, replace a runtime pod, and upgrade, drain, and uninstall the workers.
Checks: helm unittest (470 tests) and helm lint --strict pass, and the GKE example renders on the Autopilot values.
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96d1639deb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| BRAINSTORE_CONTROL_PLANE_TELEMETRY: {{ .Values.global.controlPlaneTelemetry | quote }} | ||
| NO_COLOR: "1" | ||
| EXO_SANDBOX_SERVICE_DNS_URL: {{ $sandbox.dnsUrl | quote }} | ||
| EXO_SANDBOX_SERVICE_IMAGE: {{ $sandbox.image | quote }} |
There was a problem hiding this comment.
Validate the sandbox image before rendering
When an enabled deployment sets loopRuntime.sandbox.image to null or an empty string, validation still succeeds and this emits an empty EXO_SANDBOX_SERVICE_IMAGE; the runtime then starts but cannot create any sandbox. Require a non-empty sandbox image and cover its missing, null, and empty forms.
AGENTS.md reference: AGENTS.md:L15-L17
Useful? React with 👍 / 👎.
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c6f9817322
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| ports: | ||
| - name: {{ .Values.loopRuntime.service.portName }} | ||
| protocol: TCP | ||
| port: {{ .Values.loopRuntime.service.port }} |
There was a problem hiding this comment.
Reject invalid runtime service ports
Validate loopRuntime.service.port before rendering it. When an enabled configuration explicitly sets this value to null, an empty string, 0, or a value above 65535, Helm still renders the Service and Deployment, but Kubernetes rejects their required port fields; the same value also produces unusable internal URLs and probe configuration. Add range/type validation and cover the empty and boundary cases required by the repository's input-matrix policy.
AGENTS.md reference: AGENTS.md:L15-L17
Useful? React with 👍 / 👎.
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bc1c5bc2df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| {{- range $key := list "sandbox" "image" "service" "serviceAccount" "resources" "annotations" -}} | ||
| {{- if not (kindIs "map" (get $.Values.loopRuntime $key)) -}} | ||
| {{- fail "loopRuntime defaults are missing. Values reused from a chart release that predates Loop Runtime omit them; upgrade with --reset-then-reuse-values." -}} | ||
| {{- end -}} |
There was a problem hiding this comment.
Validate nested defaults on reused-value upgrades
When a pre-Loop-Runtime release is upgraded with --reuse-values and partial overrides supply each outer map but omit nested defaults such as image.pullPolicy, service.type, service.portName, or serviceAccount.name, this guard passes because it checks only the map types. The templates then emit empty Kubernetes fields or an unnamed ServiceAccount, so admission fails instead of returning the intended --reset-then-reuse-values guidance; validate the required nested structure or overlay template defaults before rendering.
AGENTS.md reference: AGENTS.md:L14-L17
Useful? React with 👍 / 👎.
Adds an opt-in Loop Runtime for GKE. Sandboxes run on Sandbox Service workers in a separate isolated GKE cluster.
On AWS, Loop Runtime stays on ECS through the Terraform module, so the chart rejects other clouds.
The GCP Terraform module supplies the worker cluster, discovery zone, Loop Runtime service account, and reserved egress address.
discovery.endpointand mounts thesandbox-client-tlsSecret.<service>-egress: an internal load balancer on the reserved address that admits only worker ranges.LOOP_RUNTIME_URL; the VirtualService routes/loop/runtime.examples/gke-loop-runtimesupplies worker values and the mTLS Secret setup script.Input matrix
loopRuntime.enabledfalse, orloopRuntimemissing (--reuse-values)--reuse-values --set loopRuntime.enabled=true, including partial sandbox overrides with missing default maps--reset-then-reuse-valuescloud: aws,azure, or emptyreplicasunset / null / empty /0loopRuntime.service.portloopRuntime.enableSandboxAdminmissing / null / false / trueglobal.orgNameemptyORG_NAME: "*"locksBackend: objectStorage/redisVerified on GKE: cross-cluster mTLS and private DNS; sandbox create, execute, stop, and terminate; Git through the egress proxy; real Loop turns with SQL and shell tools; runtime pod replacement; and a worker upgrade with the node-preparation init container. Two concurrent turns peaked at 329MiB of runtime cgroup memory within the 2Gi pod limit. Larger histories and higher concurrency still need load testing; worker uninstall has not been exercised.
./test.shpasses (478 Helm cases, 33 Python tests, lint, cloud renders); the worker chart's three focused tests pass. Earlier Linux loop-device checks covered fresh XFS initialization, mount propagation, preservation across remounts, and rejection of ext4 and XFS subdirectory mounts. The TLS script retains the prior partial-failure recovery flow and now uses server-side apply, removing legacy key-bearing annotations.🤖 Generated with Claude Code