Repository navigation
Sync upstream authorization and object-store updates - #72
Merged
Merged
Conversation
…trate#2452) #### Breaking change `GetActorEgressPolicy`, `DeleteActorEgressPolicy`, `GetAtespaceAccessPolicy`, `DeleteAtespaceAccessPolicy` and `GetGlobalAccessPolicy` now require `name` set to `default`. Please add the `breaking-change` label. Fixes agent-substrate#2363 Adds `name` to those requests, per the style guide. Any other value is `INVALID_ARGUMENT`. `resources.SingletonName` replaces the repeated `"default"` literals. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ate#2509) 1) Make docs consistent for actor-templates wrt everything else. 2) Normalize all CLI help text on singular nouns with plurals as aliases 3) Don't accept plural commands where we can't handle more than 1 item #### Breaking change Some CLI usage of plurals breaks, but it wasn't DOING plural things. - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR
agent-substrate#2513 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
### Summary Expose the `WorkerPoolSyncer` worker goroutine count via `--worker-sync-concurrency` (default `2`, preserving existing behavior) instead of a hardcoded constant, so large-scale benchmarks can raise worker-registration concurrency without patching Go sources at runtime. ### Testing `go test ./cmd/atecontroller/...` - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
) Fixes agent-substrate#2373 ate-api-server no longer links the GCS and S3 SDKs. The plugin socket helpers move to `internal/pluginsocket`, and `hack/verify/ateapi-deps.sh` guards it. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Haven Xia <haoyuxia@google.com>
Add tool for benchmarking Substrate egress. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Yan Avlasov <yavlasov@google.com>
`DrainWorker` was the only custom method that returned its resource
directly; every other custom method returns a `{MethodName}Response`
wrapper. A wrapper keeps the API uniform and leaves room for
method-specific output in the future.
The API style guide now requires custom methods to return a
`{MethodName}Response` and no longer lets them return the resource
itself.
Part of agent-substrate#2411
…#2413) Addresses agent-substrate#2377. ## What we saw Under CPU contention (14 sandboxes per worker pod, each opening about a thousand tunnels), 1 to 8% of gVisor sandboxes died mid-run. The actor stayed `RUNNING`, `SuspendActor` succeeded, and the next resume failed with `inconsistent private memory files on restore`. Sentry logs showed `Stub thread no longer exists; killing syscall thread`. Across the ladder runs on the nightly we lost 1/299, 17/968 and 79/999 actors, and 79 and 73 of 1000 in two seven-minute repros. ## Why the bump helps The 2026-09-02 nightly predates gVisor commit `80bb741691be` ("Close exposed race window in stub ThreadID setting"): a sentry that interrupts a stub thread mid context switch can read an invalid thread id and kill the whole subprocess. Release 20260907.0 is the first tagged release containing it (GitHub compare: 25 commits ahead of the fix, 0 behind). The same repro on 20260907.0 lost 0 of 1000 actors, as did the 34-minute 1000 x 1000 rung of the egress ladder. ## Change `manifests/ate-install/sandboxconfig-gvisor.yaml` gains version `v2` with release 20260907.0 as `defaultVersion`; the nightly entry is removed. `docs/api-guide.md` mirrors it and now states that a resume resolves the default version at restore time, so after this rolls out recut golden snapshots and expect snapshots captured under the nightly to fail on their next resume. ## Test signals - `go test ./pkg/api/v1alpha1/` applies the edited manifest against the CRD and the admission policy in envtest; `./cmd/ate-setup/...`, `./cmd/atelet/...` and the ateapi sandbox tests pass. - CI `e2e-test` installs this manifest through `ate-setup` and boots, suspends and restores gVisor actors, so a green run exercises the release. - Both sha256 values were recomputed from the downloaded tarballs and match the published `.sha512` files. - The loss counts above come from the issue's gist and the ladder run, not re-verified in this PR. Prepared with help from an AI coding agent; reviewed and verified by the author.
…strate published (agent-substrate#2418) ClusterTrustBundles are cluster-scoped, so deleting the ate-system and podcertificate-controller namespaces left Substrate's four bundles behind. - Once both namespaces are gone (so the controllers can't republish), teardown deletes exactly these bundles, each only when its signer also matches: - `podidentity.podcert.ate.dev:identity:primary-bundle` (signer `podidentity.podcert.ate.dev/identity`) - `servicedns.podcert.ate.dev:identity:primary-bundle` (signer `servicedns.podcert.ate.dev/identity`) - `postgres.podcert.ate.dev:identity:primary-bundle` (signer `postgres.podcert.ate.dev/identity`) - `egress-mitm.ate.dev:mitm:primary-bundle` (signer `egress-mitm.ate.dev/mitm`) - A foreign bundle under one of these signers with another name is kept. Tested with unit tests and a kind install plus teardown: the four bundles were deleted; the kube-apiserver bundle and foreign bundles, including two under our signers, were kept.
…durable dirs (agent-substrate#2455) Use gVisor's split filesystem checkpointing from the 2026-10-10 nightly release instead of host-side tar archiving for durable-dir volumes: - Mount durable-dir volumes as container-shared tmpfs inside the gVisor sandbox in ShapeGVisor. - On MEMORY checkpoints, pass -fs-checkpoint-paths to runsc checkpoint so durable-dir volumes are checkpointed into the fs/ subdirectory alongside the process and rootfs checkpoint. - On VOLUMES checkpoints, run runsc fscheckpoint with -image-path <checkpointDir>/fs and -path <container>:<mount_path> for each durable-dir mount. - On RestoreWorkload, pass --fs-restore-image-path <checkpointDir>/fs to runsc create on the pause container when fs/fscheckpoint.pb is present, then either restore the full sandbox (MEMORY) or start fresh containers with only durable-dir volumes restored (VOLUMES). - Allow fs/<file> relative paths in snapshot file validation, local pause staging, and object store plugin transfers. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…nt-substrate#2371) The digest that turns an environment change into a rollout was patched onto the running Deployment before the deploy applied the new manifest, and a fresh install skipped it. So the first `deploy apiserver` after an install actually **rolled twice** whenever it brought a new image: once for the new annotation on the old image, then again for the new image. When the environment did change, the old image came up on the new values before the new image landed. A deploy now stamps the digest on the Deployment it applies, so a new image and a new environment are one rollout and a fresh install starts out stamped. A deploy whose manifest has no such Deployment now fails rather than applying it without the digest. `create api-server-env-vars`, which applies no Deployment, still patches the running one. Made with help from Claude Code Part of agent-substrate#1887 and it unblocks agent-substrate#1686
agent-substrate#2535) agent-substrate#2513 ### Changes - Re-enable the credential injection e2e which was skipped in agent-substrate#2518 - `testserver http` gains `/echoheaders`, which returns the request headers it received as `{"headers": {...}}`. - The suite deploys two echo origins in its own namespace: - `echo-https` on 443, serving a certificate from the cluster's service-DNS signer through a projected `podCertificate` volume. The egress gateway trusts that signer once `hack/setup-e2e-egress-tls-kind.sh` has run, and CI already runs it. - `echo-http` on 80. - The fail-closed hosts are `.invalid` names that exist only in the policy. - Coverage is unchanged: injection and no-header checks over HTTPS and plain HTTP, plus the three fail-closed cases (unfetchable secret, unserved provider, unauthorized namespace) on both schemes. ### Testing - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…ubstrate#2439) atelet held its own storage credential to download sandbox assets, and private micro-VM assets only worked on GCS or S3. Assets now go through the node's object-store plugin, which already has the credentials. - Add `AssetProvider.FetchAsset` to the plugin API. - The plugin refuses snapshot locations, stages downloads where atelet can't read them, and replaces the target only once the sha256 matches; `pkg/assetfetch` shares these checks with other plugins. - atelet asks the plugin first for every asset. Only for a `gs://` URL, and only when the plugin returns NotFound, InvalidArgument, Unimplemented or Unavailable, does it fall back to an anonymous read (which serves public buckets such as `gs://gvisor`). It never falls back when the plugin refuses a snapshot location or the size or digest is wrong. - atelet re-checks the size and sha256 of whatever it caches, since the plugin is a separate, possibly third-party process and atelet executes these bytes. Tested with the kind gVisor and micro-VM lanes before the plugin-first change; only the NFS external-volume tests failed, same as on main. Unit tests cover each fallback and non-fallback code. Related agent-substrate#2063: assets in the cluster's own store no longer wait on an anonymous GCS lookup first. #### Breaking change Plugins must implement `AssetProvider` to serve sandbox assets; public `gs://` assets still load anonymously if the plugin can't serve them. atelet exits if `ATE_STORAGE_BACKEND` is set on it; set storage settings on the snapshot-plugin sidecar. Please add the `breaking-change` label. Fixes agent-substrate#2385 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ngress router (agent-substrate#2387) The ingress router can now be started up in two modes, controlled by the `--ingress-auth-mode` flag: `deprecated-insecure` (the default), and `static-mtls`. Both modes require you to specify the new `--ingress-client-ca-file` and `--ingress-allowed-spiffe-ids` flags. By default, the validation CA is set to the pod identity CA. In deprecated-insecure mode, the router listens on both plaintext and HTTPS ports. * If a client connects on the plaintext port, they are allowed to connect to any actor without authentication and authorization. * If a client connects on the HTTPS port, but does not send a client certificate in the TLS handshake, then they are allowed to connect to any actor without authentication and authorization. * If a client connects on the HTTPS port, and provides a client certificate, then the certificate will be checked against `--ingress-client-ca-file` and `--ingress-allowed-spiffe-ids`. In static-mtls mode, the router only listens on the HTTPS port: * Clients are required to send a client certificate, and it is always checked against `--ingress-client-ca-file` and `--ingress-allowed-spiffe-ids`. These two modes allow a smooth migration: * Land this PR, which keeps the default at deprecated-insecure. * The next PR will allow ate-setup to configure the cluster into static-mtls mode. * The next PR(s) will cut over end-to-end tests and demos so that they send a client certificate. * The final PR will remove the deprecated-insecure mode.
… RPCs with OpenFGA (agent-substrate#2185) Working on agent-substrate#1563 Fixes agent-substrate#1700 Adds OpenFGA checks for the remaining actor RPCs (except the identity ones) and for all worker RPCs. These checks are enforced when `--experimental-enable-authz` is set. ### Checks | RPC | Check | Who passes | |---|---|---| | `PauseActor` / `ResumeActor` / `SuspendActor` / `RevertActor` | `can_pause` / `can_resume` / `can_suspend` / `can_revert` on `actor` | atespace editor and above | | `GetActorEgressPolicy` | `can_get_egress_policy` on `actor` | atespace viewer and above | | `Create/Update/DeleteActorEgressPolicy` | `can_create/update/delete_egress_policy` on `actor` | atespace editor and above | | `CreateWorker` / `ListWorkers` | `can_create_worker` / `can_list_workers` on `global:root` | global owner / global viewer | | `GetWorker` / `ListWorkerActorAssignments` | `can_get` / `can_list_actor_assignments` on `worker` | global viewer | | `Update/Delete/DrainWorker` | `can_update` / `can_delete` / `can_drain` on `worker` | global owner | ### Not in this PR - **System component roles (follow-up PR).** With enforcement on, the ateapi callers inside Substrate need grants: - `atenet-router`: `ResumeActor`, plus `ListActors` for its health check; - `atenet-egress`: `GetActor`, `GetActorEgressPolicy`; - `ate-controller`: the worker RPCs. Today the only way to grant them is global owner or AccessPolicy roles. A follow-up will add system roles granted from ateapi's configuration to the components' mTLS SPIFFE IDs, the same way bootstrap owners are granted. Until then, don't turn on `--experimental-enable-authz` in a cluster that relies on these components. - **Identity RPCs** (`MintActorJWT`, `MintActorCertificate`) are handled separately (see agent-substrate#2026). ### Testing - Registry tests for each new RPC, plus fail-closed cases for missing or invalid identifiers. - Interceptor tests against Postgres-backed OpenFGA covering editors, viewers, outsiders, global viewers, global owners and bootstrap owners. Removing the worker → `global:root` contextual link fails the worker cases. - OpenFGA model tests for the new relations (`fga model test`: 6/6 tests, 185/185 checks).
…i-server can use it (agent-substrate#2519) Part of agent-substrate#777 This is the first, mechanical step: later PRs add a `MintAteomActorJWT` RPC on `WorkerService`, have the snapshot plugin's control sidecar exchange that JWT for a downscoped GCS token scoped to the actor's own prefix plus its source tag, and have the node plugin transfer with that token. The design, including why downscoped tokens rather than the signed URLs the issue suggests, is in the design doc that accompanies those PRs. The snapshot plugin's control sidecar is about to verify actor JWTs before exchanging them for downscoped object-store credentials. The verifier in `oidcjwt` lived under `cmd/ateapi/internal`, where only ate-api-server can import it. This moves it to `internal/oidcjwt` so other binaries in the module can share it, keeping JWT verification on the one standard-library implementation. Pure move: the four files are unchanged (`git` records them as renames), ate-api-server's import path is updated, and the pointer in `AGENTS.md` follows the package. No behavior change. - [x] Tests pass (`go test -race ./internal/oidcjwt/... ./cmd/ateapi/...`; the `cmd/ateapi` postgres password-logging test only fails locally with `PGPASSWORD` set in the shell, on `main` as well) - [x] Appropriate changes to documentation are included in the PR (`AGENTS.md`)
…gin (agent-substrate#2540) The plugin now serves sandbox assets as well as snapshots, and its API is already `objectstore.v1`, so the name `snapshot-plugin` no longer describes it. This renames the binary, sidecar, sockets and flag to match. The `objectstore.v1` proto package and RPC names are unchanged. - `cmd/snapshot-plugin` moves to `cmd/objectstore-plugin`; the atelet and ate-api-server manifests and the kind overlays follow. - `--snapshot-plugin-socket` becomes `--objectstore-plugin-socket` on atelet and ate-api-server, with no alias. - Go identifiers, log and error text, `docs/architecture.md` and `hack/verify/ateapi-deps.sh` now say "object-store plugin". #### Breaking change The `cmd/snapshot-plugin` image is now `cmd/objectstore-plugin`, the `snapshot-plugin` sidecar container and volume in atelet and ate-api-server are now `objectstore-plugin`, the sockets move from `/run/snapshot-plugin` to `/run/objectstore-plugin`, and the `--snapshot-plugin-socket` flag on atelet and ate-api-server is now `--objectstore-plugin-socket`. The sidecars' trace `service.name` changes from `snapshot-plugin-<mode>` to `objectstore-plugin-<mode>`. Overlays that patch the `snapshot-plugin` container, pass the old flag or filter on the old service name must be updated. Needs the `breaking-change` label. Tested with `go build ./...`, `go test -race` on cmd/ateapi, cmd/atelet, cmd/objectstore-plugin, cmd/ate-setup, internal/objectstoreplugin and internal/pluginsocket, and `hack/verify/ateapi-deps.sh`. The base, kind, kind-agentgateway and agentgateway kustomize renders differ from before only by the rename. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…agent-substrate#2403) On a new workstation, a Kind demo install reports only a missing kubeconfig context. On a cluster without the control plane, it can create the demo Namespace and WorkerPool before failing while waiting for a Deployment that no controller can create. Check Docker and the resolved kubeconfig context before Kind deployments, with instructions to restore kubeconfig or create the cluster and local registry. Demo and benchmark commands check for `deployment/ate-controller` before creating resources. Permission and connection errors retain their original cause. The checks live in `ate-setup`, so both native commands and the shell shims use them, including when a demo flag precedes `--deploy-ate-system`. The controller check establishes that it is installed; it does not replace workload rollout checks. Help and delete commands do not require the new deployment checks. Cluster recovery guidance states that the creation script replaces an existing cluster with the same name. Fixes agent-substrate#2400. Validation: - Regression tests first failed on the original implementation. Tests cover native commands and Kind shims, missing context and Docker, demo and benchmark prerequisites, action order, installed controllers, Forbidden responses, resolved configuration, malformed kubeconfig, and help. - `go test -race ./cmd/ate-setup/...` passed on macOS arm64. - The original isolated reproduction passes with the fix; the missing-controller scenario makes no resource writes. - `make verify` passed on Linux amd64 with Go 1.27.1: all 145 test packages passed with the race detector, and all 14 verification scripts passed. - The Linux `ate-setup` binary built successfully from the final commit. Generated files and the Git working tree remained clean.
…ate#2446) Return the reader's close error when the download succeeds, and preserve the original download error when both fail. Add regression tests with valid and truncated zstd streams. Refs agent-substrate#2445 Tested with `go test -race ./pkg/objectstorage -count=1`.
Rebases agent-substrate#1074 (Add benchmarking for storage, by @elijahrb) onto main and runs it with the DurDir load. Replaces agent-substrate#1074. - Adds the `glutton-storage` ActorTemplate, which mounts an external CSI volume as glutton's data dir. `workloads/deploy.sh --storage-class-name NAME` deploys it, and `deploy_locust.sh` passes the flag through. Without a StorageClass it is not deployed, so clusters with no CSI driver don't wait on its golden snapshot. - The DurDir load drives it with `--durdir-template glutton-storage`. DurDir already resumes an actor, writes a file, checks the file's checksum after every resume, and suspends the actor again, so the storage benchmark needs no load of its own. DurDir does not count crashed actors under `CrashCount` yet. - glutton gets two opt-in flags. Only `glutton-storage` sets them, so the baseline and DurDir numbers don't change. `--sync-writes` fsyncs each `WriteDisk` file. `--readyz-require-writable-data-dir` fails `/readyz` until the data dir is writable, so a resume waits for the volume mount. - `csi-nfs-sc` is now defined in one file, `hack/third_party/csi-driver-nfs/deploy/example/storageclass-nfs.yaml`. `ate-setup setup csi nfs` and `hack/setup-csi-nfs-kind.sh` both apply it instead of each carrying a copy, so the e2e tests, demos and benchmarks use the same class. A test checks this. To benchmark on NFS, install with `hack/install-ate.sh --setup-csi=nfs` and pass `--storage-class-name csi-nfs-sc`. - The Locust scripts take `LOCUST_IMAGE` or `KO_DOCKER_REPO`, so they work on Kind with a local registry, and `LOCUST_IMAGE_PULL_POLICY` (default `Always`). The orchestrator and `tests.yaml` don't change. Letting a test set a StorageClass or apply extra manifests moves to a follow-up PR, where the test API can be worked out. Testing: unit tests for the glutton flags and the shared StorageClass. `go test -race` on the changed packages, golangci-lint, shellcheck and the boilerplate check pass. Not run end to end on a cluster. - [x] Tests pass --------- Co-authored-by: Elijah Rodriguez-Beltran <elijahrb@google.com>
Integrate singleton policy requests, actor lifecycle authorization, sandbox asset staging, and split filesystem checkpoints. Align the Helm charts and image builds with the objectstore-plugin rename while preserving fork workflow deadlines and development transport settings. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261013
branch
from
October 13, 2026 10:24
592a0bd to
12118a9
Compare
EItanya
changed the title
EItanya
marked this pull request as ready for review
October 13, 2026 10:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs upstream main at be3aa280f into the fork with one merge commit. Includes actor lifecycle/egress/worker authorization, private sandbox asset downloads, split filesystem checkpoints, the object-store plugin rename, and installer and benchmark updates.
Aligns the Helm charts and image build workflows with
objectstore-plugin, including isolated asset staging, plugin-only storage settings, the gVisor runtime pin, and the SandboxConfig CRD. Preserves the fork's workflow deadlines, insecure development mode, and credential fixture endpoint alongside upstream request-log redaction and header echo support.Pins agentgateway to 9557280b, a five-line client and test update on the previously pinned source. It sends
name: "default"when fetching actor egress policy; the old image omitted it, causing both E2E jobs to fail API validation. The image is built for amd64 and arm64 by the image workflow.Breaking change
substrate-crdsbefore the application chart, and upgrade gVisor SandboxConfigs with ateom-gvisor. The chart pins the 2026-10-10 nightly (v3) required for split filesystem checkpoints.snapshotPluginHelm settings toobjectStorePlugin. Move storage environment overrides fromatelet.extraEnvtoobjectStorePlugin.extraEnv; atelet now rejectsATE_STORAGE_BACKEND. The chart selects the backend throughatelet.storageBackend. Private S3 sandbox assets must uses3://URLs.name: "default", andDrainWorkerreturns aDrainWorkerResponsewrapper. Deployments enforcing OpenFGA authorization need the new actor lifecycle, egress policy, and worker permissions.Upstream's ingress
static-mtlsmode supports Envoy only; the agentgateway chart retains its existing ingress configuration.Validation: all 44 agentgateway egress integration tests pass. The full local Go race suite passes with
env -u NO_COLOR go test -race -p 1 ./...; manifest rendering, CRD parity, and all three chart lints pass. Local repository verification passed code generation, module checks, formatting, and boilerplate, then stopped during linting because the shared 20 GB scratch filesystem filled. All four CI checks pass on12118a988: unit tests and verification, plus agentgateway E2E, Helm E2E, and Helm verification.Runtime results (passed/skipped): agentgateway gVisor 124/6 and microVM 128/2; Helm gVisor 120/5, microVM 30/1, egress credentials 9/0, and identity 1/0. Skips are the existing runtime-specific cases and the Helm-only credential test in manifest installs.
Merge with a merge commit to preserve upstream ancestry.