feat(python): build Substrate driver and runtime directly on Client a…

[Sunny (sunnylovestiramisu)]

Summary

Replaces the previous mock/synthetic driver and runtime stubs with direct implementations built on top of the native Agent Substrate Python SDK (Client and Env).

This PR establishes PR 1 of the 3-part SDK overhaul, providing a robust, production-ready foundation for managing Substrate environment lifecycles and in-guest execution.

---

Key Changes

1. Direct Client & Env Integration

• SubstrateBackendDriver:
  • Manages environment lifecycle directly via Client (create, suspend, delete, reap).
  • Implements warm pool caching and reuse logic, directly draining _warm_paused_pool on reap/teardown.
  • Supports both warm pool claims (node-local fast resume) and cold environment creation.
  • Passes auth tokens via gRPC bearer metadata interceptor.
  • Configures acquire_timeout_s, ready_timeout_s, and custom tags/labels.
• SubstrateEnvClientRuntime:
  • Communicates directly with in-guest ProcessService and FileSystemService over gRPC.
  • Multiplexes requests using x-env-id and x-env-atespace gRPC metadata headers.
  • Guest command timeout is enforced in-guest; returns structured output with exit codes, stdout, and stderr.
  • Defaults command cwd="" to respect the container image's native WORKDIR.

2. Event Loop Concurrency (LoopRunner)

• Introduced a dedicated background event loop runner (LoopRunner) for synchronous facades (SandboxFleet, SubstrateBackendDriver, SubstrateEnvClientRuntime).
• Solves gRPC channel reuse issues and prevents nested/conflicting asyncio event loop errors across sequential synchronous calls.

3. Clean Error Hierarchy & Status Mapping

• Consolidated errors under ate_env.errors:
  • SandboxStartError and PreflightError now inherit from InfrastructureError.
  • Distinguishes infrastructure-level failures (timeouts, gRPC transport drops) from guest execution failures (CommandError).
  • Clean mapping from gRPC status codes (DEADLINE_EXCEEDED, UNAVAILABLE, NOT_FOUND).

4. Zero External Pydantic Dependency

• Replaced Pydantic models in config.py with standard library @dataclass(frozen=True) and a lightweight SecretStr implementation for secure token handling.

5. Fleet Lifecycle & Warm Pool

• Added context manager support (with SandboxFleet(...) as fleet:) ensuring deterministic teardown and reaping.
• Added setup(tasks=None) for batch pre-warming sandboxes before rollout iterations.
• Dropped deprecated router/data-plane artifacts (router_url, grpc_endpoint, host/port stubs).

6. Hermetic Unit Tests

• Added comprehensive unit test suite with hermetic fake ate-env-api gRPC server verifying driver lifecycle, runtime execution, and fleet strategies.

---

Tests

Unit tests passed, and tested with the full change in #75

[Tomer Glottmann (tomergee)]

Review of PR 1 at b33b984. I ran the 116 unit tests (pass), read the driver, the runtime, the loop runner, the fleet, the handle, the config, the errors, the types and the driver tests, and probed the fleet against the fake stack.

Request changes. The two blocking items from the #75 review are fixed, and most of that list is done. Two items from that list are still open, and three new defects stop a default configuration from working. The items are in order of severity.

Status of the #75 review items

Item	Status
1 one-shot asyncio.run facade	fixed: LoopRunner, one loop thread, run_coroutine_threadsafe
2 warm pool survives teardown	fixed: reap_async clears the pool; a test covers it
3 no real template name	partly: FleetConfig.template and Task.template exist; the default path is still wrong (B1)
4 USER_GUIDE examples	fixed: context manager, list acquire / release, setup(tasks), two tests
5 preflight cannot fail	fixed, a test covers it
6 failed deletes hidden	fixed: WARNING, reap raises with the list
7 unused parameters	partly: acquire_timeout_s and auth_token work; ready_timeout_s is cosmetic (M2); step_timeout_s, worker_family, node_selector, tolerations are still unused
8 exec returns str, __exit__ recycles	not done (B2)
9 router and per-actor endpoint code	fixed
10 error taxonomy	fixed
11 tests against the fake server	fixed: sync fleet, setup/teardown, preflight
12 lists run in sh -c	fixed: start_process for argv
13 host / port / ip_address	fixed

Blocking

B1. fleet.py:272 with backend/substrate.py:158 and types.py:234 — a fleet with no template configured requests a template that does not exist. Probe against the fake stack: FleetConfig(backend="substrate", endpoint=...) plus Task(image="docker.io/library/python@sha256:…") creates with template.name == "tmpl-python@sha256-2176bb31f3eb". That name exists on no cluster, and @ is not valid in a resource name, thus the API rejects it before it looks it up. The quickstart registers default-template; use it as the default. Better, after the rebase onto main: pass rootfs_image=task.image to Client.create and delete template_key, image_templates and _image_to_template; #70 derives the template per image on the server. Also at backend/substrate.py:158: default_template != "default-template" treats the real default name as "unset". Use None for unset.

B2. handle.py:52 and handle.py:139-143 — exec returns str and drops stderr and exit_code when check=False; __exit__ and __aexit__ call recycle() on success, thus a sandbox that ran agent code goes back to the pool without a reset, by default. The #75 review asked for both in PR 1 if it became its own PR. Return ExecResult; release on exit and make recycling explicit.

B3. fleet.py:87 and fleet.py:255 — close() calls teardown() before backend.close(). reap_async now raises InfrastructureError when one delete fails (item 6), thus close() raises and never stops the loop runner or the client. Inside with SandboxFleet(...) the raise also replaces the original exception. Reap in try, close the backend in finally, and re-raise after.

Before merge

M1. runtime/substrate_env_client.py:156-193 — on the client-side timeout (timeout_s + 5 s) the argv path returns timed_out=True and leaves the process running in the guest until the guest watchdog, one hour by default. Process.kill() exists (process.py:356). Kill the process before you return, on the argv path and on the shell path.

M2. backend/substrate.py:212-223 — the "readiness probe" for a claimed warm actor is client.get, and a failure only logs a warning. A warm actor deleted or crashed after it was suspended is handed to the caller, and the first command fails. On NotFoundError drop the id and take the next one or create; on another error give the actor back and raise SandboxStartError. State in the docstring that resume is implicit on first traffic, because ready_timeout_s does not measure readiness.

M3. backend/substrate.py:248 and substrate.py:260 — the create loop classifies failures with substrings: "template" in err_msg matches any message that mentions a template. The client maps the server's statuses to NotFoundError and FailedPreconditionError (errors.py); the e2e in main already asserts FailedPreconditionError for a missing base. Match the types.

M4. loop_runner.py:55-63 — stop() joins for 2 s and then calls loop.close() even when the thread is still running; close() on a running loop raises. Pending tasks are not cancelled, thus the create_task in warm_pool_async(wait=False) (substrate.py:293) is dropped silently. stop() from the runner thread joins itself and deadlocks, which __del__ can trigger. Cancel pending tasks on the loop, run shutdown_asyncgens, close only after the thread exits, and refuse stop() from the loop thread.

M5. fleet.py:272 — config.template wins over config.image_templates. A user who sets both gets the generic template for every image. Per-image mappings must win over the default.

M6. README.md:9 — the client README advertises AsyncSandboxFleet, which is PR 2. Remove it from this PR.

M7. substrate.py:387 and unwarm_pool_async — deletes run one at a time. warm_pool_async uses a semaphore and gather; use the same for reap and unwarm, or a pool of 200 takes minutes to tear down.

Nits

• substrate.py:166: self._templates is written and never read.
• substrate_env_client.py:219: NotFoundError is split by the substrings "environment" and "actor"; the server message is the contract here. Use one error, or add a code to NotFoundError.
• substrate_env_client.py:270: the synchronous stream_process pump is never cancelled when the consumer stops early; wrap the generator in try/finally and cancel the future.
• substrate.py:293: keep a reference to the fire-and-forget suspend task and log its exception.
• config.py:93: step_timeout_s is validated and never read. worker_family, node_selector, tolerations reach the driver and have no effect on Substrate; placement is the template's. Remove them or say so in the docstring.
• config.py:71: endpoint defaults to http://localhost:7777; the client strips the scheme, but every other address in the repo is host:port. Use 127.0.0.1:7777.
• __init__.py:63 and exceptions.py:43: the deprecated shim is still the source of SandboxError, and it still exports builtins.TimeoutError under the same name.
• runtime/base.py:46: write_file_via_exec and read_file_via_exec have no callers. Remove them.
• config.py: the 70-line __init__ with object.__setattr__ can be a dataclass with __post_init__; model_dump and model_validate are pydantic names on a non-pydantic class.
• backend/mock.py still uses template_key; it goes with B1.

What I ran

uv run --extra dev pytest (116 passed); a probe test against fake_stack for three template configurations (default, default-template, rootfs-base); the leftover sweep for the #75 items. Not run: the live smoke on the sandbox cluster. B1 makes a default-config live run fail before it starts, and with an explicit template the live path is the one measured on 2026-10-12.

[Tomer Glottmann (tomergee)]

Second pass at 36f81ca. Thank you, this is a different PR: B2, B3, M1 to M7 and most nits are done, the headers are in, and the 118 tests pass. One item from B1 is left, the image pass-through. It is not a blocker for this PR: it goes to PR 2 of the SDK, see below.

Blocking: none.

Before merge

1. PR description. The line about guest_pb2_grpc.py version checks describes a change that is no longer in the branch. Remove it, and add a short list of what changed after the first review, so the next reader does not diff two commits to find out.

For PR 2 of the SDK, not a blocker here

fleet.py:213 → backend/substrate.py:247 and :297. The task image does not reach the server. A probe against the fake stack with a default FleetConfig and Task(image="docker.io/library/python@sha256:…") sends template.name == "default-template" and rootfs_image == "". The sandbox runs the guest's own image, not the task's, unless the image is in image_templates. The branch is on main, thus Client.create(rootfs_image=…) is available and the server derives the template per image (#70, see the e2e test gated on ATE_ENV_ROOTFS_IMAGE). Make _ensure_template_for_image return (template, rootfs_image): a mapped image gives (mapped, None); any other image gives (config.template or "default-template", image), where the configured template is the base. Let BackendDriver.acquire and warm_pool take rootfs_image, pass it at both client.create calls, and key _warm_paused_pool by (template, image). Add a test that asserts last_create_request.rootfs_image. This is work for PR 2, not for this PR: the fleet API change (acquire and warm_pool signatures, the pool key) belongs with the async fleet and the strategies. For this PR, one sentence in the Task.image docstring that the image is a lookup key for image_templates until PR 2 is enough.

Nits

• loop_runner.py:74-76: loop.close() runs after join(timeout=2.0) also when the thread is still alive, and close() on a running loop raises. Guard it with if not self._thread.is_alive().
• config.py:93: step_timeout_s is validated and never read. Remove it, or use it as the default timeout_s of SandboxHandle.exec.
• backend/__init__.py: one extra blank line at the end of the file.

Verified

Item	Status at 36f81ca
B1 default path	fixed: default-template; None means unset; mappings win over template (M5)
B2 ExecResult, release on exit	fixed, test updated
B3 close() order	fixed: reap in try, backend in finally, re-raise; __exit__ keeps the original exception
M1 kill on client timeout	fixed
M2 warm claim	fixed: NotFoundError drops and continues, other errors give back and raise
M3 error types	fixed
M4 loop runner	fixed except the nit above
M6 README	fixed
M7 parallel reap and unwarm	fixed, semaphore 16
nits	_templates, suspend task, endpoint default, dead file helpers, stream pump cancel: fixed
hand-edited stub from #81	gone, _gen is identical to main

What I ran: uv run --extra dev pytest (118 passed); the probe above for the default and the explicit-template configurations.

[Tomer Glottmann (tomergee)]

Approving at 36f81ca. The three blockers and M1 to M7 from the first review are fixed, CI is green, 118 tests pass. The image pass-through goes to PR 2 of the SDK as agreed; the description refresh can follow there too.

[Sunny (sunnylovestiramisu)]

Updated PR description.