feat(engine): deferred values — a driver option read from another target

[raphaelvigee]

Implements the Deferred Values design, P1. Stacked on #455 — the credential driver's presentation templates are one of the two consumers.

A driver option's value has to be a literal in the BUILD file. But the value's owner is usually somewhere else — a role ARN in the Terraform that created the role, a registry per environment in a platform team's repo, a version in a release process. Copying it in makes a second source of truth that someone has to keep agreeing with the first, purely to satisfy heph.

# //infra/aws/BUILD — the value's owner
target(
    name = "role-arn",
    driver = "bash",
    deps = {"tf": ["//infra/aws:*.tf", "//infra/aws:terraform.tfstate"]},
    run = "terraform -chdir=$(dirname $SRC_TF) output -raw deployer_role_arn > $OUT",
    out = ["role_arn.txt"],
)

# //auth/BUILD — where the literal used to be
target(
    name = "aws",
    driver = "credential",
    sources = [heph.auth.oidc("github_actions", audience = WIF,
                              present = heph.auth.aws_web_identity(
                                  role = "${read://infra/aws:role-arn}"))],
)

The ARN appears exactly once in the repository, in the Terraform that defines it. heph reads it; it does not hold a copy.

What no other build system can express

Gradle, Bazel, Buck2, Pants and Nix each shipped an answer, and they all made the same structural choice: the value crosses the boundary as a bare string, and the thing that produced it never becomes a node. Three consequences follow mechanically — the producer must re-run eagerly on every build (which is why every one of them documents some version of "only use fast commands here"), the cache key can only hash the string and over-invalidate or hide it and sanction staleness, and "why did this rebuild?" bottoms out at the string. Each ecosystem then converged on the same workaround — commit a snapshot of the computed value — and every snapshot is a second source of truth that drifts, undetectably, for exactly the reason the snapshot was needed.

Here the producer is a node.

LOAD                      PARSE                        RUN
the author writes         an edge is synthesized       the bytes are substituted
role = "${read://…}"      Input{hashed:true,           role = "arn:aws:iam::…"
                                runtime:false}

WHAT EACH HASH SEES
                          def hash                     hashin
                          the reference, verbatim      the producer's hashout
                          never the resolved value     via the ordinary dep path

The def hash covers the unresolved reference, which is what keeps heph query and heph inspect def from triggering a build. The producer's content still reaches hashin, because hashin is the def hash plus the hashouts of hashed inputs. No new hashing rule is introduced at any point — the correct behaviour is a structural consequence of where the work happens. Both halves already existed: synthesizing an edge that is not in the BUILD deps list is what runner does, and a producer's content reaching a consumer's key is what inputs_result_meta does for every hashed input.

example/deferred/ demonstrates it directly — the producer declares an input it does not use, so editing that input re-runs the producer and moves nothing, while editing the one that defines the value moves the consumer's hashin.

Three layers, and a driver touches only the third

An earlier shape left three obligations on the plugin author: type the field, collect its edges at parse, substitute at run. Both "remembers" fail silently — a missed edge means the producer never builds and the value never enters the key; a missed substitution means "${read://infra:role-arn}" is used as the ARN. That is the exact bug class this exists to remove, reintroduced one layer down.

host	walks TargetSpec.config for references at any nesting depth, appends Inputs to def.inputs after parse, reads the producers, fills RunRequest.deferred. Cannot be forgotten; the driver never participates.
shared decoder	String::from_spec_value refuses ${read:…} — so a reference in out, deps, name or a glob is a parse error in every driver at once, with no driver code and no schema flag.
driver	one field type changes: role: String → role: Deferred<String>. Reading it requires a &RunRequest, so forgetting to resolve is a compile error.

The entire diff to make a field deferrable:

 #[derive(Spec)]
 struct MySpec {
-    role: String,
+    role: Deferred<String>,
 }

accepts_deferred is derived from the field types, so there is nothing to remember there either. The walk is over the raw Value tree, which is what makes it work where a flat per-field schema could not: a credential's sources = [heph.auth.oidc(…)] is a list of maps of maps, and the reference sits three levels inside it.

The refusal

A deferred value changes what a target does. It never changes which targets exist.

Rejected in deps, tools, runner, scratch, out, name and labels. That buys three things: cycle detection still works at parse, because every edge is known before anything runs; heph query and heph inspect def never trigger a build; and evaluation never blocks on one. That last is the whole of Nix's import-from-derivation problem, which nixpkgs forbids outright — what is here is IFD with the recursion cut off at one level.

A runner spec is never deferrable, as a standing rule: a runner target's fingerprint is what moves every consumer's cache key when the environment moves.

What ships, and what does not

Ships: ${read://…}, the host-side walk and edge append, resolution from the store, RunRequest.deferred, Deferred<String>, the //-address claiming rule, the whole-driver schema gate. Consumers: the credential driver's presentation templates, and the exec driver's run in both exec and bash mode.

The claiming rule, and why bash is in after all

An earlier draft excluded bash on one argument: ${src:0:3} is valid bash, so accepting references there would create a collision an escape rule could only document. The rule below removes the collision instead — heph takes a ${…} only when its argument is an absolute address. Verified against a real shell rather than reasoned about:

written	bash today	heph
${src::3}	abc — first three chars of $src	not claimed
${src:0:3}	abc — same, explicit offset	not claimed
${FOO:-default}	default	not claimed
${src://x:y}	arithmetic error, or "" unset	claimed
tmp.$$	tmp.12345	untouched — $$ is not ${

So the forms heph claims are, in bash, either an error or the empty string; nobody writes them on purpose. The :name and ./ relative forms are deliberately not claimed — ${src::3} is the exact text of a real bash idiom, and no diagnostic is worth taking it over. One predicate (template::claims) answers "is this heph's?" for the spec decoder, the engine walk and the substitution alike.

The consequence is stated rather than buried: heph substitutes, it does not quote. In exec mode a value fills one argv element; in bash it is spliced into a shell program, so a producer emitting 1.2.3; rm -rf / runs it — and those bytes may have been pulled from the shared remote cache. A code trust boundary, not just a data one.

Does not ship, ever: ${env:NAME}. It is legal bash (${var:offset}); getting its value into the def hash requires substituting before parse, which is before the decoder that refuses a reference in out, deps, runner and name; and it reaches a credential declaration, whose inputs are hashed: false by design. All three were board blockers. The value it would have carried composes from a pass_env producer plus ${read:}, which keeps the variable's name in the key, states the freshness choice with cache = False, and shows up in inspect deps. Recorded in the doc so it is not proposed again.

Does not ship yet: heph.core.read() as a Starlark function; inspect deps "via" lines and inspect def --resolved; and the OCI family's build_args, dest, labels, cache_from. (${src://…} lands in #458, stacked on this.)

Also in this PR, from a second review pass

• A measured warm-path regression, fixed. deferred_refs asked every driver for its schema before checking whether the config held a ${ at all — and #[derive(Spec)] rebuilds Vec<DriverField> on each call: 82 allocations, 7212 bytes, ~1.6 µs per target, on every build including full cache hits, in workspaces using none of this. The pre-scan now comes first. A driver whose schema() counts calls pins it, because nothing else can observe the difference.
• A nested ${ is refused. ${FOO:-${read://a:b}} closed on the inner }, so the outer form tokenized as an unknown kind, was reproduced verbatim, and the inner reference silently never resolved.
• ${read:} trims surrounding whitespace rather than exactly one trailing newline — \r\n from a Windows-committed file, a trailing space from a yq pipeline, leading spaces from an indented heredoc. An all-whitespace output is still empty, and still an error.
• Presentation templates are now BTreeMap<String, Deferred<String>> — the entire diff to make a field deferrable, as advertised — which deletes the private reject_reference/raw_string/template_map/str_map_inner machinery this branch had added to the credential driver.

Review board

hermeticity — NOT HERMETIC, fixed. A deferred value inside a credential declaration reached no consumer's key: a credential target's get_def is never called on a build path, so an edge appended to the credential's own def was never hashed by anyone, and a terraform apply that moved the ARN produced a silent cache hit. The synthesized edges for a resolved credential now land on the consumer's def, labelled with the credential's address.

code-quality — BLOCKED on two, both fixed. Resolution matched a producer by ends_with(":name"), so ${read::v} written in package zzz resolved to whichever key happened to end :v — //aaa:v, in sort order — and handed the author a different producer's bytes with no diagnostic; it now normalizes against the declaring package, the same way the walk keyed the map. And a credential's non-presentation fields (audience, runner, a rename selector) accepted a reference and used it unsubstituted.

feature-quality — BLOCKED on the escape, fixed. $ meant two things: a credential presentation is a document heph writes and owns $$ for, while a driver option is somebody else's shell text where echo tmp.$$ is the PID idiom. Folding $$ into literal text made a field whose meaning changed depending on whether a ${read:…} appeared elsewhere in it. The tokenizer now keeps Piece::Escape distinct and the two consumers use different substituters. It also caught that reserving src and env outright broke ${src:0:3} and ${env:FOO} in every run; the reservation is now read only, and src/env say "not yet" only inside a driver that accepts references.

compatibility — COMPATIBLE AFTER BUMP. ABI_SEMVER 0.9.0 → 0.10.0. RunRequest/ManagedRunRequest gained map<string, string> deferred; Schema gained bool accepts_deferred. Both additive and cold-path. The bump exists for the schema field: an old plugin's String decoder rejects nothing, so it would run the target with the reference text as the value — accepts_deferred decodes as false, so the host refuses at parse instead, naming the driver. Deferred<T>'s Hash delegates to the raw text, so retyping an existing field leaves its def hash byte-identical; without that, every target of that driver invalidates on upgrade and a mixed fleet double-populates the shared remote.

Tests

22 engine e2e tests (crates/e2e/tests/deferred.rs) covering the two hashes, the refusals, the cycle, the failure modes and the size cap, plus unit tests for the tokenizer, the walk, the derive and the ABI conversion. docs/DEFERRED_VALUES.md has the full design; example/deferred/ is a worked package. Full tst passes; the only failure is the pre-existing test_real_docker_default_builder_without_exporters_is_diagnosable, which needs a running Docker daemon.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jr413rT5LPhzVYo6QFckDm

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}