diff --git a/plugins/heph-expert/skills/heph/references/concepts.md b/plugins/heph-expert/skills/heph/references/concepts.md index a4599ac..021c655 100644 --- a/plugins/heph-expert/skills/heph/references/concepts.md +++ b/plugins/heph-expert/skills/heph/references/concepts.md @@ -8,6 +8,7 @@ The mental model behind heph. Source pages under - [Dependencies](#dependencies) - [Sandbox](#sandbox) - [Caching](#caching) +- [Scratch caches](#scratch-caches) - [Reproducibility](#reproducibility) - [Codegen](#codegen) - [Runners](#runners) @@ -119,6 +120,59 @@ are trimmed automatically at the end of the run that writes the new one — no action needed. `heph tool gc` sweeps everything else no longer reachable from any current target (removed targets, orphaned entries). +## Scratch caches + +A **scratch** is a directory a target declares, keeps between runs, and +shares with every target that references it — the one thing in a sandbox +that is neither an input nor an output. It's for a tool that already +maintains its own content-addressed cache (a compiler cache, a package +download cache, a registry blob store), never for durable state. + +The contract: **a target's outputs must be identical whether its scratch is +warm, cold, or absent.** It never enters `hashin`, so it can never invalidate +anything — changing its settings rebuilds nothing, deleting it costs only +time. Prove the contract with `heph --no-scratch inspect hashout ` and +compare against the ordinary hash. + +```python title="BUILD" +target( + name = "gocache", + driver = "scratch", + path = ".cache/go-build", # optional; omit for env-var-only + env = "GOCACHE", # defaults to SCRATCH_ + access = "shared", # "exclusive" (default) | "shared" + version = "", # what the contents depend on, beyond the addr + remote = False, # may travel through the remote cache +) + +target(name = "build", driver = "bash", scratch = ["//build:gocache"], ...) +``` + +| Field | Meaning | +|---|---| +| `path` | Mount point inside a consumer's sandbox, relative to its cwd. Optional — omitting it means nothing is placed in the tree, so no output can collect it and no dependency can be shadowed by it. | +| `env` | Env var a consumer reads the absolute path from. Point it at the tool's own variable (`GOCACHE`, `CCACHE_DIR`). | +| `access` | `"exclusive"` (default, one consumer at a time, cross-process) or `"shared"` (concurrent — only for a cache safe under concurrent access *by construction*). | +| `version` | The whole identity beyond the address. heph never guesses; state what the cache depends on (`heph.core.os() + "/" + heph.core.arch()`, a toolchain version, …) or leave empty for a portable cache. | +| `remote` | May be pulled/pushed through a configured remote cache. | +| `max_size` | Size cap, e.g. `"10GiB"`; over it the cache is dropped whole rather than trimmed. | + +`heph tool scratch ls / head / path / rm |--all / push +--all / pull --all` inspect, reclaim, and publish caches. `head` explains +which lineage a build would restore from and why — the go-to when a branch +starts unexpectedly cold. `.hephconfig`'s `scratch.scope` / +`scratch.restoreScopes` control per-branch lineages (`${git:branch}` resolves +automatically); `heph tool gc --scratch-max-size` / `--scratch-max-age-days` +sweeps by size and age. + +A mounted scratch (`path` set) must not sit under a broad output glob in the +same package — `out = "**/*"` beside a mount fails at pack time, naming the +mount. The env-var-only form (no `path`) has no such hazard. + +Full reference: . The +`heph-go` plugin covers the Go provider's own `GOCACHE`/`GOMODCACHE` scratch +caches and `heph.go.gocache_addr()`. + ## Reproducibility The central promise is **byte-identical outputs**: same inputs ⇒ same artifact, diff --git a/plugins/heph-expert/skills/heph/references/configuration.md b/plugins/heph-expert/skills/heph/references/configuration.md index 26a1e10..cf90955 100644 --- a/plugins/heph-expert/skills/heph/references/configuration.md +++ b/plugins/heph-expert/skills/heph/references/configuration.md @@ -20,6 +20,7 @@ plugins: | `plugins` | list of plugin entries | `[]` | Plugins to register. Each entry sets exactly one of `builtin`, `path`, or `url`, plus an optional `options` map. | | `homeDir` | path | unset | Where heph keeps its home and cache. | | `memCache` | `{perEntryBytes, capacityBytes}` | unset | In-memory cache sizing. | +| `scratch` | `{scope, restoreScopes, seedOnFork}` | unset | Which scratch-cache lineage a run reads and writes. | | `fuse` | `{enabled: true \| false \| "auto"}` | off | Sandbox overlay mode. | | `lock` | `{backend: fs \| mem}` | `fs` | Execute-phase lock backend. | @@ -66,6 +67,21 @@ memCache: capacityBytes: 67108864 # total budget; 0 disables the in-memory cache ``` +## `scratch` — branch-lineage policy + +Which lineage of a [scratch cache](./concepts.md#scratch-caches) a run reads +and writes — the cache's *contents* are declared by its target; this is +which copy of them a run sees. + +```yaml title=".hephconfig" +scratch: + scope: "${git:branch}" # lineage this run writes to; resolves the current branch + restoreScopes: ["master"] # lineages to read from when scope has nothing yet + seedOnFork: true # default: copy the first warm fallback into a cold scope +``` + +`restoreScopes` is read-only — a branch never writes into its fallback. + ## `fuse` — sandbox overlay Opt-in: assemble sandboxes with a FUSE overlay instead of copying inputs. diff --git a/plugins/heph-go/skills/heph-go/references/go-plugin.md b/plugins/heph-go/skills/heph-go/references/go-plugin.md index 8bda8c3..79d058a 100644 --- a/plugins/heph-go/skills/heph-go/references/go-plugin.md +++ b/plugins/heph-go/skills/heph-go/references/go-plugin.md @@ -187,6 +187,34 @@ options: cctool: "//@heph/bin:cc" # default; point elsewhere for a hermetic compiler ``` +## Build and module caches + +`GOCACHE` (one per Go module + build variant) and `GOMODCACHE` (one shared, +portable cache for downloaded modules) are wired up automatically as +[scratch caches](https://hephbuild.github.io/docs/concepts/scratch) — +nothing to configure. A host-set `GOMODCACHE` is deliberately not passed +through, so the shared cache is what every build sees. Both show up in +`heph tool scratch ls`, are droppable with `heph tool scratch rm` if one +goes bad, and are auditable with `heph run --no-scratch`. + +Share the same `GOCACHE` a module's own targets use from a hand-written +target with `heph.go.gocache_addr()`: + +```python title="BUILD" +target( + name = "custom_build", + driver = "bash", + scratch = [heph.go.gocache_addr()], # goos, goarch, gotool, tags, goexperiment, + run = "go build -o $OUT ./cmd/tool", # gcflags, ldflags, race — match the variant + out = "tool", +) +``` + +`gocache_addr()` resolves the module from the *calling* BUILD file's nearest +`go.mod`. Pass the same factors (`goos`/`goarch`/`gotool`/`tags`/…) the +variant you're building against declares, or the target warms a different +cache than the one it meant to share. + ## Linting and formatting A Go module gets `lint-check`, `lint`, `format-check`, and `format` targets the diff --git a/website/docs/concepts/scratch.md b/website/docs/concepts/scratch.md new file mode 100644 index 0000000..b4576a4 --- /dev/null +++ b/website/docs/concepts/scratch.md @@ -0,0 +1,185 @@ +--- +title: "Scratch caches" +sidebar_position: 7 +description: A named, mutable cache directory a target keeps between runs — for a compiler cache, a download cache, or any tool that maintains its own cache. +--- + +# Scratch caches + +A **scratch** is a directory a target declares, keeps between runs, and +shares with every other target that references it. It is the one thing +inside a [sandbox](/docs/concepts/sandbox) that is neither an input nor an +output — mutable, and never hashed. + +## The contract + +> A target's outputs must be identical whether its scratch directories are +> warm, cold, or absent. Losing one is always a slowdown, never a wrong +> answer. + +heph gives every other part of a target's inputs and outputs the hermetic +treatment: declared in, declared out, sandbox thrown away. A scratch is the +deliberate exception, for a tool that already maintains its own +content-addressed cache — a compiler cache, a package download cache, a +registry blob store. Because it never enters the +[input hash](/docs/concepts/caching), it can never invalidate anything: +changing a scratch's settings rebuilds nothing, and deleting one costs time +and nothing else. + +Check the contract on your own target: + +```bash title="terminal" +heph inspect hashout //build:compile +heph --no-scratch inspect hashout //build:compile +``` + +Same hash both times, or the target depends on carried-over state and is +broken. + +## Declaring one + +A scratch is declared like any other target, with `driver = "scratch"`. It +builds nothing — it only describes a cache. + +```python title="BUILD" +target( + name = "gocache", + driver = "scratch", + path = ".cache/go-build", # optional; omit for env-var-only + env = "GOCACHE", # defaults to SCRATCH_ + access = "shared", # "exclusive" (default) | "shared" + version = "", # what the contents depend on, beyond the addr + remote = False, # may travel through the remote cache +) + +target( + name = "build", + driver = "bash", + scratch = ["//build:gocache"], + run = "go build -o $OUT ./...", + out = "bin", +) +``` + +| Field | Type | Default | Meaning | +|---|---|---|---| +| `path` | `string` | unset | Where the directory mounts inside a consuming target's sandbox, relative to its cwd. Optional — most tools find their cache through an environment variable, and omitting `path` is the safer shape: with nothing mounted, no output can collect the directory and no dependency can be shadowed by it. | +| `env` | `string` | `SCRATCH_` | Environment variable a consumer reads the directory's absolute path from. Point it at the tool's own variable (`GOCACHE`, `CCACHE_DIR`) so nothing else needs wiring. | +| `access` | `string` | `"exclusive"` | `"exclusive"` — one consumer at a time, enforced across separate `heph` processes too; the safe default for a tool with no stated concurrency story. `"shared"` — concurrent consumers allowed, only for a cache that's safe under concurrent access *by construction* (content-addressed and self-verifying, the way Go's build cache is). | +| `version` | `string` | `""` | Everything the contents depend on, beyond the address — the whole of it. Two declarations at the same address share a directory if and only if they also agree on `version`. heph never guesses: state what the cache depends on, or leave it empty for a cache that's portable everywhere. See [Portability with `version`](#portability-with-version). | +| `remote` | `bool` | `false` | Whether this cache may be pulled from a remote cache automatically, and published to it with `heph tool scratch push`. See [Sharing through the remote cache](#sharing-through-the-remote-cache). | +| `max_size` | `string` | unset | A size cap, e.g. `"10GiB"`. Past it, the whole cache is dropped and starts again — heph can't tell which of a foreign tool's entries are hot, so it doesn't try to trim. | + +Reference a declaration from a consuming target with `scratch = [...]` — a +plain list of addresses, the same shape as `deps`. A target may reference +several; every target referencing the same declaration gets the same +directory. + +## Choosing `access` + +`access` is an assertion about the tool, not a wish — get it wrong and +`"shared"` corrupts the cache, or `"exclusive"` serializes work that didn't +need to be. Go's build cache and module cache are safe under concurrent +writers because they're content-addressed: an entry either matches its key +or is not used, the same property `go build -p N` already relies on. Most +tools don't document that, so `"exclusive"` — one consumer at a time, +enforced with a lock held for the whole run — is the default. + +## Portability with `version` + +`version` is opaque to heph — it isn't parsed, and heph contributes nothing +of its own to it (no host OS, no architecture) unless the BUILD file puts it +there: + +```python title="BUILD" +version = heph.core.os() + "/" + heph.core.arch() # host-specific +version = goos + "/" + goarch + "/" + go_version # target-specific +version = "" # portable (the default) +``` + +The default is empty, which is the *less* safe direction, deliberately — a +narrow default that guesses wrong (keying on the host for contents that +actually depend on the target) is worse than an obviously-too-broad one. +Changing `version` yields a fresh, empty cache without touching anything +else. + +## Auditing with `--no-scratch` + +`--no-scratch` runs against a throwaway, empty cache instead of the stored +one — proving the contract rather than assuming it. It implies `--force`: +since a scratch never enters the cache key, a cache hit would otherwise just +replay the answer the audit exists to check. The stored cache itself is +never touched. + +```bash title="terminal" +heph run --no-scratch //build:compile +``` + +A target that reads an unset scratch variable under `set -u` fails outright +rather than running cold — write `${GOCACHE:-}` if the audit should report +on the target instead of on the shell. + +## Sharing through the remote cache + +With a [remote cache](/docs/reference/configuration#caches--remote-shared-caches) +configured, `remote = True` lets a scratch's contents travel between +machines. A build pulls automatically on a cold cache; nothing is ever +published as a side effect of building — that's always an explicit step, so +CI runs it last: + +```bash title="terminal" +heph tool scratch push --all --producer "$CI_RUN_ID" # publish every remote = True cache +heph tool scratch pull --all # warm a machine ahead of time +``` + +Which lineage a run reads and writes is controlled by `scratch.scope` in +`.hephconfig` — see +[`scratch` — branch-lineage policy](/docs/reference/configuration#scratch--branch-lineage-policy). +By default every run shares one lineage; scope it per branch so work on one +branch doesn't overwrite another's cache, and a fresh branch starts from its +base instead of from nothing. + +## Inspecting and reclaiming + +```bash title="terminal" +heph tool scratch ls # every cache: address, access, size, lineages present +heph tool scratch head //build:gocache # why a build would be warm or cold +heph tool scratch path //build:gocache # the on-disk directory +heph tool scratch rm //build:gocache # drop one; always safe +heph tool scratch rm --all # drop every scratch cache +``` + +`heph tool scratch head` is the one worth remembering: it prints every +candidate lineage a build would consult, local and remote, in order, and +which one wins — the answer to "why did my branch start cold?", a question +the directory itself can't answer since the interesting part is what +*wasn't* found. + +Scratch caches also come under `heph tool gc`: + +```bash title="terminal" +heph tool gc --scratch-max-size 50GiB --scratch-max-age-days 30 +``` + +## A broad output beside a mounted scratch + +A target's own outputs must never come from inside its scratch — that would +make the artifact's bytes depend on unhashed, mutable state. Give a mounted +scratch (one with `path` set) a narrow `out` pattern in the same directory: +a glob broad enough to reach the mount (`out = "**/*"`, say) fails when heph +packs the result, naming the mount path in the error. + +An unmounted scratch (no `path`, the env-var-only form) sidesteps this +entirely — there's nothing in the sandbox tree for an output to reach, which +is why it's the shape to reach for first when a tool is happy to be told its +cache directory through an environment variable alone. + +## Who uses one already + +The [Go plugin](/docs/plugins/go#build-and-module-caches) shares one +`GOCACHE` per module and per build variant, and one portable `GOMODCACHE` +for downloaded modules, both without any configuration. + +The [OCI plugin](/docs/plugins/oci#oci_pull)'s `oci_pull` shares one +registry blob store across every pull in the workspace — two images sharing +a base layer download it once, not once per pull. diff --git a/website/docs/plugins/go.md b/website/docs/plugins/go.md index 5034552..a09263e 100644 --- a/website/docs/plugins/go.md +++ b/website/docs/plugins/go.md @@ -245,6 +245,45 @@ options: `cctool` is resolved only when a race build that needs cgo actually runs — an ordinary build, and a darwin race build, never touch it. +## Build and module caches + +Every Go list, compile, and third-party download step shares two caches +automatically — nothing to configure: + +- **`GOCACHE`** — one per Go module (the directory holding `go.mod`) and per + [build variant](#build-variants), shared by that module's `go list` and + `go tool compile` steps alike. +- **`GOMODCACHE`** — one shared, portable module-download cache for the + whole workspace. heph owns it outright: a host-set `GOMODCACHE` is not + passed through, so the shared cache is what every build sees. + +Both are [scratch caches](/docs/concepts/scratch) — visible in +`heph tool scratch ls`, droppable with `heph tool scratch rm` if one goes +bad, and auditable with `heph run --no-scratch`. Point a hand-written target +at the same `GOCACHE` a module's own targets already share with +`heph.go.gocache_addr()`: + +```python title="BUILD" +target( + name = "custom_build", + driver = "bash", + scratch = [heph.go.gocache_addr()], + run = "go build -o $OUT ./cmd/tool", + out = "tool", +) +``` + +| Argument | Default | Meaning | +|----------|---------|---------| +| `goos` / `goarch` | this machine's | Target platform of the cache to share. | +| `gotool` | `"host"` | Toolchain selector — `"host"`, a pinned version, or a target address. Match whatever the variant you're building against declares. | +| `tags` / `goexperiment` / `gcflags` / `ldflags` | `[]` | The rest of the variant's factors — pass what the variant uses, or the target warms a different cache than the one it meant to share. | +| `race` | `false` | Address the race-detector variant of the cache. | + +`gocache_addr()` resolves the module from the *calling* BUILD file's nearest +`go.mod`, so a target gets the same cache the Go driver's own targets in +that module already use. + ## Linting and formatting A Go module gets four extra targets the moment it has a `.golangci.yml` (or diff --git a/website/docs/plugins/oci.md b/website/docs/plugins/oci.md index 42f6aba..7b55bf0 100644 --- a/website/docs/plugins/oci.md +++ b/website/docs/plugins/oci.md @@ -341,6 +341,20 @@ WARN oci_pull: "alpine:3.20" currently resolves to See [Reproducibility](/docs/concepts/reproducibility) for the same principle applied elsewhere in heph. +#### Shared blob cache + +Every `oci_pull` target draws downloaded registry blobs from one shared +[scratch cache](/docs/concepts/scratch), keyed by nothing but its own +address — blobs are content-addressed by digest, so one store holds `amd64` +and `arm64` layers side by side. Two images sharing a base layer download it +once, not once per pull target. Nothing to configure; inspect or reclaim it +like any other scratch: + +```bash title="terminal" +heph tool scratch ls # the blob store shows up alongside your own caches +heph tool scratch rm --all # drop it if it ever needs a clean slate +``` + ### `oci_push` Pushes an image archive to a registry. An **action**, not an artifact — it diff --git a/website/docs/reference/configuration.md b/website/docs/reference/configuration.md index 1eda6e0..915e175 100644 --- a/website/docs/reference/configuration.md +++ b/website/docs/reference/configuration.md @@ -286,6 +286,29 @@ Run `heph tool cache measure-latency` to force a fresh latency measurement and print per-cache round-trip times. See the [Remote cache guide](/docs/guides/remote-cache) for a complete walkthrough. +## `scratch` — branch-lineage policy + +Controls which [scratch cache](/docs/concepts/scratch) lineage a run reads +and writes. A scratch's *contents* are declared by its target; this is about +which **copy** of them a given run sees. + +```yaml title=".hephconfig" +scratch: + scope: "${git:branch}" + restoreScopes: ["master"] + seedOnFork: true # default +``` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `scope` | string | `""` (one shared lineage) | The lineage this run **writes** to. `${git:branch}` resolves to the current branch, so a developer gets per-branch caches without wiring anything; CI overrides it with whatever names the branch there. | +| `restoreScopes` | string[] | `[]` | Lineages this run may **read** from when its own scope has nothing yet, tried in order — `["master"]` is the common case. Never written to: a branch can read its base's cache but never overwrite it. | +| `seedOnFork` | bool | `true` | Copy the first warm fallback lineage into a cold scope on its first build there, instead of starting from nothing. Turn off only for a large cache on a filesystem without reflink support, where the copy itself gets expensive. | + +See [Scratch caches](/docs/concepts/scratch) for the caches themselves, and +`heph run --no-scratch` for auditing that a target doesn't secretly depend on +one. + ## `fuse` — sandbox overlay Controls whether sandboxes are assembled with a FUSE overlay instead of copying diff --git a/website/sidebars.ts b/website/sidebars.ts index 7bcf2d6..0ab6361 100644 --- a/website/sidebars.ts +++ b/website/sidebars.ts @@ -22,6 +22,7 @@ const sidebars: SidebarsConfig = { 'concepts/dependencies', 'concepts/reproducibility', 'concepts/caching', + 'concepts/scratch', 'concepts/sandbox', 'concepts/codegen', 'concepts/approval',