feat(coreutils): reproducible tar, gzip and zstd by raphaelvigee · Pull Request #442 · hephbuild/heph

raphaelvigee · 2026-08-29T22:04:34Z

A heph target runs its recipe with a sandbox PATH of the host's directories, so `cp` means GNU coreutils on Linux and a BSD userland on macOS. The sandbox isolates files; it does nothing about the two hosts disagreeing. `install -D` does not exist on macOS at all, `wc` pads its output so `[ "$(wc -l < f)" = 3 ]` is Linux-only, and `sort` collates by locale — which silently changes build *outputs*, not just exit codes. A build system whose contract is "same inputs, same outputs" cannot leave the tools that produce those outputs undeclared and host-defined. So heph ships its own. `crates/coreutils` compiles 40 MIT-licensed uutils/coreutils applets into the binary and reaches them by re-exec — `heph __coreutils <applet>`, or a symlink named after the applet (argv[0] dispatch, busybox style). Dispatch is the first thing in `main()`, before logging, clap, the self-update check or any runtime: a build may invoke `cp` thousands of times and each one is a fresh process. It runs ahead of the `__supervisor` and `__runner-exec` branches, so it is tested against those argv shapes — eating one would kill the sidecar with a broken pipe rather than an error anyone could read. The shims are one directory under the heph home, materialized once per (toolbox version, binary path) and contributed to `hexecrunner`'s `PathPolicy` as a tier directly behind the target's own tools. A recipe that provisions its own `sed` still gets that `sed` — the builtins only displace the host's. Per-sandbox cost is one extra PATH entry: nothing written per target, nothing staged, nothing to tear down. `plugin-exec` takes a `CoreutilsShims` closure and a version rather than depending on `crates/coreutils`. The driver needs a directory to put on PATH and a number to hash, not knowledge of what an applet is — and most of the workspace links `plugin-exec`, so the dependency would have dragged forty utility crates into `engine`, `e2e` and `plugingo-e2e` builds and test binaries (`cargo tree -p engine | grep -c uu_`: 45 before, 0 after). The closure also keeps materialization lazy, so a `heph query` never touches the filesystem for it. `coreutils: true` with no supplied shims is a hard error, not a shrug: running against the host's utilities while the cache key claims heph's is the silently-wrong-build case. Off by default (`coreutils: true` on the exec/bash driver). Turning it on changes what every recipe's `cp` resolves to, and it moves every exec target's cache key. Cache correctness. The utilities are on a target's PATH without being declared, and nothing can tell which of them a shell command will invoke without parsing it, so `COREUTILS_VERSION` goes into the def hash whole or not at all — bumping it invalidates every exec target in every workspace, which is release-gated, not routine. Nothing is hashed while the toolbox is off, so a workspace that never opts in keeps today's keys. Cost, measured on this tree (aarch64-apple-darwin, rustc 1.96, the real release profile): +7.55 MiB stripped, +19.3%. Trimming does not help — dropping the eight lowest-value applets saves 1.37 MiB of that, because the cost is a shared uucore+clap floor, not the applet count. Startup is unchanged: `heph --version` already costs ~6 ms, essentially all of it before `main`, and the applets add nothing to it. Verified end to end on darwin/arm64: with the toolbox on, a bash target resolves `cp` to `.heph3/coreutils/v1-<hash>/bin/cp` and reports `cp (uutils coreutils) 0.10.0`; with it off the same target gets `/bin/cp`, which rejects `--version`. `heph tool coreutils list | which <name> | run <name> …` is the diagnostic surface — shadowing `cp` silently is exactly the kind of magic that produces an unanswerable bug report. Not in this change: sed, grep, find, xargs, tar, gzip and the template renderer; removing the host directories from the sandbox PATH; and making the toolbox the default. Design and measurements in docs/COREUTILS.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

Filling in a config file is what `sed -i` and `envsubst` get used for, and both put the substitution in a shell — where it depends on the host's `sed`, on quoting, and on whatever else is in scope. The builtin coreutils fix the first of those; this removes the shell from the job entirely. `template(name, src, out, vars)` renders a declared template file into a declared output, in-process. `src` is a hashed input, so editing the template rebuilds everything downstream. Two properties are load-bearing rather than incidental: A template cannot read an undeclared file. The minijinja environment is built with no loader, so `{% include %}` and `{% import %}` have nothing to resolve against. A template that could read an undeclared file would be a hole in the sandbox, not a feature. An undefined variable is an error that names itself. Undefined behaviour is strict, and the variables a template references are checked against `vars` *before* rendering — because minijinja's own message is "undefined value (in template:1)", which says something is missing without saying what. Instead: "template uses variables that `vars` does not supply: prot, tls. Supplied: features, host, port". The check compares the root of a dotted path, since minijinja reports `{{ cfg.port }}` as the undeclared name `cfg.port` and comparing the whole path would reject every template that reads a field; loop-bound names are not reported. `vars` are held in a `BTreeMap` before hashing. They arrive as a `HashMap`, whose iteration order is randomized per process, so folding them in that order would give the same target a different def hash on every run and never hit cache. `TEMPLATE_FORMAT_VERSION` is in the key too, for the reason the exec driver hashes its own format version: the rendered bytes are a function of the renderer as well as the template. `src` must produce exactly one file. Several have no defensible answer — picking the first would depend on walk order — so it is refused with the count, and none names the address that was supposed to produce it. Verified end to end: a `template` target renders `listen = 0.0.0.0:8080` from a globbed `.j2` source and a downstream bash target reads it as `$SRC`; a typo'd variable fails the target naming `prot` and listing what was supplied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

Three of the largest remaining divergences. `grep -P` does not exist on macOS; `find`'s `-printf`, `-regextype`, `-delete` and `-newermt` are all GNU-only and BSD `find` requires an explicit path; and `xargs -r` is GNU-only while the no-input default is inverted between the two. `find` and `xargs` are adapters over uutils/findutils. Its entry points take `&[&str]`, so a non-UTF-8 argument is refused by name rather than lossily converted: GNU `find` accepts arbitrary bytes in a path, this cannot, and searching a *different* path than the one asked for is worse than saying so. `grep` is a POSIX front-end over `grep-searcher`/`grep-regex`, the engine ripgrep uses, covering -EFivnclLqwxrhHs, -m, -e, -f and `--`. Two deliberate departures from GNU: - No -P. The `regex` crate has no backreferences or lookaround by design, so -P fails with that explanation rather than with "invalid option" — a pattern needing it has to be rewritten, not retried. - Never colourised. Output goes into build logs and gets parsed; a `--color=auto` that guessed from a tty would make a recipe's behaviour depend on how it was invoked. The argument parser is hand-rolled rather than clap because `grep -e -v file` must treat `-v` as the pattern, which a declarative parser fights. Line numbers are always counted and only printed under -n: the `UTF8` sink asks every match for its line number and fails with "line numbers not enabled" if the searcher was not tracking them — found by running it. Not planned: `diff`/`cmp`. The `diffutils` crate exposes its algorithms but keeps its CLI in a private `main`, so wiring it up would mean reimplementing its argument parsing for the lowest-value pair in the set. Verified against the host's GNU tools: `find . -name '*.txt'` and `grep -c`/`grep -v` produce byte-identical output, `find -printf` works (GNU-only), and `find | xargs` pipelines. `heph tool coreutils list` no longer errors when piped into `head`. COREUTILS_VERSION 1 -> 2: the applet set changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

The applets where the divergence is not a flag but the bytes. GNU tar and bsdtar disagree about whether --transform, --sort, --owner and --mtime exist at all; gzip writes the source filename and its mtime into the header unless told not to; zstd is installed by default on neither host. Archiving the same tree twice, on two machines, should produce the same bytes. With the host tools it does not — gzipping identical content a second later gives a different file, which was reproduced while writing this. So the reproducible settings are the defaults, not flags anyone has to remember: entries sorted by path, uid/gid 0 with empty owner *names*, mode normalised to the executable bit (the rest is umask), mtime from SOURCE_DATE_EPOCH or 0, and no gzip header name or timestamp. There is deliberately no flag to turn any of that off. A recipe that wants a non-reproducible archive is a recipe with a bug. Compression is detected from the magic bytes rather than the file name, so a `.tar` that is actually gzipped extracts instead of failing confusingly. Verified from the CLI: two identical trees tar to the same sha256, and identical content gzips to the same sha256 — where the host's gzip produced two different files for the same input. COREUTILS_VERSION 2 -> 3: the applet set changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0181d7hhbYWXT42Z1KQPM29Q

raphaelvigee · 2026-09-03T17:32:50Z

Superseded by consolidation. The applet halves of this PR (#442 tar/gzip/zstd, #443 sed, #444's tmpl) now land in two PRs instead of four:

feat(coreutils): the rest of the toolbox — grep, find, xargs, sed, tar, gzip, zstd #453 — grep, find, xargs, sed, tar, gzip, zstd. They touched the same four files and asked the same review question three times.
feat(template): a declared template rule, and the tmpl applet on the same renderer #440 — the template rule and the tmpl applet together, since they are one feature: a shared renderer used two ways.

No content was dropped — the restacked tree is byte-identical to the eight-commit version, and every layer builds, lints and passes its unit tests on its own. Fewer layers also matters now that stacked PRs get no CI unless labelled (#449).

raphaelvigee force-pushed the raphaelvigee/coreutils-archives branch from 04df5d1 to 1f8ae6f Compare August 29, 2026 22:44

raphaelvigee force-pushed the raphaelvigee/coreutils-find-grep branch from e0fc361 to 731246b Compare September 3, 2026 16:26

raphaelvigee force-pushed the raphaelvigee/coreutils-archives branch from 1f8ae6f to 5fe0785 Compare September 3, 2026 16:26

raphaelvigee and others added 4 commits September 3, 2026 18:55

raphaelvigee force-pushed the raphaelvigee/coreutils-find-grep branch from 731246b to a90f972 Compare September 3, 2026 16:56

raphaelvigee force-pushed the raphaelvigee/coreutils-archives branch from 5fe0785 to 596509f Compare September 3, 2026 16:56

raphaelvigee force-pushed the raphaelvigee/coreutils-find-grep branch from a90f972 to 2f656af Compare September 3, 2026 17:30

Base automatically changed from raphaelvigee/coreutils-find-grep to raphaelvigee/coreutils-template-driver September 3, 2026 17:30

raphaelvigee mentioned this pull request Sep 3, 2026

feat(coreutils): the rest of the toolbox — grep, find, xargs, sed, tar, gzip, zstd #453

Open

raphaelvigee closed this Sep 3, 2026

This was referenced Sep 3, 2026

feat(coreutils): add sed #443

Closed

feat(coreutils): add the tmpl applet, on a shared renderer #444

Closed

Navigation Menu

feat(coreutils): reproducible tar, gzip and zstd - #442

feat(coreutils): reproducible tar, gzip and zstd#442
raphaelvigee wants to merge 4 commits into
raphaelvigee/coreutils-template-driverfrom
raphaelvigee/coreutils-archives

raphaelvigee commented Aug 29, 2026

Uh oh!

raphaelvigee commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

raphaelvigee commented Aug 29, 2026

Uh oh!

raphaelvigee commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant