Skip to content

pins: goldens move with their pins, and the per-site gaps that kept bot PRs red - #328

Merged
88plug merged 5 commits into
mainfrom
claude/bump-goldens-fix
Nov 7, 2026
Merged

88plug merged 5 commits into
mainfrom
claude/bump-goldens-fix

Conversation

@88plug

@88plug 88plug commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Most bot PRs (#315, #318, #320, #321, #322, #324, #325, #326) were red because the bumper's golden substitution left test files at the old pin, or because a test hardcoded a value a bump moves. This PR fixes the substitution rule and the per-site gaps. No historical record, ledger row, bench result or README history row changes, and no held pin moves.

Root cause

hack/pinlib/goldens.py live_pairs dropped every old pin string that any scanned file outside tests/ still contained. Leftover prose was enough to block a pair:

tests/bump_test.sh reproduces the failure: four new checks fail on the old goldens.py and pass on the new one.

The new rule (hack/pinlib/goldens.py, documented in pins/README.md)

A test keeps an old string only in these three cases:

  • A live site holds it. Another pin's site outside tests/ still holds the string, for example two images at one digest. This check reads the registry's sites, not raw text.
  • The test reads a record that holds it. The test names, by repo path, a kept record that still holds the string. A kept record is a file no bumper writes, .md aside. tests/bench_test.sh checking bench/apb/h7x.sh's pinned digest stays protected.
  • It is a bare version in test code. The string is a version or tag on its own, and the test is a .py/.sh/#! script. There it is as likely an input (pinlib_test, policy_test, pinver_test) as an expectation. In recorded files (goldens, fixtures) bare versions move.

Digests and revisions also move when cut to 12 hex digits, the form plans and audits print.

Per-site fixes

Verification

After merge

The open bot PR branches were cut by the old bumper, so they stay red until the next Freshness run regenerates them on this main. Policy verdicts are unchanged: review-labelled bumps still need a human approval and their canaries.

Andrew Mello added 5 commits November 6, 2026 18:23
Golden substitution skipped any old pin string that a file outside tests/ still
held: a docstring example in hack/pinver.py, a prose note in
integrations/eas/pins.json, a bench script. So the k3s tag in the audit goldens,
the compute-runtime version in host_setup_test.sh and the Hugging Face revisions
in the model optimizer fixtures never moved, and those bumps went red.

A test now keeps an old string only when another pin's site still holds it, when
the test names a file outside tests/ that still holds it (it reads that record),
or when it is a bare version in test code, where it is as likely the test's own
input. Prose and records no test names hold nothing back.

Per-site fixes:
- tests/audit_test.sh reads the k3s tag from the Dockerfile and moves to versions
  no upstream will publish (v9.9.9-k3s1, cuda 99.9.9); 13.5.0 is a real release now.
- tests/bench_vs_test.sh reads each stack's image from bench/competitors.lock.
- tests/host_setup_test.sh reads compute-runtime, IGC, level-zero and the NPU
  archive from the launcher, as it already did for the PPA builds.
- hf-meta hook (group hf-console-fallback): each moved repo's record in the saved
  metadata bundles under tests/fixtures/hf is read again at the new revision.
- pins/registry.toml derives the launcher's vver= for vllm-xpu, llm-scaler and
  gaudi-vllm; tests/pins_test.py checks every canary package constant has a site.
- packaging/check.sh verifies the unsigned rpm with --nosignature: RPM 6.1 in
  Fedora 45 enforces signatures on -K and -i.
…a test back

A plan or an audit prints a revision or digest cut to 12 hex digits, so each
digest or revision pair also moves in that form. The names rule now reads only
kept records (files no bumper writes, prose aside): a test whose comment mentions
EXPERIMENTS.md or README.md no longer keeps every string those files hold.
…pins they check

tests/chat_ui_test.sh reads the Open WebUI version from the manifest's comment
and the digest from images.lock for that tag, so the three still have to agree
but a bump no longer edits the test. tests/detect_test.sh checks both
agent-sandbox images against the launcher's SANDBOX_VERSION. The k3d-gpu-sandbox
EAS fixture names the controller by tag, as the other sandbox fixtures do: the
refusal it tests reads the namespace, and a digest-pinned copy of the live image
moved with every bump while the expected refusal kept the old tag.
…move

Golden substitution moves a digest or revision cut to its first 12 hex digits
(sha256:<12> in audits, name@<12> in plans), but hack/pin-only-diff only knew a
cut digest followed by an ellipsis. A bump whose goldens carry the cut form was
then labelled changes-defaults and held for review: the hf-console-fallback bump
(the explain goldens' @<12> revisions) and the opt-in-manifests bump (the audit
goldens' sha256:<12> digests). Both forms are now pin tokens; a twelve-digit
number elsewhere still is not.
@88plug
88plug merged commit 968f36b into main Nov 7, 2026
82 checks passed
88plug added a commit that referenced this pull request Nov 7, 2026
Brings in the goldens work (#328), the wildct H52w run and amendment R7, and release 0.1.90.
88plug added a commit that referenced this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant