Repository navigation
pins: goldens move with their pins, and the per-site gaps that kept bot PRs red - #328
Merged
Merged
Conversation
added 5 commits
November 6, 2026 18:23
Golden substitution skipped any old pin string that a file outside tests/ still held: a docstring example in hack/pinver.py, a prose note in integrations/eas/pins.json, a bench script. So the k3s tag in the audit goldens, the compute-runtime version in host_setup_test.sh and the Hugging Face revisions in the model optimizer fixtures never moved, and those bumps went red. A test now keeps an old string only when another pin's site still holds it, when the test names a file outside tests/ that still holds it (it reads that record), or when it is a bare version in test code, where it is as likely the test's own input. Prose and records no test names hold nothing back. Per-site fixes: - tests/audit_test.sh reads the k3s tag from the Dockerfile and moves to versions no upstream will publish (v9.9.9-k3s1, cuda 99.9.9); 13.5.0 is a real release now. - tests/bench_vs_test.sh reads each stack's image from bench/competitors.lock. - tests/host_setup_test.sh reads compute-runtime, IGC, level-zero and the NPU archive from the launcher, as it already did for the PPA builds. - hf-meta hook (group hf-console-fallback): each moved repo's record in the saved metadata bundles under tests/fixtures/hf is read again at the new revision. - pins/registry.toml derives the launcher's vver= for vllm-xpu, llm-scaler and gaudi-vllm; tests/pins_test.py checks every canary package constant has a site. - packaging/check.sh verifies the unsigned rpm with --nosignature: RPM 6.1 in Fedora 45 enforces signatures on -K and -i.
…a test back A plan or an audit prints a revision or digest cut to 12 hex digits, so each digest or revision pair also moves in that form. The names rule now reads only kept records (files no bumper writes, prose aside): a test whose comment mentions EXPERIMENTS.md or README.md no longer keeps every string those files hold.
…pins they check tests/chat_ui_test.sh reads the Open WebUI version from the manifest's comment and the digest from images.lock for that tag, so the three still have to agree but a bump no longer edits the test. tests/detect_test.sh checks both agent-sandbox images against the launcher's SANDBOX_VERSION. The k3d-gpu-sandbox EAS fixture names the controller by tag, as the other sandbox fixtures do: the refusal it tests reads the namespace, and a digest-pinned copy of the live image moved with every bump while the expected refusal kept the old tag.
…move Golden substitution moves a digest or revision cut to its first 12 hex digits (sha256:<12> in audits, name@<12> in plans), but hack/pin-only-diff only knew a cut digest followed by an ellipsis. A bump whose goldens carry the cut form was then labelled changes-defaults and held for review: the hf-console-fallback bump (the explain goldens' @<12> revisions) and the opt-in-manifests bump (the audit goldens' sha256:<12> digests). Both forms are now pin tokens; a twelve-digit number elsewhere still is not.
88plug
added a commit
that referenced
this pull request
Nov 7, 2026
Brings in the goldens work (#328), the wildct H52w run and amendment R7, and release 0.1.90.
88plug
added a commit
that referenced
this pull request
Nov 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most bot PRs (#315, #318, #320, #321, #322, #324, #325, #326) were red because the bumper's golden substitution left test files at the old pin, or because a test hardcoded a value a bump moves. This PR fixes the substitution rule and the per-site gaps. No historical record, ledger row, bench result or README history row changes, and no held pin moves.
Root cause
hack/pinlib/goldens.pylive_pairsdropped every old pin string that any scanned file outsidetests/still contained. Leftover prose was enough to block a pair:v1.37.1-k3s1: kept by a docstring example inhack/pinver.pyand a note inintegrations/eas/pins.json, so the 13 audit goldens never moved (chore(k3s): k3s v1.37.2-k3s1 #322).26.35.39758.10: kept by a test input inoracle/test_oracle.py(chore(host-intel): compute-runtime 26.40.40196.8, level-zero 1.35.0, linux-tools 7.3.3 #320).tests/bump_test.shreproduces the failure: four new checks fail on the oldgoldens.pyand pass on the new one.The new rule (
hack/pinlib/goldens.py, documented inpins/README.md)A test keeps an old string only in these three cases:
tests/still holds the string, for example two images at one digest. This check reads the registry's sites, not raw text..mdaside.tests/bench_test.shcheckingbench/apb/h7x.sh's pinned digest stays protected..py/.sh/#!script. There it is as likely an input (pinlib_test,policy_test,pinver_test) as an expectation. In recorded files (goldens, fixtures) bare versions move.Digests and revisions also move when cut to 12 hex digits, the form plans and audits print.
Per-site fixes
tests/audit_test.sh: reads the k3s tag from the Dockerfile. Its fake future versions are now ones no upstream will publish (v9.9.9-k3s1, cuda99.9.9);13.5.0was the real cuda release in chore(cuda-test): cuda-test-image 13.5.0 #315.tests/bench_vs_test.sh: reads each stack's image frombench/competitors.lock(chore(images): image:images:ollama/ollama:0.40.13 0.40.15, image:images:busybox:1.37 1.38 #321).tests/host_setup_test.sh: reads compute-runtime, IGC, level-zero and the NPU archive from the launcher, as it already did for the PPA builds (chore(host-intel): compute-runtime 26.40.40196.8, level-zero 1.35.0, linux-tools 7.3.3 #320).tests/chat_ui_test.sh: reads the Open WebUI version from the manifest and the digest fromimages.lock.tests/detect_test.sh: checks the agent-sandbox images againstSANDBOX_VERSION. Thek3d-gpu-sandboxEAS fixture names the controller by tag, like its sibling fixtures (chore(opt-in-manifests): open-webui v0.13.0, agent-sandbox v1.0.10 #324).hf-metahook on grouphf-console-fallback: re-reads each moved repo's record intests/fixtures/hfat the new revision, usingconsole/model_opt.py's ownrepo_record(chore(hf-console-fallback): hf-gemma4-26b-int4-ov 7b3cbc98e24a9946e561f82c752ae26cd53fc660, hf-qwen3.8-27b 8c0b3d30f3ae96f5c7f774de704bf0ada83e422a, hf-qwen3.8-27b-int4-ov d15f4d91eccd11dd44f615ffdb539c61bf1f1e6e #318, the chore(hf-console-fallback): hf-gemma4-26b-int4-ov 7b3cbc98e24a9946e561f82c752ae26cd53fc660, hf-qwen3.8-27b 8c0b3d30f3ae96f5c7f774de704bf0ada83e422a, hf-qwen3.8-27b-int4-ov d15f4d91eccd11dd44f615ffdb539c61bf1f1e6e #299 class).pins/registry.toml: derives the launcher'svver=for vllm-xpu (chore(vllm-xpu): vllm-xpu v0.33.0 #326), and for llm-scaler and gaudi-vllm, which had the same gap.tests/pins_test.pynow fails when a canary package constant has no registry site.packaging/check.sh:rpm -K --nosignatureandrpm -i --nosignature. RPM 6.1 in Fedora 45 rejects an unsigned package on both (digests SIGNATURES NOT OK,does not verify: no signature). Checked in fedora:45 and fedora:44, and a corrupted payload still fails the digest check (chore(packaging-images): packaging-fedora 45, image:packaging-images:debian:trixie-slim trixie-slim, image:packaging-images:ghcr.io/homebrew/brew:7.0.11 7.0.12 #325).Verification
hack/bump <group>with this bumper against live upstreams and committing the result.oracle/test_adversarial.py(its 10 s Ctrl-C wait),tests/wildct_hold_test.sh,bench/admission/test_run.sh, and oncetests/eas_test.sh, whose argv sampler readsps -eo argsand saw another tree's run. All of them passed when rerun alone in the same tree.audit_test(chore(cuda-test): cuda-test-image 13.5.0 #315, chore(k3s): k3s v1.37.2-k3s1 #322),host_setup_test(chore(host-intel): compute-runtime 26.40.40196.8, level-zero 1.35.0, linux-tools 7.3.3 #320),bench_vs_test(chore(images): image:images:ollama/ollama:0.40.13 0.40.15, image:images:busybox:1.37 1.38 #321),audit_test/chat_ui_test/detect_test/eas_test(chore(opt-in-manifests): open-webui v0.13.0, agent-sandbox v1.0.10 #324),packaging/check.sh nfpm(chore(packaging-images): packaging-fedora 45, image:packaging-images:debian:trixie-slim trixie-slim, image:packaging-images:ghcr.io/homebrew/brew:7.0.11 7.0.12 #325),canary_spec_test(chore(vllm-xpu): vllm-xpu v0.33.0 #326) andmodel_opt_test(chore(hf-console-fallback): hf-gemma4-26b-int4-ov 7b3cbc98e24a9946e561f82c752ae26cd53fc660, hf-qwen3.8-27b 8c0b3d30f3ae96f5c7f774de704bf0ada83e422a, hf-qwen3.8-27b-int4-ov d15f4d91eccd11dd44f615ffdb539c61bf1f1e6e #318). All of them pass after the fix.After merge
The open bot PR branches were cut by the old bumper, so they stay red until the next Freshness run regenerates them on this main. Policy verdicts are unchanged: review-labelled bumps still need a human approval and their canaries.