Skip to content

bench(wildct): H25w run on release 0.1.89, SUPPORTED - #335

Open
88plug wants to merge 8 commits into
mainfrom
claude/h25w-run-1106
Open

88plug wants to merge 8 commits into
mainfrom
claude/h25w-run-1106

Conversation

@88plug

@88plug 88plug commented Nov 7, 2026

Copy link
Copy Markdown
Owner

Row 3 of the wildct campaign, H25w, run to its pre-registered protocol: the wildct pre-registration, rules W1-W8, and amendments R1, P1 and R2-R7. The registered scorer (probe.py score) reads T SUPPORTED.

Stacked on #332 (H18w). This branch is #332's branch, plus a merge of main (#331, amendment R7 #333, #328 and the 0.1.90 release commits), plus one commit. Merge #332 first. Until then this PR's diff also shows H18w's changes. Both PRs edit the same ledger rows, RESUME.md's status block and the reports' counts.

Before measuring: amendment R7 (#333, merged)

The H25w line says the agent manifest "keeps its default CPU Fast Draft in both arms, as H25 run 4 did", and the draft is staged for it. Since #163, the launcher drafts on the CPU by default only for a CPU class that an H34-family row measured, and none names the i7-13700. So the registered command would have served both arms with no draft.

Amendment R7 (mechanics only; dated, landed as ce4ff16 before the hold) adds K3D_GPU_AGENT_DRAFT=CPU to row.sh h25w. With it, the manifest is byte-identical to the i9's default. Main at ce4ff16 was then shipped to wildct (step 3b). Its launcher and share/ equal 0.1.89's.

The hold

  • One hold, hold.sh H25w 1500 8 bash bench/wildct/row.sh h25w, 2026-11-07T05:18:20Z to 05:36:58Z, rc 0.
  • The lock was free, and there was no load wait (load1 0.99).
  • Holds used: 3 of at most 12.
  • Release 0.1.89, frozen by rule W7 since H52w. Release 0.1.90 was cut during the hold from packaging changes; its launcher equals 0.1.89's apart from VERSION.
  • Node image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b511..., k3d v5.9.0, helm v4.3.1, kubectl v1.37.2.
  • OVMS 2026.4.1 (a928903f...) on the P-core balloon (16 CPUs, pod CPUs 0-15 in every round), with the 0.6B CPU Fast Draft in both arms.
  • Models: Qwen3-4B int4 074214fa... and the draft 28613e99....
  • Load: typical on E-cores 20-23. The GPU opt-out, W3 and W4 applied.
typical ON (shipping) OFF OFF/ON
added agent's first chunk, median 2.198 s (n = 15) 31.437 s (n = 10) 14.30x (95% CI 13.38-14.89)
per-round medians 2.154, 2.204, 2.281 31.408, 32.584

Every gate of the registered rule passes, so T is SUPPORTED:

  • c/OFF is 1.00 and a1 ON/OFF is 0.97.
  • The spread gate passes, no round is void, and 6 of 6 round pairs are >= 4x.
  • Load is MATCHED under amendment 27: pre busy spread 0.20 and during 0.68, against a limit of 5.0; no throttle and no swap.

Disclosed in the run section:

  • Round 6 (OFF) was cut by run.sh's 18:30 level cap, so OFF has 2 rounds. An OFF request is a 31 s cold prefill here, against 10.9 s on the i9.
    • The pre-registration counts the round as missing. An arm turns INCONCLUSIVE only when more than 1 of its 3 rounds is lost.
    • The load rule also reads MATCHED without round 6's windows (not gated).
  • The prediction's point ranges missed. The verdict and gates held, but OFF was 31.4 s against 7-15 s, ON 2.20 s against 0.6-1.6 s, and the ratio 14.30x against 6-14.
    • OFF ran 2.9x and ON 2.2x slower than H25 run 4 on the i9.
    • The package read 33.6-36.3 W in every window, wildct's 35 W PL1. The run does not test that cause.
  • The OVMS image was pulled by the pod, because k3d image import did not find the tag@digest ref, as in H25 run 4.

Production box

  • The snapshot before (05:18Z) and after (05:37Z) is identical: 18 containers Up, started 2026-10-29T09:11:27Z, restart count 0, and the same port codes.
  • iptables-save, ip6tables-save and nft list ruleset read 180, 53 and 286 lines before and after. Their rules are identical with the counters stripped.
  • All 18 modules are loaded and kernel.modprobe is empty.
  • No GPU command was run.
  • No cluster, container, network, volume or process of ours was left. live.lock is free, and the ssh master is closed.

Files

  • EXPERIMENTS.md: the H25w row's result and verdict, with a link to amendment R7. The prediction cell is unchanged. Also the "H25w run" section, with its Reproduce header.
  • bench/wildct/results/2026-11-07/runs/H25w/, H25w.out and campaign.log: the records, without the kubeconfig. host/h25w.txt: the host before and after.
  • bench/wildct/RESUME.md: the next row is 4, H34w.
  • docs/intel/technical-report.md: 22 SUPPORTED, 6 PENDING, and an H25w row.
  • docs/intel/partner-update-2026-11.md: the wildct line and its anchor.

Checks

  • python3 bench/shared-prefix/probe.py score on the copied records: identical to the score.json written on wildct.
  • python3 tests/ledger_drift_check.py: 43 files agree with 82 ledger rows.
  • tests/claims_check.sh: 28 files clean.
  • tests/wildct_hold_test.sh: 41 passed.
  • tests/docs_test.sh (mkdocs --strict): 63 passed.
  • python3 bench/wildct/test_wildct.py, tests/ledger_freshness_test.py, tests/known_issues_test.py, tests/ledger_drift_check_test.py and bench/shared-prefix/test_probe.py: OK.

Andrew Mello and others added 8 commits November 6, 2026 20:13
H52w, the first wildct hold, ran on 2026-11-07 from 04:03:54Z to 04:05:00Z under
bench/wildct/hold.sh with release 0.1.89 and the box owner's GPU opt-out. Every row of
the pre-registered detection table matches: hybrid active with the balloons policy and
the test pods on CPU 0 and CPU 17, exactly the six launcher ISA labels, no AVX-512, AMX
or NUMA label, nvidia and intel named as detected and opted out, no GPU resource, no GPU
plugin pod, no /dev hostPath, and the host inventory unchanged. The production snapshot
and firewall rules read the same before and after the hold.

The records are under bench/wildct/results/2026-11-07/runs/H52w/ (no kubeconfigs), the
host reads in host/h52w.txt. The ledger row and its run section, the technical report's
counts and RESUME.md's status follow. Rule W7 now holds the campaign at 0.1.89.
…read, two header lines

The technical report's SUPPORTED heading now names H52w, which moved its anchor, so the
partner update's link to the old anchor broke the docs build (mkdocs --strict: "does not
contain an anchor"). The link now points at the new anchor, the partner update lists
H52w among the SUPPORTED rows, and its open-items line no longer counts H52w as pending.

The run section said the loaded module count read 224 before the hold and at 04:11Z. No
record holds a 04:11Z read; the 224 is the preflight's (03:28Z, host/preflight.txt), and
host/h52w.txt has 18 of 18 loaded and kernel.modprobe empty before the hold only. The
text now says exactly that, and that no module read after the hold is recorded.

Two Reproduce lines now match the records: the shipped archive's bench/ differs from
main's at 28b1e70 in RESUME.md and the run records (the runners are equal), and
sample.py read / at 221.1-222.2 GB (decimal), where 207G is df -h's figure. No
prediction, verdict or measured value changes.
Row 2 of the wildct campaign, run to its registered line: one hold of
three bench/sandbox_round.sh runs on cluster k3dgpu-w-sbx with the release
node image, the box owner's GPU opt-out and rule W3. In every run
`sandbox up` exited 0 with the sandbox pod on CPU 17 (E-cores 16-23); the
host's tailscale and LAN sshd addresses were refused from the sandbox with
2 packets on kube-router's drop-log rule and reached from the control pod;
pip installed tomli_w from PyPI into /tmp/.local. H18's outcome table reads
SUPPORTED.

The production snapshot, the firewall rules and fail2ban's sshd jail read
the same before and after the hold, and nothing of the hold was left on
wildct. Records are under bench/wildct/results/2026-11-07/; RESUME.md names
H25w as the next row, and the technical report's counts move with the row.
The H18w run is stacked on #331, whose branch gained a review commit
(e8e0bd9: the technical report anchor in the partner update, the module
read and two header lines of the H52w run) after this branch was cut.
…ur record readings

The technical report's SUPPORTED heading now names H18w, which moved its
anchor, so the partner update's link to it failed mkdocs --strict (the Docs
build on this PR). The link now uses the new anchor. The partner update also
lists H18w with the wildct SUPPORTED rows and no longer counts it among the
PENDING wildct rows.

In the H18w run section and its ledger row, four readings now say what the
records hold. The 207 in "/ 207 GB available" is df -BG's GiB figure;
sample.py read 220.3-222.2 GB, and the swap reading is the runner's, not
sample.py's. The 0.08-0.09 s control connects are those to the two private
addresses (the control pod's pypi.org:443 connects took 0.09-0.14 s). The
18-of-18 module read is from before the hold; after it the records hold the
module count (224) and kernel.modprobe. The pip result is the runner's pip_rc,
as the section's own note on pip's exit says, and the technical report's row
states it the same way. No prediction, measurement or verdict changes.
# Conflicts:
#	EXPERIMENTS.md
#	bench/wildct/RESUME.md
#	bench/wildct/results/2026-11-07/runs/campaign.log
#	docs/intel/partner-update-2026-11.md
#	docs/intel/technical-report.md
Row 3 of the wildct campaign. One hold (2026-11-07T05:18:20Z to
05:36:58Z, rc 0) ran H25's bench/shared-prefix/run.sh as the H25w line
registers it, with amendment R7's K3D_GPU_AGENT_DRAFT=CPU so both arms keep
the CPU Fast Draft, from main at ce4ff16 on release 0.1.89.

Load matched under amendment 27 and no round was void. Round 6 (OFF) was
cut by the level's 18:30 cap, so OFF has 2 rounds. The added agent's first
chunk arrived in 2.198 s with prefix caching on and 31.437 s with it off
(14.30x, CI 13.38-14.89), c/OFF 1.00, a1 ON/OFF 0.97: T SUPPORTED. Both arms
ran 2.2-2.9x slower than on the i9, with the package at wildct's 35 W PL1.

The production snapshot, firewall rules and loaded modules read the same
before and after the hold, and nothing of the hold was left on wildct.
…lainly

The repository's .gitignore drops *.log, so the H25w records were committed
without nine files the run section cites or rests on: run.sh's output
(hold.log: each round's graph read-back, pod_cpus=0-15 and seed, and
elapsed=1111s for the cut round 6), the up log (nvidia and intel opted out,
Accelerators schedulable: hybrid=1), the image import log (no local copy of
the OVMS tag@digest ref, so the pod pulled it) and cache-usage-1..6.log.
They are added with -f, as H18w's hold.log and up logs were, byte-identical
to the copy taken from wildct after the hold. The records sentence now names
them.

The prediction comparison read "the ratio 14.30x just above 10x (6-14)",
which could read as 14.30 being just above 10. It now says the ratio was
14.30x against 10x (6-14), just above that range. No prediction, measured
value or verdict changes; probe.py score on the records still reproduces
score.json.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant