Skip to content

bench(wildct): H25w run on release 0.1.89, SUPPORTED #335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
88plug merged 9 commits into from
Nov 7, 2026
Merged

bench(wildct): H25w run on release 0.1.89, SUPPORTED #335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
b8a83fa
bench(wildct): H52w run on release 0.1.89, SUPPORTED
Nov 7, 2026
e8e0bd9
bench(wildct): H52w run review fixes: docs anchor, unrecorded module …
88plug Nov 7, 2026
b6c54a7
bench(wildct): H18w run on release 0.1.89, SUPPORTED
88plug Nov 7, 2026
b8addfa
Merge the H52w run's review fixes (#331) into the H18w run
88plug Nov 7, 2026
bba4afe
bench(wildct): H18w run review fixes: docs anchor, partner update, fo…
88plug Nov 7, 2026
b5ff6ed
Merge remote-tracking branch 'origin/main' into claude/h25w-run-1106
88plug Nov 7, 2026
07cb0a8
bench(wildct): H25w run on release 0.1.89, SUPPORTED
88plug Nov 7, 2026
7c99b3c
bench(wildct): H25w records keep their logs; the ratio's miss reads p…
88plug Nov 7, 2026
a070dd6
Merge main (#332, tree identical to b4cae209) into the H25w run
88plug Nov 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 135 additions & 1 deletion EXPERIMENTS.md
View file
Open in desktop

Large diffs are not rendered by default.

24 changes: 17 additions & 7 deletions bench/wildct/RESUME.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,22 @@
# wildct resume pack

How to resume the wildct campaign ([EXPERIMENTS.md](../../EXPERIMENTS.md), "Wildct
pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5 and R6, and the "H52w
run" and "H18w run" sections). This file changes no rule. Run it top to bottom, except that step
3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates
pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5, R6 and R7, and the
"H52w run", "H18w run" and "H25w run" sections). This file changes no rule. Run it top to bottom,
except that step 3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates
`bench/wildct/nfprobe.sh` (H10 runs it). Nothing past step 1 runs until every step 1 check passes.

## Where things stand (2026-11-07, after H18w)
## Where things stand (2026-11-07, after H25w)

- **Next: row 3, `H25w` (step 6).** Holds used: 2 of at most 12.
- **Next: row 4, `H34w` (step 6).** Holds used: 3 of at most 12.
- **Row 3, `H25w`,** ran on 2026-11-07 (05:18:20Z to 05:36:58Z, rc 0; `out/typical/score.json`
written) and is scored SUPPORTED (EXPERIMENTS.md "H25w run"; records in
`results/2026-11-07/runs/H25w/`, the host before and after in `results/2026-11-07/host/h25w.txt`).
It ran from main at `ce4ff16c`, which carries amendment R7 (`row.sh h25w` sets
`K3D_GPU_AGENT_DRAFT=CPU`), shipped by step 3b as `repo-ce4ff16c47ee`; its launcher and `share/`
equal 0.1.89's. Round 6 (OFF) was cut by `run.sh`'s 18:30 level cap, so OFF has 2 rounds. The
host's containers, start times, restart counts, port codes and firewall rules read the same
before and after it, and nothing of the hold was left.
- **Row 2, `H18w`,** ran on 2026-11-07 (04:22:23Z to 04:26:30Z, rc 0; `out/run-1..3.txt` each has
`sandbox_up_rc=0` and `pip_rc=0`) and is scored SUPPORTED (EXPERIMENTS.md "H18w run"; records in
`results/2026-11-07/runs/H18w/`, the host before and after in `results/2026-11-07/host/h18w.txt`).
Expand All @@ -22,6 +30,8 @@ run" and "H18w run" sections). This file changes no rule. Run it top to bottom,
before and after it, and nothing of the hold was left.
- **Release frozen at 0.1.89 (rule W7).** `H52w`, the campaign's first hold, ran 0.1.89, so every
later hold runs it too. Step 3 does not run again, and neither does step 5.
Release 0.1.90 was cut at 2026-11-07T05:25:00Z, during `H25w`, from packaging changes; its
launcher equals 0.1.89's apart from the version line, and its `share/` equals 0.1.89's.
- **Kernel modules** (each approved by the box owner on 2026-10-29; `kernel.modprobe` stays empty):
- `vxlan` and `br_netfilter` (amendment R3; `results/2026-10-29/host/modules.txt`);
- `xt_comment xt_statistic xt_multiport xt_limit xt_NFLOG xt_physdev xt_pkttype ipt_REJECT
Expand Down Expand Up @@ -71,12 +81,12 @@ run" and "H18w run" sections). This file changes no rule. Run it top to bottom,
`8eb21d5b...`, 321,145,696 bytes, the i9's bytes) and Od's `graph.pbtxt` (`c57ecf35...`,
the i9's); both hashed again on 2026-11-07;
- `h57/bin/k3d` (the shim) and `h57/weights.sha256`;
- `repo` -> `repo-85a7bc8668d1` (and the older archives);
- `repo` -> `repo-ce4ff16c47ee` (and the older archives, `repo-85a7bc8668d1` among them);
- `setup/`: `images-before.txt`, `containers-before.txt`, `volumes-before.txt`,
`networks-before.txt` and the verify logs;
- `runs/`: `smoke1*`, `smoke2*`, `diag-smoke2*`, `smoke3*` (with `smoke3-node.log`, the
node's log, and `smoke3-node-excerpt.txt`), `smoke4*` (with `smoke4-live.txt`), `smoke5*`,
`H52w*`, `H18w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones
`H52w*`, `H18w*`, `H25w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones
renamed `resume-health.txt.<UTC time>`), `install-0.1.79.log`, `install-0.1.81.log`,
`install-0.1.89.log`, `setup-images-*.log`, `IMAGE` and `campaign.log`;
- `diag/`: `diag-up.sh`, the command `diag-smoke2` ran; `iptprobe-in.sh`, an ad-hoc form of
Expand Down
227 changes: 227 additions & 0 deletions bench/wildct/results/2026-11-07/host/h25w.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,227 @@
# wildct around the H25w hold, 2026-11-07 (UTC). Read-only reads as andrew, except the firewall
# dumps (iptables-save, ip6tables-save and nft list ruleset), as root.
# The production snapshot: running containers, published-port HTTP codes from the host, and each
# running container's StartedAt and RestartCount.

== state before the hold (2026-11-07T05:18Z), after step 3b shipped main at ce4ff16c (amendment R7)
## date 2026-11-07T05:18:11+00:00
## boot: 2026-10-29 05:11:09
## kernel: 6.6.156-2-MANJARO
## disk
/dev/nvme0n1p2 3667G 3274G 207G 95% /
## mem
MemTotal: 131524088 kB
MemAvailable: 119958892 kB
## loadavg: 0.64 0.46 0.46 1/3707 49457
VERSION="0.1.89" # kept in lockstep with packaging/aur PKGBUILD by auto-release.yml
## repo -> repo-ce4ff16c47ee COMMIT=ce4ff16c47ee447664abd6ddead869dd1bc80ce8
## runs/IMAGE: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4
## tools
k3d version v5.9.0
k3s version v1.35.5-k3s1 (default)
Client Version: v1.37.2
v4.3.1+gc0d08d0
## clusters (k3d cluster list; names only)
wtm (the box owner's; exited; not this campaign's). No k3dgpu-w-* cluster.
## our containers
ps rc=0
## live.lock
-rw-r--r-- 1 andrew andrew 0 Nov 6 23:22 /home/andrew/k3dgpu-bench-wildct/live.lock
lock-free
hold procs rc=1
H18w pid=46184 alive=no rc=0
H52w pid=40856 alive=no rc=0
convert pid=24108 alive=no rc=0
diag-smoke2 pid=17199 alive=no rc=1
smoke2 pid=32217 alive=no rc=1
smoke3 pid=62226 alive=no rc=0
smoke4 pid=48184 alive=no rc=0
smoke5 pid=31149 alive=no rc=0
## modules of /etc/modules-load.d/k3dgpu-wildct.conf
loaded: 18 of 18; missing: 0; 224 modules loaded in all
modprobe=[]
## newest k3d-gpu release at 05:18Z (gh release list): aur-v0.1.89 (2026-11-07T00:23:41Z), as installed;
## rule W7 froze it at the first hold (H52w)

== the row's command, models and images, read before the launch (row.sh h25w with WILDCT_ROW_PRINT=1)
## row print
IMAGE=cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4
MODELS=/home/andrew/k3dgpu-bench-wildct/models
OUT=/home/andrew/k3dgpu-bench-wildct/runs/H25w/out
LEVELS=typical
CLUSTER=k3dgpu-w-h25
SEED_BASE=2560
HOLD=1440
LOAD_CPUS=20-23
K3D_GPU_VENDORS=auto,-nvidia,-intel
K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1%
K3D_GPU_AGENT_DRAFT=CPU
bash /home/andrew/k3dgpu-bench-wildct/repo/bench/shared-prefix/run.sh
## row check
check rc=0
## launcher vs release
LAUNCHER-MATCH
share diff done
## node image
cryptoandcoffee/k3d-gpu@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 sha256:5811390ea98da96ebe01a3338291d4f462bb8c16e0063c99939944190547b8c8
## cpu model
model name : 13th Gen Intel(R) Core(TM) i7-13700
## staged models (sha256 of openvino_model.bin: target, and the draft staged under it)
074214fa29ab1b479536fc7184ef3036e7b355a5835e8590b61b70a8de1df2f9 OpenVINO/Qwen3-4B-int4-ov/openvino_model.bin
28613e994a6e1aaf54a6fe8677eda35e8e86684e653dde09f3ba960b8e53e152 OpenVINO/Qwen3-4B-int4-ov/OpenVINO-Qwen3-0.6B-int4-ov/openvino_model.bin
## verify logs
/home/andrew/k3dgpu-bench-wildct/setup/dl.log:VERIFY ok
/home/andrew/k3dgpu-bench-wildct/setup/dl.log:VERIFY 1 bad
/home/andrew/k3dgpu-bench-wildct/setup/verify-h57.log:VERIFY ok
newer-than-verify done
## autoquant cache
XDG_STATE_HOME=[]
ls: cannot access '/home/andrew/.local/state/k3d-gpu': No such file or directory
## OVMS image
AGENT_OVMS_IMAGE="openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58"
sha256:6348136997e05a7ee2c2b08e47bb15deb6e5096c62ab3fb821907ded7e2afd7c openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58
## stress-ng image (load.py's pin): present
ghcr.io/colinianking/stress-ng@sha256:2ea9a7c3828eda1b1ca3d0b0c5a669ec252f9caf97a6a33aedcea6b2a66d1cbe (image sha256:eb124d52...)
## cpu_atom, cpu_core
16-23
0-15

== production snapshot before the hold (2026-11-07T05:18Z)
bmd-telemetry-jaeger-1|Up
bmd-telemetry-prometheus-1|Up
feed-my-algorithm-app-1|Up
feed-my-algorithm-controller-1|Up(healthy)
feed-my-algorithm-db-1|Up(healthy)
feed-my-algorithm-redroid-1|Up
feed-my-algorithm-worker-1|Up
know-my-neighbor-db-1|Up(healthy)
proxmox-dc-manager|Up(healthy)
searxng-core|Up
searxng-valkey|Up(healthy)
state-of-my-town-app|Up
state-of-my-town-archivebox|Up(healthy)
state-of-my-town-postgres|Up(healthy)
tos-rate-calculator|Up(healthy)
webapp-adminer-1|Up
webapp-db-1|Up(healthy)
webapp-web-1|Up
port 3000 200
port 3080 200
port 3100 200
port 4317 000
port 5433 000
port 5434 000
port 5544 000
port 5565 000
port 8010 404
port 8443 301
port 8888 404
port 8890 200
port 9090 302
port 13133 404
port 16686 200
/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0
/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0
/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0
/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0
/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0
/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0
/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0
/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0
/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0
/searxng-core 2026-10-29T09:11:27.943106017Z 0
/searxng-valkey 2026-10-29T09:11:27.920115646Z 0
/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0
/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0
/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0
/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0
/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0
/webapp-db-1 2026-10-29T09:11:27.950054666Z 0
/webapp-web-1 2026-10-29T09:11:27.947765425Z 0
firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286

== production snapshot after the hold (2026-11-07T05:37Z)
bmd-telemetry-jaeger-1|Up
bmd-telemetry-prometheus-1|Up
feed-my-algorithm-app-1|Up
feed-my-algorithm-controller-1|Up(healthy)
feed-my-algorithm-db-1|Up(healthy)
feed-my-algorithm-redroid-1|Up
feed-my-algorithm-worker-1|Up
know-my-neighbor-db-1|Up(healthy)
proxmox-dc-manager|Up(healthy)
searxng-core|Up
searxng-valkey|Up(healthy)
state-of-my-town-app|Up
state-of-my-town-archivebox|Up(healthy)
state-of-my-town-postgres|Up(healthy)
tos-rate-calculator|Up(healthy)
webapp-adminer-1|Up
webapp-db-1|Up(healthy)
webapp-web-1|Up
port 3000 200
port 3080 200
port 3100 200
port 4317 000
port 5433 000
port 5434 000
port 5544 000
port 5565 000
port 8010 404
port 8443 301
port 8888 404
port 8890 200
port 9090 302
port 13133 404
port 16686 200
/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0
/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0
/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0
/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0
/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0
/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0
/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0
/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0
/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0
/searxng-core 2026-10-29T09:11:27.943106017Z 0
/searxng-valkey 2026-10-29T09:11:27.920115646Z 0
/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0
/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0
/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0
/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0
/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0
/webapp-db-1 2026-10-29T09:11:27.950054666Z 0
/webapp-web-1 2026-10-29T09:11:27.947765425Z 0
firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286

== comparison
production snapshot: identical
iptables-save rules, packet counters stripped: identical
ip6tables-save rules, packet counters stripped: identical
nft list ruleset, counters stripped: identical

== after-hold checks (RESUME.md step 6)
== after H25w 2026-11-07T05:37:17Z
0
== H25w lock taken 2026-11-07T05:18:20Z
== H25w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h25w 2026-11-07T05:18:20Z
== H25w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T05:36:58Z
list rc=0
ps rc=0
strays rc=0
/dev/nvme0n1p2 3667G 3274G 207G 95% /
lock-free
net rc=1 (grep found no k3dgpu network)
vol rc=1 (grep found no k3dgpu volume)

== host state after the hold
## date 2026-11-07T05:37:41+00:00
## modules: 224 loaded
modprobe=[]
clusters rc=0
ours grep rc=1 (1 = none)
lock-free
hold procs rc=1 (1 = none)
/dev/nvme0n1p2 3667G 3274G 207G 95% /
## thermal_throttle files: 24
## sshd: active
4 changes: 4 additions & 0 deletions bench/wildct/results/2026-11-07/runs/H25w.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
== H25w waiting for the lock 2026-11-07T05:18:20Z
== H25w lock taken 2026-11-07T05:18:20Z
== H25w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h25w 2026-11-07T05:18:20Z
== H25w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T05:36:58Z
Loading
Loading