diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index b18750bf..039f8cf4 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -85,7 +85,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57dp | On the i9-13900T P-cores under the shared host's own load, llama.cpp with Gemma 4 QAT Q4_0 and the QAT MTP drafter decodes faster unpinned (8 threads, no `-C`/`--cpu-strict`) than strictly pinned one thread per physical P-core: N=1 ratio >= 1.10. | Written 2026-10-21, after H57d, before any H57dp measurement. SUPPORTED (0.5); Ld-mtp-u/Ld-mtp in [1.0, 1.8] (0.6). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): 2 holds of at most 75 min, H57d's protocol. SUPPORTED: pooled median ratio >= 1.10, CI lower bound > 1.0, faster in both holds. REFUTED: CI upper bound < 1.10. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched, foreign P-core load spread 2.97 > 2.0 CPU-s/s (Od 0.45, Ld-mtp 3.42, Ld-mtp-u 3.11), so the scorer stops before the statistic. Not gated: Ld-mtp-u/Ld-mtp 1.108 (95% CI 1.072-1.142), faster in both holds (1.108, 1.100); Ld-mtp-u 13.39, Ld-mtp 12.09 tok/s pooled N=1; affinity as registered. | INCONCLUSIVE | | H57de | H57da with the unpinned llama.cpp arm: no arm decodes Gemma 4 QAT at N=1 5% or more faster than k3d-gpu's OVMS pick Od on the i9-13900T P-cores. | Written 2026-10-21, before any H57de measurement. REFUTED (0.5); Ld-mtp-u/Od in [1.0, 1.6] (0.5). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): same holds as H57dp. REFUTED: Ld-mtp-u/Od CI lower bound >= 1.05 and faster in both holds. SUPPORTED: CI upper bound < 1.05. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched (the same gate, spread 2.97 > 2.0). Od also ran without the P-core balloon (cpuset 0-15,17-31: the runner's `agent manifest` call named no cluster, so after #183 it missed the hold's kube context), so it was not the registered Od. Not gated: Ld-mtp-u/Od 1.369 (1.345-1.396), Od 9.78 tok/s pooled N=1. | INCONCLUSIVE | | H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. Result ([H52w run](#h52w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign). `up` took 32 s, named nvidia and intel as opted out and printed `Accelerators: hybrid`; the balloons policy pod Running, `k3d-gpu test`'s P-core pod on CPU 0 and its E-core pod on CPU 17; the launcher's ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT and CETSS, no AVX512 or AMX label from any source; no NUMA label; allocatable cpu, ephemeral-storage, hugepages, memory and pods only; no GPU plugin pod, no `/dev` hostPath and no GPU resource (`GPU-SCAN clean`); the node container `gpus=[]`; the inventory unchanged. Every row matches. | SUPPORTED | -| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. | PENDING | +| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. Result ([H18w run](#h18w-run-wildct-i7-13700-2026-11-07)): one hold of 3 runs on release 0.1.89 (rule W7's release for the campaign), a fresh cluster each. In every run `sandbox up` exited 0 with the warm pool's `readyReplicas` 1 and the sandbox pod on CPU 17, inside the E-cores 16-23; from the sandbox, TCP to `100.66.38.121:22` and to `192.168.8.238:22` was refused in 0.11-0.12 s with 2 packets on the policy's drop-log rule, and from the control pod both connected; `pypi.org` resolved, `pypi.org:443` connected and `pip install --user tomli_w==1.2.0` gave `pip_rc=0`, with the import reading `/tmp/.local`. Every run meets (a), (b) and (c). | SUPPORTED | | H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | | H34w | H34's hypothesis on wildct: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, both on the 16-CPU P-core balloon, under matched background load. A replication: original models, runner, scorer and rule. | Written 2026-10-20, before any H34w measurement. At `typical`: median paired B/A decode tok/s 1.35 (1.15-1.6) and median paired B/A package J/token 0.76 (0.6-0.92). SUPPORTED by H34's rule at 0.6, REFUTED 0.25, INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, `typical` only, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h34`, the GPU opt-out in place of H34's `intel,hybrid`, W2, W3, W4. | PENDING | @@ -8909,6 +8909,114 @@ the hold. Rule W7: this was the campaign's first hold, so every later wildct row runs release 0.1.89. Holds used: 1 of at most 12. The next row in the registered order is H18w. +### H18w run (wildct, i7-13700, 2026-11-07) + +**Reproduce:** + +```text +# provenance +date: 2026-11-07T04:22:23Z to 04:26:30Z (one hold; lock free, no load wait: load1 0.52); + runs 1-3 began 04:22:29Z, 04:23:31Z and 04:25:25Z +hypothesis: H18w (H18's runner and outcome table, as the wildct pre-registration's H18w line lists them) +host: wildct, reached by IP 100.66.38.121: Intel Core i7-13700 (family 6 model 183), 8 P-cores + (cpu_core 0-15) and 8 E-cores (cpu_atom 16-23), 1 NUMA node, 125 GiB RAM, Manjaro; Docker + 29.7.2, live-restore off, cgroup systemd/2 (step 1, 03:32:22Z) +host kernel: 6.6.156-2-MANJARO +accelerators: hybrid=1; UHD 770 (i915), RTX 4000 SFF Ada and RTX 2000 Ada (nvidia), opted out and not + used (rule W8) +network: tailscale0 100.66.38.121, default-route source 192.168.8.238, sshd on 0.0.0.0:22 and [::]:22; + TARGETS as the runner resolved them: 100.66.38.121:22 192.168.8.238:22 pypi.org:443 +k3d-gpu: release 0.1.89 (aur-v0.1.89, .deb sha256 8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc), + rule W7's release since the first hold (H52w) +bench scripts: main at 85a7bc86, shipped as the archive repo-85a7bc8668d1; sandbox_round.sh runs its + scripts/k3d-gpu (equal to 0.1.89's apart from VERSION) and share/ (equal to 0.1.89's) +node image: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4, + k3s v1.37.1+k3s1; k3d v5.9.0, helm v4.3.1, kubectl v1.37.2 +runtime: agent-sandbox v1.0.8 controller and python-runtime-sandbox v1.0.8, both pinned by digest in + share/ (H18 ran v1.0.7); NetworkPolicy enforced by k3s's embedded kube-router +model: none +environment: IMAGE=, K3D_GPU_CLUSTER=k3dgpu-w-sbx, K3D_GPU_VENDORS=auto,-nvidia,-intel, + K3D_GPU_K3S_ARGS=, K3D_GPU_RAPL_EXEC="docker exec k3dgpu-w-rapl" (rule W4) +energy source: intel-rapl:0 energy_uj (package-0), confounds only: package W over 5 s idle before each run +runs: 3 counted, one fresh cluster each, in one hold; no pilot +host load: load1 0.48-1.51, foreign CPU 0.51-1.65 CPU-s/s, MemAvailable 113.4-114.7 GiB, package + 61-69 C, / 220.3-222.2 GB available (sample.py, 49 samples; df -BG read 207G before and + after); swap used 0 MiB at each run's start (the runner's read) +``` + +Command, on wildct: RESUME.md step 6's `launch H18w 2100 3 h18w`, which runs `bench/wildct/hold.sh H18w +2100 3 bash bench/wildct/row.sh h18w`, and so `IMAGE= OUT=~/k3dgpu-bench-wildct/runs/H18w/out +K3D_GPU_CLUSTER=k3dgpu-w-sbx K3D_GPU_VENDORS=auto,-nvidia,-intel K3D_GPU_K3S_ARGS= +bench/sandbox_round.sh ` for n = 1, 2, 3, the registered line. Records (each run's key=value block, +`up` and `sandbox up` logs, the hold's samples; the kubeconfigs were not copied): +[bench/wildct/results/2026-11-07/runs/H18w/](bench/wildct/results/2026-11-07/runs/H18w/). The host +before and after the hold: [bench/wildct/results/2026-11-07/host/h18w.txt](bench/wildct/results/2026-11-07/host/h18w.txt). +The scored lines of each run: + +```bash +grep -hE '^(run=|sandbox_up_rc|warm_ready|pod=|probe|policy_chain|dns_pypi|env=|pip_rc)' \ + bench/wildct/results/2026-11-07/runs/H18w/out/run-{1,2,3}.txt +``` + +Each `up` named nvidia and intel as opted out, printed `Accelerators: hybrid` and ended with +`Accelerators schedulable: hybrid=1`. + +| Run | `sandbox up` (s) | sandbox CPUs | sandbox -> 100.66.38.121:22 | control -> same | sandbox -> 192.168.8.238:22 | control -> same | drop-log pkts | pypi.org:443 | pip `tomli_w` | load (1 min) | throttle delta | +|-----|------|----|------|------|------|------|---|------|------|------|------| +| 1 | 12.3 | 17 | refused, 0.12 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 7.4 s | 0.48 | 0 | +| 2 | 69.9 | 17 | refused, 0.12 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 1.2 s | 1.23 | 0 | +| 3 | 15.0 | 17 | refused, 0.11 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 1.7 s | 1.15 | 0 | + +In each run: + +- `sandbox up` exited 0, the warm pool's `readyReplicas` was 1, and its status check printed + `ok: python-ecores-: CPUs 17 (E-cores 16-23)`. The runner's own read of the pod's + `Cpus_allowed_list` was 17, with the host's `cpu_atom` 16-23. +- Every blocked connect printed `NOCONNECT ConnectionRefusedError [Errno 111] Connection refused`. + Each of kube-router's policy chains (one per cluster) counted 2 packets on its NFLOG rule + ("DROP by policy agent-sandbox/python-ecores-network-policy"). Each control connect to those two + addresses took 0.08-0.09 s. +- `pypi.org` resolved to 151.101.0.223 (runs 1 and 2) and 151.101.192.223 (run 3), both Fastly. The + sandbox's connect to `pypi.org:443` took 2.33, 0.22 and 0.14 s. +- The sandbox ran with `HOME=/tmp uid=1000`. pip's last line of output was its upgrade notice, and the + import read `/tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py`. + +`sandbox up` wall time: median 15.0 s, spread (max - min) 57.6 s, n = 3. Run 2's warm pool was 67 s +old when `sandbox up` read it ready (9 s and 12 s in runs 1 and 3); each fresh cluster pulls the +controller and runtime images, and the records do not say more. None of the checks is a timing +threshold. Confounds: package power over the 5 s before each run 35.6, 35.3 and 34.6 W; swap used 0 MiB; +the package throttle count delta, summed over the host's 24 counters, 0 in every run. + +**Verdict: SUPPORTED.** Every run meets (a), (b) and (c), the first line of H18's outcome table. On a +second client Raptor Lake host, `k3d-gpu sandbox up` from the release kept each sandbox on an E-core +and behind the default-deny egress policy: the tailscale and LAN addresses were refused from the +sandbox and reached from a control pod without the policy, and PyPI stayed reachable for a `pip install +--user`. The prediction (every run meets (a), (b) and (c), 0.8) held. Scope: as H18's, k3s v1.37.1 +kube-router and IPv4 targets only; the three registered targets; agent-sandbox v1.0.8, the release's. + +Not scored: + +- Rule W2 keeps CPUs 16-19 free for the production services, except for H18w's one 1-CPU sandbox pod + and k3s's own pods. The sandbox pod got CPU 17, inside that range, and ran the probes and the pip + install there (1.2-7.4 s). +- As in H18, `pip_rc` is the exit of `pip install ... | tail -1 && python3 -c 'import tomli_w ...'` in + the pod's `sh`, so pip's own exit reaches it only through the import that follows. The import read + the package from `/tmp/.local`, where only the `--user` install puts it. +- The control pod's six connects to the host's sshd (two per run) left fail2ban's sshd jail as it was: + `Total failed` 0 and `Total banned` 2 before and after the hold. The jail runs the filter's `normal` + mode, with `maxretry` 5. + +Production box. The snapshot before the hold (04:20Z) and after it (04:27Z) read the same: 18 +containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the same port codes. The +line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and +after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` +cluster or container, no process of the hold and no network or volume of ours was left, and +`live.lock` was free. Before the hold (04:20Z) all 18 modules of +`/etc/modules-load.d/k3dgpu-wildct.conf` were loaded and `kernel.modprobe` was empty; after it +(04:27Z) the host had 224 modules loaded and `kernel.modprobe` was still empty. + +Holds used: 2 of at most 12. The next row in the registered order is H25w. + ### Wildct amendment R7: H25w's CPU Fast Draft (2026-11-07) Recorded 2026-11-07, after the H52w and H18w holds and before any H25w hold. Mechanics only. No diff --git a/bench/wildct/RESUME.md b/bench/wildct/RESUME.md index 1870ae39..a453a0bc 100644 --- a/bench/wildct/RESUME.md +++ b/bench/wildct/RESUME.md @@ -2,18 +2,24 @@ How to resume the wildct campaign ([EXPERIMENTS.md](../../EXPERIMENTS.md), "Wildct pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5 and R6, and the "H52w -run" section). This file changes no rule. Run it top to bottom, except that step 3b (ship -`origin/main` to wildct) runs first when the `repo` on wildct predates `bench/wildct/nfprobe.sh` -(H10 runs it). Nothing past step 1 runs until every step 1 check passes. - -## Where things stand (2026-11-07, after H52w) - -- **Next: row 2, `H18w` (step 6).** Row 1, `H52w`, ran on 2026-11-07 (04:03:54Z to 04:05:00Z, - rc 0, every line of `out/rc.txt` a pass) and is scored SUPPORTED (EXPERIMENTS.md "H52w run"; - records in `results/2026-11-07/runs/H52w/`, the host before and after in - `results/2026-11-07/host/h52w.txt`). The host's containers, start times, restart counts, port - codes and firewall rules read the same before and after it, and nothing of the hold was left. - Holds used: 1 of at most 12. +run" and "H18w run" sections). This file changes no rule. Run it top to bottom, except that step +3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates +`bench/wildct/nfprobe.sh` (H10 runs it). Nothing past step 1 runs until every step 1 check passes. + +## Where things stand (2026-11-07, after H18w) + +- **Next: row 3, `H25w` (step 6).** Holds used: 2 of at most 12. +- **Row 2, `H18w`,** ran on 2026-11-07 (04:22:23Z to 04:26:30Z, rc 0; `out/run-1..3.txt` each has + `sandbox_up_rc=0` and `pip_rc=0`) and is scored SUPPORTED (EXPERIMENTS.md "H18w run"; records in + `results/2026-11-07/runs/H18w/`, the host before and after in `results/2026-11-07/host/h18w.txt`). + The host's containers, start times, restart counts, port codes and firewall rules read the same + before and after it, fail2ban's sshd jail counters did not change, and nothing of the hold was + left. +- **Row 1, `H52w`,** ran on 2026-11-07 (04:03:54Z to 04:05:00Z, rc 0, every line of `out/rc.txt` a + pass) and is scored SUPPORTED (EXPERIMENTS.md "H52w run"; records in + `results/2026-11-07/runs/H52w/`, the host before and after in `results/2026-11-07/host/h52w.txt`). + The host's containers, start times, restart counts, port codes and firewall rules read the same + before and after it, and nothing of the hold was left. - **Release frozen at 0.1.89 (rule W7).** `H52w`, the campaign's first hold, ran 0.1.89, so every later hold runs it too. Step 3 does not run again, and neither does step 5. - **Kernel modules** (each approved by the box owner on 2026-10-29; `kernel.modprobe` stays empty): @@ -70,8 +76,8 @@ run" section). This file changes no rule. Run it top to bottom, except that step `networks-before.txt` and the verify logs; - `runs/`: `smoke1*`, `smoke2*`, `diag-smoke2*`, `smoke3*` (with `smoke3-node.log`, the node's log, and `smoke3-node-excerpt.txt`), `smoke4*` (with `smoke4-live.txt`), `smoke5*`, - `H52w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones renamed - `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, + `H52w*`, `H18w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones + renamed `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, `install-0.1.89.log`, `setup-images-*.log`, `IMAGE` and `campaign.log`; - `diag/`: `diag-up.sh`, the command `diag-smoke2` ran; `iptprobe-in.sh`, an ad-hoc form of `nfprobe.sh`; the second load's read-only checks `modsnap.sh` (modules and firewall, as diff --git a/bench/wildct/results/2026-11-07/host/h18w.txt b/bench/wildct/results/2026-11-07/host/h18w.txt new file mode 100644 index 00000000..ee0099e8 --- /dev/null +++ b/bench/wildct/results/2026-11-07/host/h18w.txt @@ -0,0 +1,231 @@ +# wildct around the H18w hold, 2026-11-07 (UTC). Read-only reads as andrew, except the firewall +# dumps (iptables-save, ip6tables-save and nft list ruleset) and fail2ban-client status, as root. +# The production snapshot: running containers, published-port HTTP codes from the host, and each +# running container's StartedAt and RestartCount. + +== state before the hold (2026-11-07T04:20Z) +## date 2026-11-07T04:20:28+00:00 +## boot: 2026-10-29 05:11:09 +## kernel: 6.6.156-2-MANJARO +## disk +/dev/nvme0n1p2 3667G 3273G 207G 95% / +## mem +MemTotal: 131524088 kB +MemAvailable: 120085356 kB +## loadavg: 0.43 0.45 0.48 1/3658 32271 +VERSION="0.1.89" # kept in lockstep with packaging/aur PKGBUILD by auto-release.yml +## repo -> repo-85a7bc8668d1 COMMIT=85a7bc8668d18a3233496f0a3a23b09bb63ee20f +## runs/IMAGE: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +## tools +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +## clusters (k3d cluster list; names only) +wtm (the box owner's; exited; not this campaign's). No k3dgpu-w-* cluster. +## our containers +ps rc=0 +## live.lock +-rw-r--r-- 1 andrew andrew 0 Nov 6 23:03 /home/andrew/k3dgpu-bench-wildct/live.lock +lock-free +hold procs rc=1 +H52w pid=40856 alive=no rc=0 +convert pid=24108 alive=no rc=0 +diag-smoke2 pid=17199 alive=no rc=1 +smoke2 pid=32217 alive=no rc=1 +smoke3 pid=62226 alive=no rc=0 +smoke4 pid=48184 alive=no rc=0 +smoke5 pid=31149 alive=no rc=0 +## modules of /etc/modules-load.d/k3dgpu-wildct.conf +loaded: 18 of 18; missing: 0 +modprobe=[] +## newest k3d-gpu release at 04:21Z (gh release list): aur-v0.1.89 (2026-11-07T00:23:41Z), as installed; +## rule W7 froze it at the first hold (H52w) + +== the row's command and targets, read before the launch (row.sh h18w with WILDCT_ROW_PRINT=1) +## row print +IMAGE=cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +OUT=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out +K3D_GPU_CLUSTER=k3dgpu-w-sbx +K3D_GPU_VENDORS=auto,-nvidia,-intel +K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1% +bash /home/andrew/k3dgpu-bench-wildct/repo/bench/sandbox_round.sh (runs 1-3) +## row check +check rc=0 +## TARGETS as sandbox_round.sh resolves them +TARGETS=100.66.38.121:22 192.168.8.238:22 pypi.org:443 +## sshd listen +LISTEN 0 0 0.0.0.0:22 0.0.0.0:* +LISTEN 0 0 *:22 *:* +## launcher vs release +LAUNCHER-MATCH +share diff done +## node image +cryptoandcoffee/k3d-gpu@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 sha256:5811390ea98da96ebe01a3338291d4f462bb8c16e0063c99939944190547b8c8 +## sandbox images in share + image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v1.0.8@sha256:e1d96312487eba7d8f001e848a9d4e644b7deea8c2a4564ccc318abb729be731 + image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v1.0.8@sha256:41da84809426098cb04cdafbb41855f15e3604f7455ef53816bad666e532e787 +## cpu_atom, cpu_core +16-23 +0-15 + +== fail2ban (sshd jail: maxretry 5, findtime 1d, filter mode normal) before the hold +Status for the jail: sshd +|- Filter +| |- Currently failed: 0 +| |- Total failed: 0 +| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd + _COMM=sshd-session +`- Actions + |- Currently banned: 2 + |- Total banned: 2 + +== production snapshot before the hold (2026-11-07T04:20Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== production snapshot after the hold (2026-11-07T04:27Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== fail2ban after the hold +Status for the jail: sshd +|- Filter +| |- Currently failed: 0 +| |- Total failed: 0 +| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd + _COMM=sshd-session +`- Actions + |- Currently banned: 2 + |- Total banned: 2 + +== comparison +production snapshot: identical +iptables-save rules, packet counters stripped: identical +ip6tables-save rules, packet counters stripped: identical +nft list ruleset, counters stripped: identical +fail2ban sshd jail counters: identical + +== after-hold checks (RESUME.md step 6) +== after H18w 2026-11-07T04:26:59Z +0 +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z +list rc=0 +ps rc=0 +strays rc=0 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +lock-free +net rc=1 (grep found no k3dgpu network) +vol rc=1 (grep found no k3dgpu volume) + +== host state after the hold +## date 2026-11-07T04:27:50+00:00 +## modules: 224 loaded +modprobe=[] +clusters rc=0 +ours grep rc=1 (1 = none) +lock-free +hold procs rc=1 (1 = none) +/dev/nvme0n1p2 3667G 3273G 207G 95% / +## thermal_throttle files: 24 +## sshd: active diff --git a/bench/wildct/results/2026-11-07/runs/H18w.out b/bench/wildct/results/2026-11-07/runs/H18w.out new file mode 100644 index 00000000..10276bf5 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w.out @@ -0,0 +1,4 @@ +== H18w waiting for the lock 2026-11-07T04:22:23Z +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z diff --git a/bench/wildct/results/2026-11-07/runs/H18w/hold.log b/bench/wildct/results/2026-11-07/runs/H18w/hold.log new file mode 100644 index 00000000..18b4c112 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/hold.log @@ -0,0 +1,48 @@ +run=1 date=2026-11-07T04:22:29Z idle_pkg_w=35.6 load1=0.48 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=12.3 +status: ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-hfgtf cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=2.33 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.10 CONNECT +policy_chain=KUBE-NWPLCY-MPEOAIZRZ6XJTD26 policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=7.4 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 +run=2 date=2026-11-07T04:23:31Z idle_pkg_w=35.3 load1=1.23 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=69.9 +status: ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-cthm6 cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.22 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.14 CONNECT +policy_chain=KUBE-NWPLCY-6TDZP7NK6CZGKVRG policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.2 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 +run=3 date=2026-11-07T04:25:25Z idle_pkg_w=34.6 load1=1.15 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=15.0 +status: ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-7nd5c cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.11 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.14 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.09 CONNECT +policy_chain=KUBE-NWPLCY-WQVD77VIFGWKCLQF policy_drop_pkts=2 +dns_pypi=151.101.192.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.7 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/launcher.diff b/bench/wildct/results/2026-11-07/runs/H18w/launcher.diff new file mode 100644 index 00000000..e69de29b diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt new file mode 100644 index 00000000..d9e6cb4d --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt @@ -0,0 +1,16 @@ +run=1 date=2026-11-07T04:22:29Z idle_pkg_w=35.6 load1=0.48 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=12.3 +status: ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-hfgtf cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=2.33 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.10 CONNECT +policy_chain=KUBE-NWPLCY-MPEOAIZRZ6XJTD26 policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=7.4 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt new file mode 100644 index 00000000..03eb63fd --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt @@ -0,0 +1,16 @@ +run=2 date=2026-11-07T04:23:31Z idle_pkg_w=35.3 load1=1.23 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=69.9 +status: ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-cthm6 cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.22 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.14 CONNECT +policy_chain=KUBE-NWPLCY-6TDZP7NK6CZGKVRG policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.2 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt new file mode 100644 index 00000000..6b9f6dea --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt @@ -0,0 +1,16 @@ +run=3 date=2026-11-07T04:25:25Z idle_pkg_w=34.6 load1=1.15 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=15.0 +status: ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-7nd5c cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.11 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.14 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.09 CONNECT +policy_chain=KUBE-NWPLCY-WQVD77VIFGWKCLQF policy_drop_pkts=2 +dns_pypi=151.101.192.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.7 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log new file mode 100644 index 00000000..d5c87d6b --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 9s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-hfgtf True DependenciesReady 8s + ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log new file mode 100644 index 00000000..2b78e5db --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 67s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-cthm6 True DependenciesReady 66s + ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log new file mode 100644 index 00000000..cc4ffad8 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 12s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-7nd5c True DependenciesReady 12s + ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log new file mode 100644 index 00000000..a4902dda --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log @@ -0,0 +1,34 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-1 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +Waiting for daemon set "nri-resource-policy-balloons" rollout to finish: 0 of 1 updated pods are available... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log new file mode 100644 index 00000000..aca51d04 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log @@ -0,0 +1,34 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-2 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +Waiting for daemon set "nri-resource-policy-balloons" rollout to finish: 0 of 1 updated pods are available... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log new file mode 100644 index 00000000..0d231735 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log @@ -0,0 +1,33 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-3 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/pid b/bench/wildct/results/2026-11-07/runs/H18w/pid new file mode 100644 index 00000000..d8033cb0 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/pid @@ -0,0 +1 @@ +46184 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/post.txt b/bench/wildct/results/2026-11-07/runs/H18w/post.txt new file mode 100644 index 00000000..9957da65 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/post.txt @@ -0,0 +1,17 @@ +2026-11-07T04:26:29Z + 23:26:29 up 8 days, 19:15, 2 users, load average: 1.25, 0.93, 0.67 +%CPU PID USER COMMAND + 2.7 6519 andrew rustdesk + 2.6 1157 root tailscaled + 1.7 8054 andrew firefox + 1.5 7117 andrew rustdesk + 1.4 1 root systemd + 1.4 9806 andrew WebExtensions + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.4 12980 andrew 1password + 0.3 25800 andrew localsearch-ext + total used free shared buff/cache available +Mem: 125 11 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/pre.txt b/bench/wildct/results/2026-11-07/runs/H18w/pre.txt new file mode 100644 index 00000000..554587b2 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/pre.txt @@ -0,0 +1,17 @@ +2026-11-07T04:22:23Z + 23:22:23 up 8 days, 19:11, 2 users, load average: 0.52, 0.47, 0.48 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 1.8 8054 andrew firefox + 1.5 29612 andrew rustdesk + 1.4 1 root systemd + 1.4 9806 andrew WebExtensions + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.5 7035 andrew rustdesk + 0.4 12980 andrew 1password + 0.3 3313 root dockerd + total used free shared buff/cache available +Mem: 125 10 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/rc b/bench/wildct/results/2026-11-07/runs/H18w/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/rc @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl b/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl new file mode 100644 index 00000000..c5d34895 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl @@ -0,0 +1,49 @@ +{"ts": 1794025349.324, "seconds": 5.0, "load1": 0.48, "busy_p": 0.28, "busy_e_free": 0.146, "busy_e_load": 0.118, "busy_all": 0.544, "mine_cpu": 0.001, "foreign_cpu": 0.543, "other_containers_cpu": 0.047, "mem_available_gib": 114.45, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025354.325, "seconds": 5.001, "load1": 0.52, "busy_p": 0.856, "busy_e_free": 0.3, "busy_e_load": 0.274, "busy_all": 1.43, "mine_cpu": 0.001, "foreign_cpu": 1.429, "other_containers_cpu": 0.072, "mem_available_gib": 114.16, "pkg_temp_c": 64.0, "root_avail_gb": 222.2} +{"ts": 1794025359.326, "seconds": 5.001, "load1": 0.56, "busy_p": 0.752, "busy_e_free": 0.232, "busy_e_load": 0.186, "busy_all": 1.17, "mine_cpu": 0.0, "foreign_cpu": 1.17, "other_containers_cpu": 0.063, "mem_available_gib": 114.05, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025364.327, "seconds": 5.001, "load1": 0.68, "busy_p": 0.48, "busy_e_free": 0.182, "busy_e_load": 0.144, "busy_all": 0.806, "mine_cpu": 0.0, "foreign_cpu": 0.806, "other_containers_cpu": 0.069, "mem_available_gib": 114.0, "pkg_temp_c": 69.0, "root_avail_gb": 222.2} +{"ts": 1794025369.328, "seconds": 5.001, "load1": 0.62, "busy_p": 0.276, "busy_e_free": 0.138, "busy_e_load": 0.12, "busy_all": 0.534, "mine_cpu": 0.0, "foreign_cpu": 0.534, "other_containers_cpu": 0.041, "mem_available_gib": 114.01, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025374.329, "seconds": 5.001, "load1": 0.57, "busy_p": 0.388, "busy_e_free": 0.18, "busy_e_load": 0.136, "busy_all": 0.704, "mine_cpu": 0.0, "foreign_cpu": 0.704, "other_containers_cpu": 0.086, "mem_available_gib": 113.96, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025379.345, "seconds": 5.015, "load1": 0.85, "busy_p": 1.773, "busy_e_free": 0.54, "busy_e_load": 0.399, "busy_all": 2.712, "mine_cpu": 1.906, "foreign_cpu": 0.806, "other_containers_cpu": 0.063, "mem_available_gib": 113.64, "pkg_temp_c": 63.0, "root_avail_gb": 221.2} +{"ts": 1794025384.346, "seconds": 5.001, "load1": 1.02, "busy_p": 1.861, "busy_e_free": 0.588, "busy_e_load": 0.52, "busy_all": 2.969, "mine_cpu": 1.685, "foreign_cpu": 1.284, "other_containers_cpu": 0.094, "mem_available_gib": 113.56, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025389.348, "seconds": 5.001, "load1": 0.94, "busy_p": 0.734, "busy_e_free": 0.236, "busy_e_load": 0.186, "busy_all": 1.156, "mine_cpu": 0.486, "foreign_cpu": 0.67, "other_containers_cpu": 0.069, "mem_available_gib": 113.58, "pkg_temp_c": 63.0, "root_avail_gb": 220.7} +{"ts": 1794025394.349, "seconds": 5.002, "load1": 1.1, "busy_p": 1.31, "busy_e_free": 0.558, "busy_e_load": 0.326, "busy_all": 2.193, "mine_cpu": 1.256, "foreign_cpu": 0.937, "other_containers_cpu": 0.072, "mem_available_gib": 113.49, "pkg_temp_c": 63.0, "root_avail_gb": 220.3} +{"ts": 1794025399.351, "seconds": 5.002, "load1": 1.18, "busy_p": 0.366, "busy_e_free": 0.224, "busy_e_load": 0.13, "busy_all": 0.72, "mine_cpu": 0.194, "foreign_cpu": 0.526, "other_containers_cpu": 0.041, "mem_available_gib": 113.47, "pkg_temp_c": 61.0, "root_avail_gb": 220.3} +{"ts": 1794025404.352, "seconds": 5.001, "load1": 1.08, "busy_p": 0.378, "busy_e_free": 0.138, "busy_e_load": 0.112, "busy_all": 0.628, "mine_cpu": 0.08, "foreign_cpu": 0.548, "other_containers_cpu": 0.083, "mem_available_gib": 114.44, "pkg_temp_c": 65.0, "root_avail_gb": 220.3} +{"ts": 1794025409.367, "seconds": 5.015, "load1": 1.23, "busy_p": 0.467, "busy_e_free": 0.136, "busy_e_load": 0.114, "busy_all": 0.716, "mine_cpu": 0.002, "foreign_cpu": 0.714, "other_containers_cpu": 0.051, "mem_available_gib": 114.43, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025414.368, "seconds": 5.001, "load1": 1.22, "busy_p": 0.624, "busy_e_free": 0.216, "busy_e_load": 0.184, "busy_all": 1.024, "mine_cpu": 0.001, "foreign_cpu": 1.023, "other_containers_cpu": 0.063, "mem_available_gib": 114.26, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794025419.369, "seconds": 5.001, "load1": 1.28, "busy_p": 1.074, "busy_e_free": 0.3, "busy_e_load": 0.222, "busy_all": 1.596, "mine_cpu": 0.0, "foreign_cpu": 1.596, "other_containers_cpu": 0.068, "mem_available_gib": 114.05, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025424.37, "seconds": 5.001, "load1": 1.18, "busy_p": 0.67, "busy_e_free": 0.168, "busy_e_load": 0.144, "busy_all": 0.982, "mine_cpu": 0.0, "foreign_cpu": 0.982, "other_containers_cpu": 0.061, "mem_available_gib": 114.0, "pkg_temp_c": 64.0, "root_avail_gb": 222.2} +{"ts": 1794025429.371, "seconds": 5.001, "load1": 1.08, "busy_p": 0.298, "busy_e_free": 0.12, "busy_e_load": 0.096, "busy_all": 0.514, "mine_cpu": 0.0, "foreign_cpu": 0.514, "other_containers_cpu": 0.044, "mem_available_gib": 114.01, "pkg_temp_c": 68.0, "root_avail_gb": 222.2} +{"ts": 1794025434.372, "seconds": 5.001, "load1": 1.0, "busy_p": 0.338, "busy_e_free": 0.11, "busy_e_load": 0.076, "busy_all": 0.524, "mine_cpu": 0.0, "foreign_cpu": 0.524, "other_containers_cpu": 0.081, "mem_available_gib": 113.98, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025439.386, "seconds": 5.015, "load1": 1.0, "busy_p": 0.54, "busy_e_free": 0.239, "busy_e_load": 0.187, "busy_all": 0.967, "mine_cpu": 0.0, "foreign_cpu": 0.967, "other_containers_cpu": 0.06, "mem_available_gib": 113.9, "pkg_temp_c": 62.0, "root_avail_gb": 222.1} +{"ts": 1794025444.388, "seconds": 5.001, "load1": 1.24, "busy_p": 2.805, "busy_e_free": 0.724, "busy_e_load": 0.648, "busy_all": 4.177, "mine_cpu": 3.156, "foreign_cpu": 1.021, "other_containers_cpu": 0.12, "mem_available_gib": 113.59, "pkg_temp_c": 63.0, "root_avail_gb": 221.2} +{"ts": 1794025449.389, "seconds": 5.001, "load1": 1.3, "busy_p": 1.314, "busy_e_free": 0.42, "busy_e_load": 0.376, "busy_all": 2.109, "mine_cpu": 1.065, "foreign_cpu": 1.044, "other_containers_cpu": 0.075, "mem_available_gib": 113.52, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025454.39, "seconds": 5.001, "load1": 1.19, "busy_p": 0.392, "busy_e_free": 0.132, "busy_e_load": 0.136, "busy_all": 0.66, "mine_cpu": 0.137, "foreign_cpu": 0.523, "other_containers_cpu": 0.054, "mem_available_gib": 113.54, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025459.392, "seconds": 5.001, "load1": 1.1, "busy_p": 0.378, "busy_e_free": 0.126, "busy_e_load": 0.13, "busy_all": 0.634, "mine_cpu": 0.113, "foreign_cpu": 0.521, "other_containers_cpu": 0.042, "mem_available_gib": 113.53, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025464.393, "seconds": 5.001, "load1": 1.09, "busy_p": 0.324, "busy_e_free": 0.168, "busy_e_load": 0.108, "busy_all": 0.6, "mine_cpu": 0.035, "foreign_cpu": 0.565, "other_containers_cpu": 0.091, "mem_available_gib": 113.54, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025469.408, "seconds": 5.015, "load1": 1.0, "busy_p": 0.305, "busy_e_free": 0.122, "busy_e_load": 0.12, "busy_all": 0.546, "mine_cpu": 0.041, "foreign_cpu": 0.505, "other_containers_cpu": 0.054, "mem_available_gib": 113.54, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025474.409, "seconds": 5.002, "load1": 0.92, "busy_p": 0.31, "busy_e_free": 0.14, "busy_e_load": 0.138, "busy_all": 0.588, "mine_cpu": 0.034, "foreign_cpu": 0.554, "other_containers_cpu": 0.06, "mem_available_gib": 113.56, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025479.411, "seconds": 5.001, "load1": 1.01, "busy_p": 0.558, "busy_e_free": 0.2, "busy_e_load": 0.174, "busy_all": 0.932, "mine_cpu": 0.05, "foreign_cpu": 0.882, "other_containers_cpu": 0.07, "mem_available_gib": 114.71, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025484.412, "seconds": 5.001, "load1": 1.17, "busy_p": 1.14, "busy_e_free": 0.252, "busy_e_load": 0.194, "busy_all": 1.586, "mine_cpu": 0.064, "foreign_cpu": 1.522, "other_containers_cpu": 0.079, "mem_available_gib": 113.4, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025489.413, "seconds": 5.001, "load1": 1.15, "busy_p": 0.402, "busy_e_free": 0.148, "busy_e_load": 0.126, "busy_all": 0.676, "mine_cpu": 0.048, "foreign_cpu": 0.628, "other_containers_cpu": 0.041, "mem_available_gib": 113.44, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025494.414, "seconds": 5.001, "load1": 1.06, "busy_p": 0.354, "busy_e_free": 0.15, "busy_e_load": 0.1, "busy_all": 0.604, "mine_cpu": 0.037, "foreign_cpu": 0.567, "other_containers_cpu": 0.093, "mem_available_gib": 113.42, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025499.429, "seconds": 5.014, "load1": 1.38, "busy_p": 0.682, "busy_e_free": 0.207, "busy_e_load": 0.189, "busy_all": 1.079, "mine_cpu": 0.367, "foreign_cpu": 0.712, "other_containers_cpu": 0.055, "mem_available_gib": 113.42, "pkg_temp_c": 62.0, "root_avail_gb": 220.8} +{"ts": 1794025504.43, "seconds": 5.001, "load1": 1.51, "busy_p": 1.036, "busy_e_free": 0.414, "busy_e_load": 0.294, "busy_all": 1.744, "mine_cpu": 1.102, "foreign_cpu": 0.642, "other_containers_cpu": 0.094, "mem_available_gib": 113.4, "pkg_temp_c": 63.0, "root_avail_gb": 220.3} +{"ts": 1794025509.431, "seconds": 5.001, "load1": 1.39, "busy_p": 0.302, "busy_e_free": 0.134, "busy_e_load": 0.14, "busy_all": 0.576, "mine_cpu": 0.046, "foreign_cpu": 0.53, "other_containers_cpu": 0.06, "mem_available_gib": 113.42, "pkg_temp_c": 61.0, "root_avail_gb": 220.3} +{"ts": 1794025514.433, "seconds": 5.002, "load1": 1.27, "busy_p": 0.32, "busy_e_free": 0.14, "busy_e_load": 0.144, "busy_all": 0.604, "mine_cpu": 0.045, "foreign_cpu": 0.559, "other_containers_cpu": 0.062, "mem_available_gib": 113.44, "pkg_temp_c": 62.0, "root_avail_gb": 220.3} +{"ts": 1794025519.434, "seconds": 5.001, "load1": 1.25, "busy_p": 0.71, "busy_e_free": 0.398, "busy_e_load": 0.192, "busy_all": 1.3, "mine_cpu": 0.0, "foreign_cpu": 1.3, "other_containers_cpu": 0.044, "mem_available_gib": 114.44, "pkg_temp_c": 65.0, "root_avail_gb": 220.6} +{"ts": 1794025524.435, "seconds": 5.001, "load1": 1.15, "busy_p": 0.46, "busy_e_free": 0.126, "busy_e_load": 0.086, "busy_all": 0.672, "mine_cpu": 0.001, "foreign_cpu": 0.671, "other_containers_cpu": 0.08, "mem_available_gib": 114.43, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025529.447, "seconds": 5.012, "load1": 1.22, "busy_p": 0.928, "busy_e_free": 0.251, "busy_e_load": 0.223, "busy_all": 1.403, "mine_cpu": 0.001, "foreign_cpu": 1.402, "other_containers_cpu": 0.053, "mem_available_gib": 114.14, "pkg_temp_c": 66.0, "root_avail_gb": 222.2} +{"ts": 1794025534.448, "seconds": 5.001, "load1": 1.2, "busy_p": 0.67, "busy_e_free": 0.222, "busy_e_load": 0.19, "busy_all": 1.082, "mine_cpu": 0.0, "foreign_cpu": 1.082, "other_containers_cpu": 0.07, "mem_available_gib": 114.06, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025539.449, "seconds": 5.001, "load1": 1.35, "busy_p": 0.524, "busy_e_free": 0.206, "busy_e_load": 0.138, "busy_all": 0.868, "mine_cpu": 0.0, "foreign_cpu": 0.868, "other_containers_cpu": 0.063, "mem_available_gib": 113.95, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025544.45, "seconds": 5.001, "load1": 1.24, "busy_p": 0.31, "busy_e_free": 0.146, "busy_e_load": 0.114, "busy_all": 0.57, "mine_cpu": 0.0, "foreign_cpu": 0.57, "other_containers_cpu": 0.064, "mem_available_gib": 113.96, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025549.451, "seconds": 5.001, "load1": 1.14, "busy_p": 0.316, "busy_e_free": 0.168, "busy_e_load": 0.16, "busy_all": 0.644, "mine_cpu": 0.0, "foreign_cpu": 0.644, "other_containers_cpu": 0.04, "mem_available_gib": 113.86, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025554.452, "seconds": 5.001, "load1": 1.37, "busy_p": 1.07, "busy_e_free": 0.328, "busy_e_load": 0.256, "busy_all": 1.654, "mine_cpu": 0.0, "foreign_cpu": 1.654, "other_containers_cpu": 0.098, "mem_available_gib": 113.74, "pkg_temp_c": 62.0, "root_avail_gb": 221.7} +{"ts": 1794025559.464, "seconds": 5.013, "load1": 1.34, "busy_p": 0.307, "busy_e_free": 0.128, "busy_e_load": 0.124, "busy_all": 0.559, "mine_cpu": 0.027, "foreign_cpu": 0.532, "other_containers_cpu": 0.047, "mem_available_gib": 113.72, "pkg_temp_c": 61.0, "root_avail_gb": 221.7} +{"ts": 1794025564.466, "seconds": 5.001, "load1": 1.23, "busy_p": 0.61, "busy_e_free": 0.272, "busy_e_load": 0.192, "busy_all": 1.074, "mine_cpu": 0.476, "foreign_cpu": 0.598, "other_containers_cpu": 0.077, "mem_available_gib": 113.69, "pkg_temp_c": 63.0, "root_avail_gb": 221.4} +{"ts": 1794025569.467, "seconds": 5.001, "load1": 1.37, "busy_p": 1.312, "busy_e_free": 0.468, "busy_e_load": 0.338, "busy_all": 2.117, "mine_cpu": 1.125, "foreign_cpu": 0.992, "other_containers_cpu": 0.074, "mem_available_gib": 113.6, "pkg_temp_c": 63.0, "root_avail_gb": 221.3} +{"ts": 1794025574.469, "seconds": 5.002, "load1": 1.34, "busy_p": 1.016, "busy_e_free": 0.364, "busy_e_load": 0.14, "busy_all": 1.519, "mine_cpu": 0.755, "foreign_cpu": 0.764, "other_containers_cpu": 0.057, "mem_available_gib": 113.54, "pkg_temp_c": 63.0, "root_avail_gb": 220.8} +{"ts": 1794025579.47, "seconds": 5.002, "load1": 1.4, "busy_p": 1.729, "busy_e_free": 0.676, "busy_e_load": 0.338, "busy_all": 2.743, "mine_cpu": 2.006, "foreign_cpu": 0.737, "other_containers_cpu": 0.061, "mem_available_gib": 113.49, "pkg_temp_c": 62.0, "root_avail_gb": 220.3} +{"ts": 1794025584.471, "seconds": 5.001, "load1": 1.36, "busy_p": 0.738, "busy_e_free": 0.462, "busy_e_load": 0.316, "busy_all": 1.516, "mine_cpu": 0.0, "foreign_cpu": 1.516, "other_containers_cpu": 0.091, "mem_available_gib": 114.43, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794025589.485, "seconds": 5.013, "load1": 1.25, "busy_p": 0.345, "busy_e_free": 0.15, "busy_e_load": 0.118, "busy_all": 0.612, "mine_cpu": 0.0, "foreign_cpu": 0.612, "other_containers_cpu": 0.051, "mem_available_gib": 114.39, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} diff --git a/bench/wildct/results/2026-11-07/runs/H18w/wait_s b/bench/wildct/results/2026-11-07/runs/H18w/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/campaign.log b/bench/wildct/results/2026-11-07/runs/campaign.log index 66cf4dac..896b5dfa 100644 --- a/bench/wildct/results/2026-11-07/runs/campaign.log +++ b/bench/wildct/results/2026-11-07/runs/campaign.log @@ -25,3 +25,7 @@ == H52w lock taken 2026-11-07T04:03:54Z == H52w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h52w 2026-11-07T04:03:54Z == H52w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:05:00Z +== H18w waiting for the lock 2026-11-07T04:22:23Z +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z diff --git a/docs/intel/partner-update-2026-11.md b/docs/intel/partner-update-2026-11.md index 0e2ab7a6..ad2804c8 100644 --- a/docs/intel/partner-update-2026-11.md +++ b/docs/intel/partner-update-2026-11.md @@ -58,8 +58,8 @@ All drafts are unsent and need a maintainer's approval before anything is filed. SUPPORTED rows, each scoped to its host: - On the Xeon 6741P: the CPU engine choice [ledger:H59], DRAM-resident serving of Qwen3.8-Flash-Next [ledger:H60], and H25 replicated [ledger:H25x]. -- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-to-wildcts-i7-13700). -- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w]. +- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-and-h18w-to-wildcts-i7-13700). +- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w], and H18 replicated, agent sandboxes on its E-cores behind default-deny egress [ledger:H18w]. ## Two results that help EAT @@ -73,7 +73,7 @@ v0.28.0 is past the v0.24.0 CPU attention hang that stalled EAT 1.0.3's stock ar - **The Keycloak argv secret.** F3 stays REFUTED. `k3d-gpu eas up` now patches enterprise-inference's `keycloak_config` role so that neither the admin password nor the client secret is a `kcadm.sh` argument ([State and secrets](eas.md#state-and-secrets)). The patch is checked with a `ps` sampler against a stand-in for kcadm; no live run has checked it yet. A fresh `eas up --agent-toolkit` with F3's secret scan would. - **Intel's default install in a guest (H62).** A re-run needs the `cpu: auto` sizing to leave room for EAT's sandbox router, or a guest shape where it does. - **k3d-gpu gaps found on the Xeon.** `/dev/dri/by-path` for oneCCL across B70s (H53); the fit refusing rather than falling back to the CPU, and no DRAM bandwidth cached on a new host (H58); placing one-card arms on one card (H54b, H54c). -- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H52w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. +- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H18w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. - **EAT 1.1.0 through `eat-up.sh`.** Pinned since 2026-10-29 and pre-registered as F4; not run yet. - **EAS upstream moves.** EAS #61 merged on 2026-10-30 (EAS `main` `0f7bc2b`). The pin stays at `97f6157`; moving past #61 needs patch 0002 regenerated. EAS #50 and #52 (OpenShift) are open and would move 0006. EAS #71 (teardown hang) is open; `eas down` uses its workaround. - **ERAG and Arc under EAS.** EAS's Enterprise RAG layer is not wired into `k3d-gpu eas` yet, and EAS has no Intel GPU path on this branch; Arc and Gaudi stay with `k3d-gpu agent up`. diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index b251544d..92b12015 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,17 +1,17 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d and H52w runs (82 rows). +Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d, H52w and H18w runs (82 rows). ## Abstract k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w and H18w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. -- Twenty hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w to wildct, an i7-13700. H52w is the newest: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. +- 21 hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w and H18w to wildct, an i7-13700. H18w is the newest: H18 replicated on that host. In 3 of 3 runs `k3d-gpu sandbox up` from the release kept the sandbox on an E-core (CPU 17) behind default-deny egress: the host's tailscale and LAN addresses were refused from the sandbox and reached from a control pod, and `pip install --user` from PyPI worked [ledger:H18w]. Before it, H52w: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. - Twenty are REFUTED. Six of them changed shipped defaults. The newest, H28x, is H28 replicated on the Xeon 6741P: eight pods that mmap one 1 GiB model through an image volume again shared one page-cache copy (PSS 132.6 MiB per pod, 1.00x the size on the node against 9.00x for per-pod copies), but they were ready in 0.714 of the copy arm's time, not the 0.5 or less the row needed, so only the memory part replicated [ledger:H28x]. Before it, H7x is H7 run to its full protocol on the Xeon 6741P: at matched 16-bit weights (Qwen3-4B, 16 CPUs), OVMS 2026.4.1 decoded at 1.22x the rate of EAT 1.1.0's stock vLLM CPU engine (v0.28.0) in all five block pairs, so OVMS's decode lead is not only weight precision; vLLM prefilled the long prompt faster [ledger:H7x]. Before it, H58 is the zero-touch thesis on the Xeon 6741P, with the two serving pairs holding about half of every B70. With no flag, `k3d-gpu up` advertised three of the four B70s and no DSA work queue. `k3d-gpu agent up --model` then refused all three models. Gemma 4 26B-A4B and Qwen3.8-27B went to OVMS on the GPU, where their int4 IRs missed the best card's free-memory budget by 0.1 and 0.5 GiB. Qwen3.8-Flash-Next was refused because no DRAM bandwidth was cached. So EAT's banking flow never came up [ledger:H58]. Before it, H52b is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. - 34 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in a KVM guest on the Xeon 6741P) is INCONCLUSIVE because Intel's own install did not finish: enterprise-inference's `cpu: auto` sized the Gemma 4 pod to 76 of the guest's 95.4 allocatable CPUs, so EAT's sandbox router could not be scheduled, twice, and no hold ran. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H55 (SLO-gated admission of an E-core neighbour on the i9-13900T) is INCONCLUSIVE: all 25 runs completed on a busy host, but in one round the controller's run closed three request windows, one short of its gate; not gated, the controller held the neighbour at its lowest quota in four of five rounds, and no arm kept prefill within the row's bar of quiet. H59d (the engine choice on the i9-13900T: llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter against OVMS with OpenVINO's base-model int4 IR) is INCONCLUSIVE because the llama.cpp arm was not deliverable in two of its four holds: in both holds where it followed OVMS, its server was OOMKilled at the fit's 45Gi pod limit (4 x 262,144 tokens), while the same render served in both holds where it ran first. Not gated, in the one performance hold it served it decoded 1.05x OVMS per request at four concurrent requests, and GSM8K read 50 against 49 of 50. So with no backend set, the i9-13900T keeps OVMS. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 8 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 7 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -48,7 +48,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p ## Results -### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w to wildct's i7-13700) +### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w and H18w to wildct's i7-13700) | Row | Result | Scope limit | |---|---|---| @@ -71,6 +71,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H60 | Qwen3.8-Flash-Next (125B MoE, ggml-org IQ4_NL GGUF in tmpfs) served by llama.cpp v0.8.0 from the Xeon 6741P's DRAM decoded 16.00 tok/s at one stream, 0.312 of the 51.3 tok/s ceiling that 305 GB/s gives at 5.94 GB read per token. A same-hold Gemma 4 QAT Q4_0 control on the same build read 0.379 (48.52 tok/s), so the ratio was 0.823, CI 0.819-0.831, inside the 0.7-1.3 band; GSM8K items 0-49 scored 50/50 [ledger:H60] | One hold, 48 threads on the 48 physical cores, CPU only, greedy, thinking off for Flash-Next (the control opened its thinking block). The control's decode window starts about 3 tokens after its first chunk, so its rate reads about 1% high; timed from the first chunk, the ratio is 0.833. The ceiling uses a fixed STREAM figure; package watts include about 130 W of other tenants' idle draw | | H40 | The agent harness reproduces the BFCL board: bf16 Qwen3-4B-Instruct-2507 on vLLM CUDA scored 88.0% on the fixed 100-id subset in BFCL's own mode (board-weighted 84.9%, band 77.9-91.9%), every category inside its binomial interval [ledger:H40] | A check of the harness, not of the hardware; one 100-id subset, one run, on the NVIDIA dGPU | | H52w | On wildct (an i7-13700 with two NVIDIA cards and a UHD 770 iGPU), a plain `k3d-gpu up` with the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`) detected every CPU engine present and nothing phantom: the hybrid P/E cores with the balloons policy (`k3d-gpu test`'s P-core pod on CPU 0, its E-core pod on CPU 17) and the launcher's six ISA labels (AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS); no AVX-512, AMX or NUMA label, and no DSA, IAA, QAT, NPU or Gaudi resource or pod. Both GPU vendors were named as detected and opted out, no GPU resource was allocatable and no pod mounted a `/dev` path [ledger:H52w] | One hold, release 0.1.89. Detection with the GPUs opted out: says nothing about a GPU reaching a pod on this host | +| H18w | H18 replicated on wildct (an i7-13700, E-cores 16-23) with the release node image and the box owner's GPU opt-out: in 3 of 3 runs `k3d-gpu sandbox up` exited 0 with the sandbox pod on CPU 17; TCP from the sandbox to the host's tailscale (`100.66.38.121:22`) and LAN (`192.168.8.238:22`) addresses was refused, with 2 packets on kube-router's drop-log rule, while a control pod without the policy connected to both; `pip install --user tomli_w==1.2.0` from PyPI installed into `/tmp/.local`, where the import found it [ledger:H18w] | One hold, release 0.1.89, agent-sandbox v1.0.8, kube-router on k3s v1.37.1, IPv4 targets only | ### REFUTED (all but H5, H7x, H27, H27x, H28x, H34x, H43, H45, H48, H50, H51 and H52b changed a default)