From b8a83fac7e6b1cd1ab72fd3f417417980e23c189 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Fri, 6 Nov 2026 20:13:32 -0800 Subject: [PATCH 1/4] bench(wildct): H52w run on release 0.1.89, SUPPORTED H52w, the first wildct hold, ran on 2026-11-07 from 04:03:54Z to 04:05:00Z under bench/wildct/hold.sh with release 0.1.89 and the box owner's GPU opt-out. Every row of the pre-registered detection table matches: hybrid active with the balloons policy and the test pods on CPU 0 and CPU 17, exactly the six launcher ISA labels, no AVX-512, AMX or NUMA label, nvidia and intel named as detected and opted out, no GPU resource, no GPU plugin pod, no /dev hostPath, and the host inventory unchanged. The production snapshot and firewall rules read the same before and after the hold. The records are under bench/wildct/results/2026-11-07/runs/H52w/ (no kubeconfigs), the host reads in host/h52w.txt. The ledger row and its run section, the technical report's counts and RESUME.md's status follow. Rule W7 now holds the campaign at 0.1.89. --- EXPERIMENTS.md | 89 ++++- bench/wildct/RESUME.md | 41 ++- bench/wildct/results/2026-11-07/host/h52w.txt | 169 ++++++++++ bench/wildct/results/2026-11-07/runs/H52w.out | 4 + .../2026-11-07/runs/H52w/launcher.diff | 0 .../2026-11-07/runs/H52w/out/addons.txt | 23 ++ .../2026-11-07/runs/H52w/out/allocatable.json | 1 + .../2026-11-07/runs/H52w/out/daemonsets.txt | 10 + .../2026-11-07/runs/H52w/out/detect-auto.txt | 3 + .../2026-11-07/runs/H52w/out/detect.txt | 2 + .../2026-11-07/runs/H52w/out/doctor.txt | 21 ++ .../results/2026-11-07/runs/H52w/out/down.txt | 7 + .../results/2026-11-07/runs/H52w/out/env.txt | 5 + .../2026-11-07/runs/H52w/out/gpu-scan.txt | 34 ++ .../2026-11-07/runs/H52w/out/inventory-1.txt | 28 ++ .../2026-11-07/runs/H52w/out/inventory-2.txt | 28 ++ .../2026-11-07/runs/H52w/out/inventory.diff | 0 .../runs/H52w/out/node-container.txt | 1 + .../2026-11-07/runs/H52w/out/node.json | 310 ++++++++++++++++++ .../results/2026-11-07/runs/H52w/out/pods.txt | 9 + .../results/2026-11-07/runs/H52w/out/rc.txt | 9 + .../results/2026-11-07/runs/H52w/out/test.txt | 5 + .../results/2026-11-07/runs/H52w/out/up.txt | 33 ++ .../2026-11-07/runs/H52w/out/version.txt | 1 + bench/wildct/results/2026-11-07/runs/H52w/pid | 1 + .../results/2026-11-07/runs/H52w/post.txt | 17 + .../results/2026-11-07/runs/H52w/pre.txt | 17 + bench/wildct/results/2026-11-07/runs/H52w/rc | 1 + .../results/2026-11-07/runs/H52w/sample.jsonl | 13 + .../results/2026-11-07/runs/H52w/wait_s | 1 + .../results/2026-11-07/runs/campaign.log | 4 + docs/intel/technical-report.md | 11 +- 32 files changed, 874 insertions(+), 24 deletions(-) create mode 100644 bench/wildct/results/2026-11-07/host/h52w.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w.out create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/launcher.diff create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/addons.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/allocatable.json create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/daemonsets.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/detect-auto.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/detect.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/doctor.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/down.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/env.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/gpu-scan.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/inventory-1.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/inventory-2.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/inventory.diff create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/node-container.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/node.json create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/pods.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/rc.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/test.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/up.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/out/version.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/pid create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/post.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/pre.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/rc create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/sample.jsonl create mode 100644 bench/wildct/results/2026-11-07/runs/H52w/wait_s diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 632a9475..a3617323 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -84,7 +84,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57db | Inside EAT's banking flow (B24) with Gemma 4 26B-A4B QAT on the i9-13900T, k3d-gpu auto on the P-cores delivers far more tasks within budget than EAT stock CPU (vLLM CPU bf16 on the QAT bf16 weights, `xeon-values.yaml` entry `cpu-gemma4-26b-a4b`), interleaved pairs under rule L3. | Written 2026-10-20: not runnable here as registered (memory), so INCONCLUSIVE (not run) is expected (0.9). | [H57d pre-registration](#h57d-pre-registration-gemma-4-26b-a4b-qat-on-the-amx-less-client-cpu-i9-13900t-2026-10-20), H57db: runs only if the memory gate passes at a hold's start (MemAvailable >= 100 GiB: 51.6 GB of bf16 weights resident plus `VLLM_CPU_KVCACHE_SPACE` 40 GiB), with H50's procedure, rule L3 (PR #157) and at most 1 hold, where fewer than 16 complete pairs read INCONCLUSIVE by L3; otherwise INCONCLUSIVE (not run), with the reading recorded. Not run ([H57d run](#h57d-run-i9-13900t-2026-10-21)): memory gate failed, MemAvailable 42.9 GB at the campaign's start (< 100 GiB). | INCONCLUSIVE | | H57dp | On the i9-13900T P-cores under the shared host's own load, llama.cpp with Gemma 4 QAT Q4_0 and the QAT MTP drafter decodes faster unpinned (8 threads, no `-C`/`--cpu-strict`) than strictly pinned one thread per physical P-core: N=1 ratio >= 1.10. | Written 2026-10-21, after H57d, before any H57dp measurement. SUPPORTED (0.5); Ld-mtp-u/Ld-mtp in [1.0, 1.8] (0.6). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): 2 holds of at most 75 min, H57d's protocol. SUPPORTED: pooled median ratio >= 1.10, CI lower bound > 1.0, faster in both holds. REFUTED: CI upper bound < 1.10. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched, foreign P-core load spread 2.97 > 2.0 CPU-s/s (Od 0.45, Ld-mtp 3.42, Ld-mtp-u 3.11), so the scorer stops before the statistic. Not gated: Ld-mtp-u/Ld-mtp 1.108 (95% CI 1.072-1.142), faster in both holds (1.108, 1.100); Ld-mtp-u 13.39, Ld-mtp 12.09 tok/s pooled N=1; affinity as registered. | INCONCLUSIVE | | H57de | H57da with the unpinned llama.cpp arm: no arm decodes Gemma 4 QAT at N=1 5% or more faster than k3d-gpu's OVMS pick Od on the i9-13900T P-cores. | Written 2026-10-21, before any H57de measurement. REFUTED (0.5); Ld-mtp-u/Od in [1.0, 1.6] (0.5). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): same holds as H57dp. REFUTED: Ld-mtp-u/Od CI lower bound >= 1.05 and faster in both holds. SUPPORTED: CI upper bound < 1.05. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched (the same gate, spread 2.97 > 2.0). Od also ran without the P-core balloon (cpuset 0-15,17-31: the runner's `agent manifest` call named no cluster, so after #183 it missed the hold's kube context), so it was not the registered Od. Not gated: Ld-mtp-u/Od 1.369 (1.345-1.396), Od 9.78 tok/s pooled N=1. | INCONCLUSIVE | -| H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. | PENDING | +| H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. Result ([H52w run](#h52w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign). `up` took 32 s, named nvidia and intel as opted out and printed `Accelerators: hybrid`; the balloons policy pod Running, `k3d-gpu test`'s P-core pod on CPU 0 and its E-core pod on CPU 17; the launcher's ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT and CETSS, no AVX512 or AMX label from any source; no NUMA label; allocatable cpu, ephemeral-storage, hugepages, memory and pods only; no GPU plugin pod, no `/dev` hostPath and no GPU resource (`GPU-SCAN clean`); the node container `gpus=[]`; the inventory unchanged. Every row matches. | SUPPORTED | | H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. | PENDING | | H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | @@ -8819,6 +8819,93 @@ before `H52w` starts. Then step 3 runs again, and so does the wiring smoke, as ` old node images names every release the campaign installed. Step 1 and the smoke pass, so the next step is step 6's first hold, `H52w`. +### H52w run (wildct, i7-13700, 2026-11-07) + +**Reproduce:** + +```text +# provenance +date: 2026-11-07T04:03:54Z to 04:05:00Z (one hold; lock free, no load wait: load1 0.72) +hypothesis: H52w +host: wildct, reached by IP 100.66.38.121: Intel Core i7-13700 (family 6 model 183, microcode + 0x137; 0x133 at registration), 8 P-cores (CPUs 0-15) and 8 E-cores (CPUs 16-23), 1 NUMA + node, 125 GiB RAM, Manjaro; Docker 29.7.2, live-restore off, cgroup systemd/2 (step 1, 03:32:22Z) +host kernel: 6.6.156-2-MANJARO +accelerators: host inventory: UHD 770 8086:a780 (i915), RTX 4000 SFF Ada 10de:27b0 and RTX 2000 Ada + 10de:28b0 (nvidia); no DSA, IAA, QAT, NPU or Gaudi; no GPU used (rule W8) +k3d-gpu: release 0.1.89 (aur-v0.1.89, .deb sha256 8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc), + unpacked in ~/k3dgpu-bench-wildct/k3d-gpu; still the newest release when the hold started +bench scripts: main at 85a7bc86, shipped as the archive repo-85a7bc8668d1 (its launcher equals 0.1.89's apart + from VERSION; its bench/ and scripts/ equal main's at 28b1e70b) +node image: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4, + k3s v1.37.1+k3s1, containerd 2.3.4-k3s1; k3d v5.9.0, helm v4.3.1, kubectl v1.37.2 +runtime: k3d-gpu test pods busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec +environment: K3D_GPU_VENDORS=auto,-nvidia,-intel, K3D_GPU_K3S_ARGS=, K3D_GPU_CLUSTER=k3dgpu-w-h52, + K3D_GPU_RAPL_EXEC="docker exec k3dgpu-w-rapl" (rule W4; H52w reads no energy) +host load: load1 0.61-1.14, foreign CPU 0.47-1.77 CPU-s/s, MemAvailable 113.6-114.5 GiB, package 61-67 C, + / 207 GB available (sample.py, 13 samples) +``` + +Command, on wildct: RESUME.md step 6's `launch H52w 1800 3 h52w`, which runs `bench/wildct/hold.sh H52w +1800 3 bash bench/wildct/row.sh h52w`, and so `bench/wildct/h52w.sh ~/k3dgpu-bench-wildct/runs/H52w/out`, +the registered procedure. Records (inventory twice, `detect` as found and with the opt-out, `doctor`, +`up`, the node object, pods, DaemonSets, k3s's addons, the pod GPU scan, `k3d-gpu test`, `down`, the +hold's samples; the kubeconfigs were not copied): +[bench/wildct/results/2026-11-07/runs/H52w/](bench/wildct/results/2026-11-07/runs/H52w/). The host +before and after the hold: [bench/wildct/results/2026-11-07/host/h52w.txt](bench/wildct/results/2026-11-07/host/h52w.txt). + +`up` took 32 s. It printed "opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); +not deployed, not advertised", then `Accelerators: hybrid` and `Accelerators schedulable: hybrid=1`. +The node was Ready with no taint. The pod wait (`ready.sh`) passed after 20 s: seven pods 1/1 Running +with 0 restarts, and the gateway API CRD job Completed. Node allocatable: `cpu` 24, +`ephemeral-storage`, `hugepages-1Gi` 0, `hugepages-2Mi` 0, `memory` and `pods` 110; nothing extended. + +| Engine | Expected (pre-registered) | Read | Row | +|---|---|---|---| +| Core types | hybrid active, `k3d-gpu.io/hybrid-cpu=true`, the balloons policy pod Running; `k3d-gpu test`'s P-core pod inside 0-15 and E-core pod inside 16-23 | `Accelerators: hybrid`; label `k3d-gpu.io/hybrid-cpu=true`; `nri-resource-policy-balloons` 1/1 Running; the P-core pod's `Cpus_allowed_list` 0, the E-core pod's 17 | yes | +| CPU ISA | launcher labels `cpu-cpuid.` AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS (`=true`), exactly these; NFD's own `cpu-cpuid.*` labels recorded; no `AVX512*` label from any source | the launcher's `--node-label`s (the node's `k3s.io/node-args`) set exactly those six `cpu-cpuid.` labels; NFD's own 49 `cpu-cpuid.` labels (its `feature-labels` annotation) are in `node.json`, none of them one of the six; no `AVX512` label | yes | +| AMX | amx not active; no `AMX*` label | not in `Accelerators`; no AMX label | yes | +| NUMA | numa not active; no `k3d-gpu.io/numa-nodes` label | not active; no NUMA label | yes | +| NVIDIA GPU | detected by `detect` as found; `up` names it as opted out; no `nvidia.com/gpu` allocatable, NVIDIA plugin not running, no `--gpus` on the node container | `detect` as found: `nvidia nvidia.com/gpu`; `up` names it as opted out; no `nvidia.com/` resource; `nvidia-device-plugin-daemonset` 0 desired, no pod; node container `gpus=[]`, `devices=[]`, runtime `runc`, `restarts=0` | yes | +| Intel GPU | detected by `detect` as found; `up` names it as opted out; no `gpu.intel.com/*` allocatable; no `intel-gpu-plugin` or DRA pod | `detect` as found: `intel gpu.intel.com/i915`; `up` names it as opted out; no `gpu.intel.com/` resource; `up` passed `--disable intel-gpu-plugin --disable intel-gpu-dra`, and neither is in k3s's addon list; no Intel GPU pod (`intel-gpu-plugin-wsl` 0 desired) | yes | +| DSA, IAA, QAT, NPU, Gaudi | none active, named or advertised | none in `Accelerators`, none named by `up`, no resource; the plugin DaemonSets the image ships (`intel-dsa-plugin`, `intel-iaa-plugin`, `intel-qat-plugin`, `intel-npu-plugin`, `habanalabs-device-plugin-daemonset`) 0 desired, no pod | yes | +| Pods | no pod mounts a hostPath under `/dev/dri`, `/dev/nvidia*` or `/dev`, and no container asks for a GPU resource (`gpu-scan.txt` "clean") | 8 pods; no hostPath under `/dev` (the balloons pod's `/sys`, `/proc`, NRI directories and the kubelet's pod-resources directory; NFD's `/sys`, `/proc/swaps` and its host mounts under `/k3d-gpu/host`); no `nvidia.com/` or `gpu.intel.com/` request; `GPU-SCAN clean` | yes | +| Host | the second inventory equals the first apart from its first line | `inventory unchanged` (`inventory.diff` empty) | yes | + +**Verdict: SUPPORTED.** Every row matches. On a client Raptor Lake host with two NVIDIA cards and an +Intel iGPU, a plain `up` with the box owner's opt-out found the hybrid cores and the six ISA extensions +the launcher labels, and nothing the host does not have. It named both GPU vendors as detected and +opted out, and nothing in the cluster mounted or asked for a GPU. Each part of the prediction held: +hybrid and the test pods' placement (0.8), the six launcher labels with no AVX512 or AMX label (0.85), +no NUMA, DSA, IAA, QAT, NPU or Gaudi label, resource or pod (0.9; their plugin DaemonSet objects exist +with 0 pods, as on every k3d-gpu cluster), and the GPU opt-out (0.85). + +Not scored: + +- NFD describes the GPUs from PCI sysfs in labels. Its own: `feature.node.kubernetes.io/pci-0300_10de.present`, + `pci-0380_8086.present` and `pci-0380_8086.sriov.capable`. From the node image's NodeFeatureRules + `intel-dp-devices` and `intel-gpu-platform-labeling` (`share/node-feature-discovery.yml`): + `intel.feature.node.kubernetes.io/gpu=true` and `gpu.intel.com/device-id.0380-a780.present=true` + with `.count=1`. They are labels, not resources: no plugin that selects them ran, and nothing was + allocatable. The launcher's `nvidia.com/gpu.present=false` keeps the NVIDIA plugin off the node. The + table does not rule labels out, so they are recorded here and not scored. +- `up` annotated the node `k3d-gpu.io/favored-cpus=8-11`. `doctor` also reported the E-core clusters + 16-19 and 20-23 and no `rdt_a`. +- Host facts that moved since the registration, none of them in a row's expectation: microcode 0x137 + (0x133), kernel 6.6.156-2 (6.6.144), and the i915 iGPU is now card0/renderD128 (the registration's + table says card1/renderD129), with the NVIDIA cards on card1 and card2. + +Production box. The snapshot before the hold (04:02Z) and after it (04:06Z) read the same: 18 +containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the same port codes. The +line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and +after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` +cluster or container, no process of the hold and no network or volume of ours was left, and +`live.lock` was free. The host's loaded module count read 224 before the hold and at 04:11Z, +with all 18 modules of `/etc/modules-load.d/k3dgpu-wildct.conf` loaded and `kernel.modprobe` empty. + +Rule W7: this was the campaign's first hold, so every later wildct row runs release 0.1.89. Holds +used: 1 of at most 12. The next row in the registered order is H18w. + ### H5 amendment A1: stage-1 harness and definitions (2026-10-19) Written 2026-10-19T22:30-07:00, before any H5 measurement. It fixes what the diff --git a/bench/wildct/RESUME.md b/bench/wildct/RESUME.md index 3a500e15..1870ae39 100644 --- a/bench/wildct/RESUME.md +++ b/bench/wildct/RESUME.md @@ -1,16 +1,21 @@ # wildct resume pack How to resume the wildct campaign ([EXPERIMENTS.md](../../EXPERIMENTS.md), "Wildct -pre-registration", rules W1-W8, and wildct amendments R1, P1, R2, R3, R4, R5 and R6). This file -changes no rule. Run it top to bottom, except that step 3b (ship `origin/main` to wildct) runs -first when the `repo` on wildct predates `bench/wildct/nfprobe.sh` (H10 runs it). Nothing past -step 1 runs until every step 1 check passes. - -## Where things stand (2026-11-07, after smoke5) - -- **Ready for the first hold, `H52w` (step 6, row 1).** The kernel modules k3s needs are loaded - and persisted, step 1 passes on release 0.1.89, and the wiring smoke `smoke5` passed. No hold - has run; nothing has been measured. +pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5 and R6, and the "H52w +run" section). This file changes no rule. Run it top to bottom, except that step 3b (ship +`origin/main` to wildct) runs first when the `repo` on wildct predates `bench/wildct/nfprobe.sh` +(H10 runs it). Nothing past step 1 runs until every step 1 check passes. + +## Where things stand (2026-11-07, after H52w) + +- **Next: row 2, `H18w` (step 6).** Row 1, `H52w`, ran on 2026-11-07 (04:03:54Z to 04:05:00Z, + rc 0, every line of `out/rc.txt` a pass) and is scored SUPPORTED (EXPERIMENTS.md "H52w run"; + records in `results/2026-11-07/runs/H52w/`, the host before and after in + `results/2026-11-07/host/h52w.txt`). The host's containers, start times, restart counts, port + codes and firewall rules read the same before and after it, and nothing of the hold was left. + Holds used: 1 of at most 12. +- **Release frozen at 0.1.89 (rule W7).** `H52w`, the campaign's first hold, ran 0.1.89, so every + later hold runs it too. Step 3 does not run again, and neither does step 5. - **Kernel modules** (each approved by the box owner on 2026-10-29; `kernel.modprobe` stays empty): - `vxlan` and `br_netfilter` (amendment R3; `results/2026-10-29/host/modules.txt`); - `xt_comment xt_statistic xt_multiport xt_limit xt_NFLOG xt_physdev xt_pkttype ipt_REJECT @@ -65,7 +70,7 @@ step 1 runs until every step 1 check passes. `networks-before.txt` and the verify logs; - `runs/`: `smoke1*`, `smoke2*`, `diag-smoke2*`, `smoke3*` (with `smoke3-node.log`, the node's log, and `smoke3-node-excerpt.txt`), `smoke4*` (with `smoke4-live.txt`), `smoke5*`, - `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones renamed + `H52w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones renamed `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, `install-0.1.89.log`, `setup-images-*.log`, `IMAGE` and `campaign.log`; - `diag/`: `diag-up.sh`, the command `diag-smoke2` ran; `iptprobe-in.sh`, an ad-hoc form of @@ -79,15 +84,14 @@ step 1 runs until every step 1 check passes. - Images: `cryptoandcoffee/k3d-gpu:0.1.89`, `ghcr.io/k3d-io/k3d-tools:5.9.0`, the five refs of `setup-images.sh`, and `python:3.13-slim@sha256:70729b46...` (the RAPL reader's image until 2026-10-29). `cryptoandcoffee/k3d-gpu:0.1.71`, `:0.1.79` and `:0.1.81` were removed in step 3c. -- **Never existed.** No inference request. No measurement. `smoke1`, `smoke2`, `diag-smoke2`, - `smoke3`, `smoke4` and `smoke5` are discarded and disclosed (amendments R1, R2, R3, R5 and R6); - `smoke5` is the campaign's passing wiring smoke for 0.1.89. +- **Never existed.** No inference request. No measurement before `H52w`. `smoke1`, `smoke2`, + `diag-smoke2`, `smoke3`, `smoke4` and `smoke5` are discarded and disclosed (amendments R1, R2, + R3, R5 and R6); `smoke5` is the campaign's passing wiring smoke for 0.1.89. - **Never touched.** No GPU, nor anything that was on the host before the campaign, apart from the 18 kernel modules (and their three dependencies) and their `modules-load.d` file, which the box owner approved. -- **Newest release on 2026-11-07: k3d-gpu 0.1.89**, installed in step 3c. Rule W7: the first hold - runs the newest release. If a newer one is cut before `H52w` starts, step 3 runs again, and so - does step 5, as `smoke6`. +- **The campaign's release: k3d-gpu 0.1.89**, installed in step 3c and still the newest when + `H52w` started (rule W7: the first hold runs the newest release; later holds keep it). - `.deb` sha256: `8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc`. - Node image: `cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4`. - Its launcher and `share/` equal main's at `85a7bc86`, apart from the version line. @@ -175,7 +179,8 @@ timeout 30 docker ps -a --filter name=k3dgpu-w- --filter name=k3d-gpu-bench-load ## 3. Upgrade to the newest release -Rule W7: the newest k3d-gpu release at the first hold. No hold has run yet, so upgrade now. +Rule W7: the newest k3d-gpu release at the first hold. The first hold, `H52w`, ran 0.1.89 on +2026-11-07, so this step does not run again; it is kept as the record of how 0.1.89 was installed. **3a [c]. Pick the release and check it matches main.** diff --git a/bench/wildct/results/2026-11-07/host/h52w.txt b/bench/wildct/results/2026-11-07/host/h52w.txt new file mode 100644 index 00000000..d42c4645 --- /dev/null +++ b/bench/wildct/results/2026-11-07/host/h52w.txt @@ -0,0 +1,169 @@ +# wildct around the H52w hold, 2026-11-07 (UTC). Read-only reads as andrew, except the firewall +# dumps (iptables-save, ip6tables-save and nft list ruleset, as root). +# The production snapshot: running containers, published-port HTTP codes from the host, and each +# running container's StartedAt and RestartCount. + +== state before the hold (2026-11-07T04:03Z) +## date 2026-11-07T04:03:05+00:00 +## boot: 2026-10-29 05:11:09 +## kernel: 6.6.156-2-MANJARO +## disk +/dev/nvme0n1p2 3667G 3273G 207G 95% / +## mem +MemTotal: 131524088 kB +MemAvailable: 120102312 kB +## loadavg: 0.32 0.43 0.51 2/3671 34072 +VERSION="0.1.89" # kept in lockstep with packaging/aur PKGBUILD by auto-release.yml +## repo -> repo-85a7bc8668d1 COMMIT=85a7bc8668d18a3233496f0a3a23b09bb63ee20f +## runs/IMAGE: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +## tools +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +## clusters (k3d cluster list; names only) +wtm (the box owner's; exited; not this campaign's). No k3dgpu-w-* cluster. +## our containers +ps rc=0 +## live.lock +-rw-r--r-- 1 andrew andrew 0 Nov 6 22:34 /home/andrew/k3dgpu-bench-wildct/live.lock +lock-free +hold procs rc=1 +convert pid=24108 alive=no rc=0 +diag-smoke2 pid=17199 alive=no rc=1 +smoke2 pid=32217 alive=no rc=1 +smoke3 pid=62226 alive=no rc=0 +smoke4 pid=48184 alive=no rc=0 +smoke5 pid=31149 alive=no rc=0 +## modules of /etc/modules-load.d/k3dgpu-wildct.conf +loaded: 18 of 18; missing: 0 +modprobe=[] +## newest k3d-gpu release at 04:03Z (gh release list): aur-v0.1.89 (2026-11-07T00:23:41Z), as installed + +== production snapshot before the hold (2026-11-07T04:02Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== production snapshot after the hold (2026-11-07T04:06Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== comparison +production snapshot: identical +iptables-save rules, packet counters stripped: identical +ip6tables-save rules, packet counters stripped: identical +nft list ruleset, counters stripped: identical + +== after-hold checks (RESUME.md step 6) +== after H52w 2026-11-07T04:05:26Z +0 +== H52w lock taken 2026-11-07T04:03:54Z +== H52w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h52w 2026-11-07T04:03:54Z +== H52w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:05:00Z +list rc=0 +ps rc=0 +strays rc=0 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +lock-free +net rc=1 (grep found no k3dgpu network) +vol rc=1 (grep found no k3dgpu volume) diff --git a/bench/wildct/results/2026-11-07/runs/H52w.out b/bench/wildct/results/2026-11-07/runs/H52w.out new file mode 100644 index 00000000..4aaa3ca6 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w.out @@ -0,0 +1,4 @@ +== H52w waiting for the lock 2026-11-07T04:03:54Z +== H52w lock taken 2026-11-07T04:03:54Z +== H52w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h52w 2026-11-07T04:03:54Z +== H52w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:05:00Z diff --git a/bench/wildct/results/2026-11-07/runs/H52w/launcher.diff b/bench/wildct/results/2026-11-07/runs/H52w/launcher.diff new file mode 100644 index 00000000..e69de29b diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/addons.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/addons.txt new file mode 100644 index 00000000..fffa9c11 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/addons.txt @@ -0,0 +1,23 @@ +NAMESPACE NAME SOURCE CHECKSUM +kube-system aggregated-metrics-reader /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml 63b058ebe88b8179b459777318be49b591fefe629d9258b10974aeed224b5530 +kube-system auth-delegator /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml f55fee16219349d1d803b7025f9fde2b6fca741e3900bfedcf64f34dc1c23786 +kube-system auth-reader /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml 97a74f054fe2972fcc6ffb909224d9cb27f136cc152983c9ac96ff45579015c2 +kube-system ccm /var/lib/rancher/k3s/server/manifests/ccm.yaml 15c8482702cd79ec145e960ab92791a0e73b6e7577df7729ae8b523483e3cc93 +kube-system coredns /var/lib/rancher/k3s/server/manifests/coredns.yaml 6e4ed729369f032bb0c11be07a896af9e35b9503a9a027c570facb58e2ca4118 +kube-system gateway-api-crd /var/lib/rancher/k3s/server/manifests/gateway-api-crd.yaml f4bf3801d1e4dd67459fb31673fa8260963e721520d92e9fa0c337a9d7ec7e2c +kube-system gaudi-device-plugin /var/lib/rancher/k3s/server/manifests/gaudi-device-plugin.yaml 8559d298ea9e8d1a453cdbf46bea3358ce6d50a50510c729d6a335e44e4a3671 +kube-system intel-dsa-plugin /var/lib/rancher/k3s/server/manifests/intel-dsa-plugin.yaml 4c1542e627a6704b1894e4853958e8714f556a1de24f5e70bfb4665dac08904b +kube-system intel-gpu-wsl-plugin /var/lib/rancher/k3s/server/manifests/intel-gpu-wsl-plugin.yaml 14e12a5eee107fe94cbc2ed344663f35c669a5b3bfb3f3b15a5559c440060e22 +kube-system intel-iaa-plugin /var/lib/rancher/k3s/server/manifests/intel-iaa-plugin.yaml fb60b16b8ec772c875b25de112fd6c9ca0a1fc3495687ba2c2f0ab0273ba4173 +kube-system intel-npu-plugin /var/lib/rancher/k3s/server/manifests/intel-npu-plugin.yaml 6f9f7ab0525d04934b20808e563138e6955a32fcb05fd97da39923e79602cc51 +kube-system intel-qat-plugin /var/lib/rancher/k3s/server/manifests/intel-qat-plugin.yaml e3119a61b6f37b11a6f6829c69617d7d993abaef4ffba245def18cb7970c9d33 +kube-system local-storage /var/lib/rancher/k3s/server/manifests/local-storage.yaml ff345e02b4c7b3aaf398c14a042bec30ad304f47f2d292ba7d45c17fd10eb2aa +kube-system metrics-apiservice /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml 03266df7891b56dfffebb855a031b524ed20c6736846417ccd491dd91c6c0ec3 +kube-system metrics-server-deployment /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml 528152585ac18dd47c180b31bf667288cab779caa23516d094a2417b527c5ef1 +kube-system metrics-server-service /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml 0ba8f8a9133f38c226e33384b8019a882440480cb8b7b45b0cd9f62bebb06a6d +kube-system node-feature-discovery /var/lib/rancher/k3s/server/manifests/node-feature-discovery.yaml ecfb44e89ff944ec8191b74d13ff4d7d22721167dd1f41fdea1d2a6658e1621d +kube-system nri-balloons /var/lib/rancher/k3s/server/manifests/nri-balloons.yaml 97e43d9f823d2f35586ce42e7919aa5d47ec628df5b1a2c5f8244cb3ef4990f6 +kube-system nvidia-device-plugin /var/lib/rancher/k3s/server/manifests/nvidia-device-plugin.yaml 0588b1bcd89a9663ea43ab2cbc71fa47dc6d875bd44e651105ce2fcfbc82128e +kube-system resource-reader /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml d7e21b07d9edf7a1d4e169342f470405970621de59864f998dd84c2048fb9e64 +kube-system rolebindings /var/lib/rancher/k3s/server/manifests/rolebindings.yaml 6b4dc430eb4c8fc1a94efd18efa74647f561326923254c23b3001ef488457c1f +kube-system runtimes /var/lib/rancher/k3s/server/manifests/runtimes.yaml 4ea3be58c76250c61e65a257c8279b42d11568f913233eda98ac0d77ef2a8e37 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/allocatable.json b/bench/wildct/results/2026-11-07/runs/H52w/out/allocatable.json new file mode 100644 index 00000000..87735766 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/allocatable.json @@ -0,0 +1 @@ +{"cpu":"24","ephemeral-storage":"3934360281088","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"131421688Ki","pods":"110"} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/daemonsets.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/daemonsets.txt new file mode 100644 index 00000000..1e92c9bb --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/daemonsets.txt @@ -0,0 +1,10 @@ +NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE +habana-system habanalabs-device-plugin-daemonset 0 0 0 0 0 habana.ai/gaudi.present=true 45s +kube-system intel-dsa-plugin 0 0 0 0 0 intel.feature.node.kubernetes.io/dsa=true,kubernetes.io/arch=amd64 45s +kube-system intel-gpu-plugin-wsl 0 0 0 0 0 k3d-gpu.io/wsl-gpu=true,kubernetes.io/arch=amd64 45s +kube-system intel-iaa-plugin 0 0 0 0 0 intel.feature.node.kubernetes.io/iaa=true,kubernetes.io/arch=amd64 44s +kube-system intel-npu-plugin 0 0 0 0 0 intel.feature.node.kubernetes.io/npu=true,kubernetes.io/arch=amd64 44s +kube-system intel-qat-plugin 0 0 0 0 0 intel.feature.node.kubernetes.io/qat=true,kubernetes.io/arch=amd64 44s +kube-system nri-resource-policy-balloons 1 1 1 1 1 kubernetes.io/os=linux 41s +kube-system nvidia-device-plugin-daemonset 0 0 0 0 0 38s +node-feature-discovery nfd-worker 1 1 1 1 1 44s diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/detect-auto.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/detect-auto.txt new file mode 100644 index 00000000..1f68319d --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/detect-auto.txt @@ -0,0 +1,3 @@ +nvidia nvidia.com/gpu +intel gpu.intel.com/i915 +hybrid k3d-gpu.io/hybrid-cpu=true diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/detect.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/detect.txt new file mode 100644 index 00000000..44547a49 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/detect.txt @@ -0,0 +1,2 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +hybrid k3d-gpu.io/hybrid-cpu=true diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/doctor.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/doctor.txt new file mode 100644 index 00000000..ce2aaa79 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/doctor.txt @@ -0,0 +1,21 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Checking docker... + ok: docker running +==> Checking k3d... + ok: k3d version v5.9.0 +==> Checking kubectl... + ok +==> Detected accelerators: hybrid +==> Intel CPU built-in accelerators... + ok: CPUID (read through the CPUID instruction): none of AVX10, AMX-FP16, AMX-COMPLEX, AVX-VNNI-INT8, AVX-VNNI-INT16, AVX-NE-CONVERT, AVX-IFMA, APX_F — ISA /proc/cpuinfo does not print (Intel ISE 319433-061 table 1-3: CPUID.(7,1), leaf 24H); the AMX-FP16/AMX-COMPLEX labels follow these bits + ok: CPU ISA: AVXVNNI CETIBT CETSS GFNI SHA VAES — up labels nodes feature.node.kubernetes.io/cpu-cpuid.=true (NFD names) + ok: hybrid CPU: P-cores 0-15, E-cores 16-23 — up deploys the NRI balloons policy; label pods k3d-gpu.io/cores=performance|efficient|lowpower|ecluster|pcore-physical|latency + ok: E-core clusters (shared L2): 16-19 20-23 — k3d-gpu.io/cores=ecluster gives a pod 4 E-cores of one cluster + ok: physical P-cores (one thread per core): 0,2,4,6,8,10,12,14 — k3d-gpu.io/cores=pcore-physical and latency show pods one thread per core + ok: favored CPUs 8-11 (highest_perf 67, next 65) — up annotates nodes k3d-gpu.io/favored-cpus=8-11 and points the latency balloon at them + info: noisy-neighbour isolation: cpu-cap — no rdt_a flag, so cache and DRAM bandwidth stay shared; give a bandwidth-heavy neighbour a CPU limit. Holding it to one E-core L2 cluster (k3d-gpu.io/cores=ecluster) is not recommended: H26 measures it against the same workers spread over the E-cores (EXPERIMENTS.md H26) +==> Experimental features (auto unless K3D_GPU_EXPERIMENTAL=off or )... + ok: dra available (auto) + ok: image-volume available (auto) + +==> All checks passed. diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/down.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/down.txt new file mode 100644 index 00000000..722d4175 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/down.txt @@ -0,0 +1,7 @@ +==> Deleting k3d cluster 'k3dgpu-w-h52'... +INFO[0000] Deleting cluster 'k3dgpu-w-h52' +INFO[0000] Deleting cluster network 'k3d-k3dgpu-w-h52' +INFO[0001] Deleting 1 attached volumes... +INFO[0001] Removing cluster details from default kubeconfig... +INFO[0001] Removing standalone kubeconfig file (if there is one)... +INFO[0001] Successfully deleted cluster k3dgpu-w-h52! diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/env.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/env.txt new file mode 100644 index 00000000..740a0230 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/env.txt @@ -0,0 +1,5 @@ +K3D_GPU_CLUSTER=k3dgpu-w-h52 +K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1% +K3D_GPU_RAPL_EXEC=docker exec k3dgpu-w-rapl +K3D_GPU_SHARE=/home/andrew/k3dgpu-bench-wildct/k3d-gpu/usr/share/k3d-gpu +K3D_GPU_VENDORS=auto,-nvidia,-intel diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/gpu-scan.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/gpu-scan.txt new file mode 100644 index 00000000..df0785af --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/gpu-scan.txt @@ -0,0 +1,34 @@ +kube-system/coredns-577d995dff-c5gdf hostPath +kube-system/coredns-577d995dff-c5gdf hostPath +kube-system/coredns-577d995dff-c5gdf hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/helm-install-gateway-api-crd-xmcqs hostPath +kube-system/local-path-provisioner-6858d854cf-6ncrq hostPath +kube-system/local-path-provisioner-6858d854cf-6ncrq hostPath +kube-system/metrics-server-778c4649b4-rx7ts hostPath +kube-system/metrics-server-778c4649b4-rx7ts hostPath +kube-system/nri-resource-policy-balloons-d4qkh hostPath /var/lib/nri-resource-policy +kube-system/nri-resource-policy-balloons-d4qkh hostPath /sys +kube-system/nri-resource-policy-balloons-d4qkh hostPath /proc +kube-system/nri-resource-policy-balloons-d4qkh hostPath /var/run/nri-resource-policy +kube-system/nri-resource-policy-balloons-d4qkh hostPath /var/run/nri +kube-system/nri-resource-policy-balloons-d4qkh hostPath /var/lib/kubelet/pod-resources +kube-system/nri-resource-policy-balloons-d4qkh hostPath +node-feature-discovery/nfd-gc-8d4595cbc-p7vd8 hostPath +node-feature-discovery/nfd-master-74b44b767d-chpdk hostPath +node-feature-discovery/nfd-master-74b44b767d-chpdk hostPath +node-feature-discovery/nfd-worker-w2kjq hostPath /k3d-gpu/host/boot +node-feature-discovery/nfd-worker-w2kjq hostPath /proc/swaps +node-feature-discovery/nfd-worker-w2kjq hostPath /k3d-gpu/host/etc +node-feature-discovery/nfd-worker-w2kjq hostPath /sys +node-feature-discovery/nfd-worker-w2kjq hostPath /k3d-gpu/host/lib +node-feature-discovery/nfd-worker-w2kjq hostPath /k3d-gpu/host/lib +node-feature-discovery/nfd-worker-w2kjq hostPath /etc/kubernetes/node-feature-discovery/features.d/ +node-feature-discovery/nfd-worker-w2kjq hostPath +node-feature-discovery/nfd-worker-w2kjq hostPath +GPU-SCAN clean diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-1.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-1.txt new file mode 100644 index 00000000..2a526fa7 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-1.txt @@ -0,0 +1,28 @@ +# date 2026-11-07T04:03:54Z host wildct kernel 6.6.156-2-MANJARO +cpu: 13th Gen Intel(R) Core(TM) i7-13700 family 6 model 183 microcode 0x137 +online 0-23 nodes node0 smt=1 +cpu_core 0-15 cpu_atom 16-23 cpu_lowpower +cpu0:0-1:5100000 cpu1:0-1:5100000 cpu2:2-3:5100000 cpu3:2-3:5100000 cpu4:4-5:5100000 cpu5:4-5:5100000 cpu6:6-7:5100000 cpu7:6-7:5100000 cpu8:8-9:5200000 cpu9:8-9:5200000 cpu10:10-11:5200000 cpu11:10-11:5200000 cpu12:12-13:5100000 cpu13:12-13:5100000 cpu14:14-15:5100000 cpu15:14-15:5100000 cpu16:16:4100000 cpu17:17:4100000 cpu18:18:4100000 cpu19:19:4100000 cpu20:20:4100000 cpu21:21:4100000 cpu22:22:4100000 cpu23:23:4100000 +flags: avx avx2 avx_vnni f16c fma gfni ibt sha_ni user_shstk vaes +rapl package-0 pl1_uw 35000000 pl2_uw 40000000 +mem_total_kb 131524088 thp [always] madvise never +## pci display, 3D and accelerator functions (class 03xx, 0b40, 1200) +0000:00:02.0 0x8086:0xa780 class=0x038000 driver=i915 +0000:01:00.0 0x10de:0x27b0 class=0x030000 driver=nvidia +0000:02:00.0 0x10de:0x28b0 class=0x030000 driver=nvidia +## drm (sysfs names only) +card0 0x8086:0xa780 i915 +card1 0x10de:0x27b0 nvidia +card2 0x10de:0x28b0 nvidia +renderD128 0x8086:0xa780 i915 +renderD129 0x10de:0x27b0 nvidia +renderD130 0x10de:0x28b0 nvidia +## device directories present (not opened) +/dev/dsa: absent +/dev/iax: absent +/dev/accel: absent +/dev/qat_adf_ctl: absent +/dev/nvidiactl: present +/dev/dri: present +## modules +i915 nvidia nvidia_uvm diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-2.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-2.txt new file mode 100644 index 00000000..9016b175 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/inventory-2.txt @@ -0,0 +1,28 @@ +# date 2026-11-07T04:04:58Z host wildct kernel 6.6.156-2-MANJARO +cpu: 13th Gen Intel(R) Core(TM) i7-13700 family 6 model 183 microcode 0x137 +online 0-23 nodes node0 smt=1 +cpu_core 0-15 cpu_atom 16-23 cpu_lowpower +cpu0:0-1:5100000 cpu1:0-1:5100000 cpu2:2-3:5100000 cpu3:2-3:5100000 cpu4:4-5:5100000 cpu5:4-5:5100000 cpu6:6-7:5100000 cpu7:6-7:5100000 cpu8:8-9:5200000 cpu9:8-9:5200000 cpu10:10-11:5200000 cpu11:10-11:5200000 cpu12:12-13:5100000 cpu13:12-13:5100000 cpu14:14-15:5100000 cpu15:14-15:5100000 cpu16:16:4100000 cpu17:17:4100000 cpu18:18:4100000 cpu19:19:4100000 cpu20:20:4100000 cpu21:21:4100000 cpu22:22:4100000 cpu23:23:4100000 +flags: avx avx2 avx_vnni f16c fma gfni ibt sha_ni user_shstk vaes +rapl package-0 pl1_uw 35000000 pl2_uw 40000000 +mem_total_kb 131524088 thp [always] madvise never +## pci display, 3D and accelerator functions (class 03xx, 0b40, 1200) +0000:00:02.0 0x8086:0xa780 class=0x038000 driver=i915 +0000:01:00.0 0x10de:0x27b0 class=0x030000 driver=nvidia +0000:02:00.0 0x10de:0x28b0 class=0x030000 driver=nvidia +## drm (sysfs names only) +card0 0x8086:0xa780 i915 +card1 0x10de:0x27b0 nvidia +card2 0x10de:0x28b0 nvidia +renderD128 0x8086:0xa780 i915 +renderD129 0x10de:0x27b0 nvidia +renderD130 0x10de:0x28b0 nvidia +## device directories present (not opened) +/dev/dsa: absent +/dev/iax: absent +/dev/accel: absent +/dev/qat_adf_ctl: absent +/dev/nvidiactl: present +/dev/dri: present +## modules +i915 nvidia nvidia_uvm diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/inventory.diff b/bench/wildct/results/2026-11-07/runs/H52w/out/inventory.diff new file mode 100644 index 00000000..e69de29b diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/node-container.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/node-container.txt new file mode 100644 index 00000000..a792184f --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/node-container.txt @@ -0,0 +1 @@ +sha256:5811390ea98da96ebe01a3338291d4f462bb8c16e0063c99939944190547b8c8 cryptoandcoffee/k3d-gpu:0.1.89 privileged=true devices=[] runtime=runc gpus=[] restarts=0 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/node.json b/bench/wildct/results/2026-11-07/runs/H52w/out/node.json new file mode 100644 index 00000000..7c6ca91c --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/node.json @@ -0,0 +1,310 @@ +{ + "apiVersion": "v1", + "items": [ + { + "apiVersion": "v1", + "kind": "Node", + "metadata": { + "annotations": { + "alpha.kubernetes.io/provided-node-ip": "172.19.0.2", + "flannel.alpha.coreos.com/backend-data": "{\"VNI\":1,\"VtepMAC\":\"e6:dc:29:36:32:d5\"}", + "flannel.alpha.coreos.com/backend-type": "vxlan", + "flannel.alpha.coreos.com/kube-subnet-manager": "true", + "flannel.alpha.coreos.com/public-ip": "172.19.0.2", + "k3d-gpu.io/favored-cpus": "8-11", + "k3s.io/hostname": "k3d-k3dgpu-w-h52-server-0", + "k3s.io/internal-ip": "172.19.0.2", + "k3s.io/node-args": "[\"server\",\"--disable\",\"traefik\",\"--disable\",\"intel-gpu-plugin\",\"--disable\",\"intel-gpu-dra\",\"--kubelet-arg\",\"eviction-hard=memory.available\\u003c100Mi,nodefs.available\\u003c2Gi,imagefs.available\\u003c2Gi,nodefs.inodesFree\\u003c1%\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.VAES=true\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.AVXVNNI=true\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.CETSS=true\",\"--node-label\",\"nvidia.com/gpu.present=false\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.CETIBT=true\",\"--node-label\",\"experimental.k3d-gpu.io/dra=true\",\"--node-label\",\"experimental.k3d-gpu.io/image-volume=true\",\"--node-label\",\"k3d-gpu.io/hybrid-cpu=true\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.GFNI=true\",\"--node-label\",\"feature.node.kubernetes.io/cpu-cpuid.SHA=true\",\"--tls-san\",\"0.0.0.0\"]", + "k3s.io/node-config-hash": "QM5ZSOE5LGLJJJMA2IFULLWLI7DRSVTDXSCZSSDKQT7DPF24YOJQ====", + "k3s.io/node-env": "{\"K3S_KUBECONFIG_OUTPUT\":\"/output/kubeconfig.yaml\",\"K3S_TOKEN\":\"********\"}", + "nfd.node.kubernetes.io/feature-labels": "cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.BHI_CTRL,cpu-cpuid.CMPXCHG8,cpu-cpuid.FLUSH_L1D,cpu-cpuid.FMA3,cpu-cpuid.FSRM,cpu-cpuid.FXSR,cpu-cpuid.FXSROPT,cpu-cpuid.HRESET,cpu-cpuid.HYBRID_CPU,cpu-cpuid.IA32_ARCH_CAP,cpu-cpuid.IA32_CORE_CAP,cpu-cpuid.IBPB,cpu-cpuid.IDPRED_CTRL,cpu-cpuid.KEYLOCKER,cpu-cpuid.LAHF,cpu-cpuid.MD_CLEAR,cpu-cpuid.MOVBE,cpu-cpuid.MOVDIR64B,cpu-cpuid.MOVDIRI,cpu-cpuid.OSXSAVE,cpu-cpuid.PCONFIG,cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES,cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS,cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES,cpu-cpuid.PSFD,cpu-cpuid.RRSBA_CTRL,cpu-cpuid.SERIALIZE,cpu-cpuid.SPEC_CTRL_SSBD,cpu-cpuid.STIBP,cpu-cpuid.STOSB_SHORT,cpu-cpuid.SYSCALL,cpu-cpuid.SYSEE,cpu-cpuid.TME,cpu-cpuid.VMX,cpu-cpuid.VPCLMULQDQ,cpu-cpuid.WAITPKG,cpu-cpuid.X86_64_V1,cpu-cpuid.X86_64_V2,cpu-cpuid.X86_64_V3,cpu-cpuid.X87,cpu-cpuid.XGETBV1,cpu-cpuid.XSAVE,cpu-cpuid.XSAVEC,cpu-cpuid.XSAVEOPT,cpu-cpuid.XSAVES,cpu-cstate.enabled,cpu-hardware_multithreading,cpu-model.family,cpu-model.hypervisor,cpu-model.id,cpu-model.vendor_id,cpu-pstate.scaling_governor,cpu-pstate.status,cpu-pstate.turbo,gpu.intel.com/device-id.0380-a780.count,gpu.intel.com/device-id.0380-a780.present,intel.feature.node.kubernetes.io/gpu,kernel-config.NO_HZ,kernel-config.NO_HZ_FULL,kernel-config.PREEMPT,kernel-version.full,kernel-version.major,kernel-version.minor,kernel-version.revision,memory-swap,pci-0300_10de.present,pci-0380_8086.present,pci-0380_8086.sriov.capable,storage-nonrotationaldisk,system-os_release.ID", + "node.alpha.kubernetes.io/ttl": "0", + "volumes.kubernetes.io/controller-managed-attach-detach": "true" + }, + "creationTimestamp": "2026-11-07T04:04:01Z", + "finalizers": [ + "wrangler.cattle.io/node" + ], + "labels": { + "beta.kubernetes.io/arch": "amd64", + "beta.kubernetes.io/instance-type": "k3s", + "beta.kubernetes.io/os": "linux", + "experimental.k3d-gpu.io/dra": "true", + "experimental.k3d-gpu.io/image-volume": "true", + "feature.node.kubernetes.io/cpu-cpuid.ADX": "true", + "feature.node.kubernetes.io/cpu-cpuid.AESNI": "true", + "feature.node.kubernetes.io/cpu-cpuid.AVX": "true", + "feature.node.kubernetes.io/cpu-cpuid.AVX2": "true", + "feature.node.kubernetes.io/cpu-cpuid.AVXVNNI": "true", + "feature.node.kubernetes.io/cpu-cpuid.BHI_CTRL": "true", + "feature.node.kubernetes.io/cpu-cpuid.CETIBT": "true", + "feature.node.kubernetes.io/cpu-cpuid.CETSS": "true", + "feature.node.kubernetes.io/cpu-cpuid.CMPXCHG8": "true", + "feature.node.kubernetes.io/cpu-cpuid.FLUSH_L1D": "true", + "feature.node.kubernetes.io/cpu-cpuid.FMA3": "true", + "feature.node.kubernetes.io/cpu-cpuid.FSRM": "true", + "feature.node.kubernetes.io/cpu-cpuid.FXSR": "true", + "feature.node.kubernetes.io/cpu-cpuid.FXSROPT": "true", + "feature.node.kubernetes.io/cpu-cpuid.GFNI": "true", + "feature.node.kubernetes.io/cpu-cpuid.HRESET": "true", + "feature.node.kubernetes.io/cpu-cpuid.HYBRID_CPU": "true", + "feature.node.kubernetes.io/cpu-cpuid.IA32_ARCH_CAP": "true", + "feature.node.kubernetes.io/cpu-cpuid.IA32_CORE_CAP": "true", + "feature.node.kubernetes.io/cpu-cpuid.IBPB": "true", + "feature.node.kubernetes.io/cpu-cpuid.IDPRED_CTRL": "true", + "feature.node.kubernetes.io/cpu-cpuid.KEYLOCKER": "true", + "feature.node.kubernetes.io/cpu-cpuid.LAHF": "true", + "feature.node.kubernetes.io/cpu-cpuid.MD_CLEAR": "true", + "feature.node.kubernetes.io/cpu-cpuid.MOVBE": "true", + "feature.node.kubernetes.io/cpu-cpuid.MOVDIR64B": "true", + "feature.node.kubernetes.io/cpu-cpuid.MOVDIRI": "true", + "feature.node.kubernetes.io/cpu-cpuid.OSXSAVE": "true", + "feature.node.kubernetes.io/cpu-cpuid.PCONFIG": "true", + "feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_CYCLES": "true", + "feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_INSTRUCTIONS": "true", + "feature.node.kubernetes.io/cpu-cpuid.PMU_FIXEDCOUNTER_REFCYCLES": "true", + "feature.node.kubernetes.io/cpu-cpuid.PSFD": "true", + "feature.node.kubernetes.io/cpu-cpuid.RRSBA_CTRL": "true", + "feature.node.kubernetes.io/cpu-cpuid.SERIALIZE": "true", + "feature.node.kubernetes.io/cpu-cpuid.SHA": "true", + "feature.node.kubernetes.io/cpu-cpuid.SPEC_CTRL_SSBD": "true", + "feature.node.kubernetes.io/cpu-cpuid.STIBP": "true", + "feature.node.kubernetes.io/cpu-cpuid.STOSB_SHORT": "true", + "feature.node.kubernetes.io/cpu-cpuid.SYSCALL": "true", + "feature.node.kubernetes.io/cpu-cpuid.SYSEE": "true", + "feature.node.kubernetes.io/cpu-cpuid.TME": "true", + "feature.node.kubernetes.io/cpu-cpuid.VAES": "true", + "feature.node.kubernetes.io/cpu-cpuid.VMX": "true", + "feature.node.kubernetes.io/cpu-cpuid.VPCLMULQDQ": "true", + "feature.node.kubernetes.io/cpu-cpuid.WAITPKG": "true", + "feature.node.kubernetes.io/cpu-cpuid.X86_64_V1": "true", + "feature.node.kubernetes.io/cpu-cpuid.X86_64_V2": "true", + "feature.node.kubernetes.io/cpu-cpuid.X86_64_V3": "true", + "feature.node.kubernetes.io/cpu-cpuid.X87": "true", + "feature.node.kubernetes.io/cpu-cpuid.XGETBV1": "true", + "feature.node.kubernetes.io/cpu-cpuid.XSAVE": "true", + "feature.node.kubernetes.io/cpu-cpuid.XSAVEC": "true", + "feature.node.kubernetes.io/cpu-cpuid.XSAVEOPT": "true", + "feature.node.kubernetes.io/cpu-cpuid.XSAVES": "true", + "feature.node.kubernetes.io/cpu-cstate.enabled": "true", + "feature.node.kubernetes.io/cpu-hardware_multithreading": "true", + "feature.node.kubernetes.io/cpu-model.family": "6", + "feature.node.kubernetes.io/cpu-model.hypervisor": "none", + "feature.node.kubernetes.io/cpu-model.id": "183", + "feature.node.kubernetes.io/cpu-model.vendor_id": "Intel", + "feature.node.kubernetes.io/cpu-pstate.scaling_governor": "powersave", + "feature.node.kubernetes.io/cpu-pstate.status": "active", + "feature.node.kubernetes.io/cpu-pstate.turbo": "true", + "feature.node.kubernetes.io/kernel-config.NO_HZ": "true", + "feature.node.kubernetes.io/kernel-config.NO_HZ_FULL": "true", + "feature.node.kubernetes.io/kernel-config.PREEMPT": "true", + "feature.node.kubernetes.io/kernel-version.full": "6.6.156-2-MANJARO", + "feature.node.kubernetes.io/kernel-version.major": "6", + "feature.node.kubernetes.io/kernel-version.minor": "6", + "feature.node.kubernetes.io/kernel-version.revision": "156", + "feature.node.kubernetes.io/memory-swap": "true", + "feature.node.kubernetes.io/pci-0300_10de.present": "true", + "feature.node.kubernetes.io/pci-0380_8086.present": "true", + "feature.node.kubernetes.io/pci-0380_8086.sriov.capable": "true", + "feature.node.kubernetes.io/storage-nonrotationaldisk": "true", + "feature.node.kubernetes.io/system-os_release.ID": "manjaro", + "gpu.intel.com/device-id.0380-a780.count": "1", + "gpu.intel.com/device-id.0380-a780.present": "true", + "intel.feature.node.kubernetes.io/gpu": "true", + "k3d-gpu.io/hybrid-cpu": "true", + "kubernetes.io/arch": "amd64", + "kubernetes.io/hostname": "k3d-k3dgpu-w-h52-server-0", + "kubernetes.io/os": "linux", + "node-role.kubernetes.io/control-plane": "true", + "node.kubernetes.io/instance-type": "k3s", + "nvidia.com/gpu.present": "false" + }, + "name": "k3d-k3dgpu-w-h52-server-0", + "resourceVersion": "783", + "uid": "b3b627f2-9745-4254-80a6-d911830e148e" + }, + "spec": { + "podCIDR": "10.42.0.0/24", + "podCIDRs": [ + "10.42.0.0/24" + ], + "providerID": "k3s://k3d-k3dgpu-w-h52-server-0" + }, + "status": { + "addresses": [ + { + "address": "172.19.0.2", + "type": "InternalIP" + }, + { + "address": "k3d-k3dgpu-w-h52-server-0", + "type": "Hostname" + } + ], + "allocatable": { + "cpu": "24", + "ephemeral-storage": "3934360281088", + "hugepages-1Gi": "0", + "hugepages-2Mi": "0", + "memory": "131421688Ki", + "pods": "110" + }, + "capacity": { + "cpu": "24", + "ephemeral-storage": "3936507764736", + "hugepages-1Gi": "0", + "hugepages-2Mi": "0", + "memory": "131524088Ki", + "pods": "110" + }, + "conditions": [ + { + "lastHeartbeatTime": "2026-11-07T04:04:32Z", + "lastTransitionTime": "2026-11-07T04:04:01Z", + "message": "kubelet has sufficient memory available", + "reason": "KubeletHasSufficientMemory", + "status": "False", + "type": "MemoryPressure" + }, + { + "lastHeartbeatTime": "2026-11-07T04:04:32Z", + "lastTransitionTime": "2026-11-07T04:04:01Z", + "message": "kubelet has no disk pressure", + "reason": "KubeletHasNoDiskPressure", + "status": "False", + "type": "DiskPressure" + }, + { + "lastHeartbeatTime": "2026-11-07T04:04:32Z", + "lastTransitionTime": "2026-11-07T04:04:01Z", + "message": "kubelet has sufficient PID available", + "reason": "KubeletHasSufficientPID", + "status": "False", + "type": "PIDPressure" + }, + { + "lastHeartbeatTime": "2026-11-07T04:04:32Z", + "lastTransitionTime": "2026-11-07T04:04:02Z", + "message": "kubelet is posting ready status", + "reason": "KubeletReady", + "status": "True", + "type": "Ready" + } + ], + "daemonEndpoints": { + "kubeletEndpoint": { + "Port": 10250 + } + }, + "declaredFeatures": [ + "ExtendWebSocketsToKubelet", + "InPlacePodLevelResourcesVerticalScaling", + "InPlacePodVerticalScalingInitContainers", + "RestartAllContainersOnContainerExits" + ], + "features": { + "supplementalGroupsPolicy": true + }, + "images": [ + { + "names": [ + "registry.k8s.io/nfd/node-feature-discovery@sha256:6fc5c9af9176e1acb5ce57a4a1d0c0d3eb25cd34f5bdd0efefebf0efe45a8379" + ], + "sizeBytes": 87537491 + }, + { + "names": [ + "docker.io/rancher/klipper-helm@sha256:5425a2b4613d2458dbdff99218e8d487ebb59c20e7a941f061c0777070f0aac1", + "docker.io/rancher/klipper-helm:v0.13.3-build20260727" + ], + "sizeBytes": 74172006 + }, + { + "names": [ + "ghcr.io/containers/nri-plugins/nri-resource-policy-balloons@sha256:bacbef6f002f432b9fbe61ff0c583536da7ae816300ca82921ee697d759a1e80" + ], + "sizeBytes": 36675793 + }, + { + "names": [ + "docker.io/rancher/mirrored-coredns-coredns@sha256:7efd3c635b03efd68c4e8398fc45f0d993d0e9ab016f72c1cefb0fd6d01aa286", + "docker.io/rancher/mirrored-coredns-coredns:1.14.7" + ], + "sizeBytes": 24261571 + }, + { + "names": [ + "docker.io/rancher/mirrored-metrics-server@sha256:d9862115e7c7881280d3d75ca26bda8ffc0fc213315979575bf23ce9826205c0", + "docker.io/rancher/mirrored-metrics-server:v0.9.0" + ], + "sizeBytes": 23258428 + }, + { + "names": [ + "docker.io/rancher/local-path-provisioner@sha256:e757967a5ec338f6a9b371c5a9688bedaa8c3578ea3dd4db329ea0084be0a86f", + "docker.io/rancher/local-path-provisioner:v0.0.37" + ], + "sizeBytes": 15066012 + }, + { + "names": [ + "docker.io/rancher/mirrored-pause@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4", + "docker.io/rancher/mirrored-pause:3.10.2" + ], + "sizeBytes": 320805 + } + ], + "nodeInfo": { + "architecture": "amd64", + "bootID": "ceab9bc3-622c-4141-8a21-f766e6754032", + "containerRuntimeVersion": "containerd://2.3.4-k3s1", + "kernelVersion": "6.6.156-2-MANJARO", + "kubeProxyVersion": "", + "kubeletVersion": "v1.37.1+k3s1", + "machineID": "", + "operatingSystem": "linux", + "osImage": "Ubuntu 26.04.1 LTS", + "runningInUserNamespace": false, + "swap": { + "capacity": 412316852224 + }, + "systemUUID": "8b8e6400-ac2b-11ee-bd49-3cd1af4a3300" + }, + "runtimeHandlers": [ + { + "features": { + "recursiveReadOnlyMounts": true, + "userNamespaces": true + }, + "name": "" + }, + { + "features": { + "recursiveReadOnlyMounts": true, + "userNamespaces": true + }, + "name": "nvidia" + }, + { + "features": { + "recursiveReadOnlyMounts": true, + "userNamespaces": true + }, + "name": "runc" + }, + { + "features": { + "recursiveReadOnlyMounts": false, + "userNamespaces": false + }, + "name": "runhcs-wcow-process" + } + ] + } + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/pods.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/pods.txt new file mode 100644 index 00000000..b4e65b28 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/pods.txt @@ -0,0 +1,9 @@ +NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +kube-system coredns-577d995dff-c5gdf 1/1 Running 0 40s 10.42.0.3 k3d-k3dgpu-w-h52-server-0 +kube-system helm-install-gateway-api-crd-xmcqs 0/1 Completed 0 40s 10.42.0.6 k3d-k3dgpu-w-h52-server-0 +kube-system local-path-provisioner-6858d854cf-6ncrq 1/1 Running 0 40s 10.42.0.2 k3d-k3dgpu-w-h52-server-0 +kube-system metrics-server-778c4649b4-rx7ts 1/1 Running 0 40s 10.42.0.7 k3d-k3dgpu-w-h52-server-0 +kube-system nri-resource-policy-balloons-d4qkh 1/1 Running 0 40s 10.42.0.5 k3d-k3dgpu-w-h52-server-0 +node-feature-discovery nfd-gc-8d4595cbc-p7vd8 1/1 Running 0 40s 10.42.0.4 k3d-k3dgpu-w-h52-server-0 +node-feature-discovery nfd-master-74b44b767d-chpdk 1/1 Running 0 40s 10.42.0.9 k3d-k3dgpu-w-h52-server-0 +node-feature-discovery nfd-worker-w2kjq 1/1 Running 0 41s 10.42.0.8 k3d-k3dgpu-w-h52-server-0 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/rc.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/rc.txt new file mode 100644 index 00000000..ddbc066e --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/rc.txt @@ -0,0 +1,9 @@ +detect(auto) rc=0 +detect rc=0 +doctor rc=0 +up rc=0 in 32 s +pods ready: yes after 20 s +GPU-SCAN clean +test rc=0 +down rc=0 +inventory unchanged diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/test.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/test.txt new file mode 100644 index 00000000..4de16754 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/test.txt @@ -0,0 +1,5 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Deploying performance-core test pod (busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec; host performance CPUs: 0-15)... +Cpus_allowed_list: 0 +==> Deploying efficient-core test pod (busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec; host efficient CPUs: 16-23)... +Cpus_allowed_list: 17 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/up.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/up.txt new file mode 100644 index 00000000..de946eed --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/up.txt @@ -0,0 +1,33 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-h52' with image cryptoandcoffee/k3d-gpu:0.1.89... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-h52' +INFO[0000] Created image volume k3d-k3dgpu-w-h52-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-h52-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-h52-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-h52' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-h52-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-h52' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-h52) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H52w/out/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-h52-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-h52-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H52w/out/version.txt b/bench/wildct/results/2026-11-07/runs/H52w/out/version.txt new file mode 100644 index 00000000..9a8d52ad --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/out/version.txt @@ -0,0 +1 @@ +k3d-gpu 0.1.89 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/pid b/bench/wildct/results/2026-11-07/runs/H52w/pid new file mode 100644 index 00000000..484467e8 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/pid @@ -0,0 +1 @@ +40856 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/post.txt b/bench/wildct/results/2026-11-07/runs/H52w/post.txt new file mode 100644 index 00000000..305fd2e4 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/post.txt @@ -0,0 +1,17 @@ +2026-11-07T04:05:00Z + 23:05:00 up 8 days, 18:53, 2 users, load average: 0.88, 0.64, 0.58 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 2.4 8054 andrew firefox + 1.9 9806 andrew WebExtensions + 1.6 29612 andrew rustdesk + 1.4 1 root systemd + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.6 12980 andrew 1password + 0.5 7035 andrew rustdesk + 0.3 54042 andrew localsearch-ext + total used free shared buff/cache available +Mem: 125 10 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/pre.txt b/bench/wildct/results/2026-11-07/runs/H52w/pre.txt new file mode 100644 index 00000000..fa60024f --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/pre.txt @@ -0,0 +1,17 @@ +2026-11-07T04:03:54Z + 23:03:54 up 8 days, 18:52, 2 users, load average: 0.72, 0.56, 0.55 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 2.5 8054 andrew firefox + 1.9 9806 andrew WebExtensions + 1.6 29612 andrew rustdesk + 1.4 1 root systemd + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.6 12980 andrew 1password + 0.5 7035 andrew rustdesk + 0.3 3313 root dockerd + total used free shared buff/cache available +Mem: 125 10 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/rc b/bench/wildct/results/2026-11-07/runs/H52w/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/rc @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/H52w/sample.jsonl b/bench/wildct/results/2026-11-07/runs/H52w/sample.jsonl new file mode 100644 index 00000000..39097732 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/sample.jsonl @@ -0,0 +1,13 @@ +{"ts": 1794024239.613, "seconds": 5.0, "load1": 0.66, "busy_p": 0.848, "busy_e_free": 0.484, "busy_e_load": 0.434, "busy_all": 1.766, "mine_cpu": 0.0, "foreign_cpu": 1.766, "other_containers_cpu": 0.075, "mem_available_gib": 114.27, "pkg_temp_c": 67.0, "root_avail_gb": 222.2} +{"ts": 1794024244.614, "seconds": 5.001, "load1": 0.61, "busy_p": 0.964, "busy_e_free": 0.324, "busy_e_load": 0.274, "busy_all": 1.562, "mine_cpu": 0.0, "foreign_cpu": 1.562, "other_containers_cpu": 0.074, "mem_available_gib": 114.09, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794024249.615, "seconds": 5.001, "load1": 0.64, "busy_p": 0.54, "busy_e_free": 0.202, "busy_e_load": 0.176, "busy_all": 0.918, "mine_cpu": 0.0, "foreign_cpu": 0.918, "other_containers_cpu": 0.058, "mem_available_gib": 114.02, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794024254.617, "seconds": 5.001, "load1": 0.75, "busy_p": 0.34, "busy_e_free": 0.148, "busy_e_load": 0.134, "busy_all": 0.622, "mine_cpu": 0.0, "foreign_cpu": 0.622, "other_containers_cpu": 0.094, "mem_available_gib": 114.02, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794024259.618, "seconds": 5.001, "load1": 0.93, "busy_p": 0.258, "busy_e_free": 0.104, "busy_e_load": 0.104, "busy_all": 0.466, "mine_cpu": 0.0, "foreign_cpu": 0.466, "other_containers_cpu": 0.043, "mem_available_gib": 114.0, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794024264.619, "seconds": 5.001, "load1": 0.85, "busy_p": 0.598, "busy_e_free": 0.256, "busy_e_load": 0.206, "busy_all": 1.06, "mine_cpu": 0.0, "foreign_cpu": 1.06, "other_containers_cpu": 0.048, "mem_available_gib": 113.87, "pkg_temp_c": 61.0, "root_avail_gb": 222.0} +{"ts": 1794024269.636, "seconds": 5.017, "load1": 0.79, "busy_p": 1.802, "busy_e_free": 0.53, "busy_e_load": 0.311, "busy_all": 2.643, "mine_cpu": 1.688, "foreign_cpu": 0.955, "other_containers_cpu": 0.082, "mem_available_gib": 113.69, "pkg_temp_c": 63.0, "root_avail_gb": 221.1} +{"ts": 1794024274.637, "seconds": 5.001, "load1": 0.8, "busy_p": 0.796, "busy_e_free": 0.45, "busy_e_load": 0.23, "busy_all": 1.476, "mine_cpu": 0.847, "foreign_cpu": 0.629, "other_containers_cpu": 0.067, "mem_available_gib": 113.65, "pkg_temp_c": 62.0, "root_avail_gb": 221.1} +{"ts": 1794024279.638, "seconds": 5.001, "load1": 1.14, "busy_p": 0.384, "busy_e_free": 0.096, "busy_e_load": 0.116, "busy_all": 0.596, "mine_cpu": 0.114, "foreign_cpu": 0.482, "other_containers_cpu": 0.053, "mem_available_gib": 113.66, "pkg_temp_c": 61.0, "root_avail_gb": 221.1} +{"ts": 1794024284.64, "seconds": 5.001, "load1": 1.05, "busy_p": 0.386, "busy_e_free": 0.112, "busy_e_load": 0.082, "busy_all": 0.58, "mine_cpu": 0.038, "foreign_cpu": 0.542, "other_containers_cpu": 0.086, "mem_available_gib": 113.65, "pkg_temp_c": 61.0, "root_avail_gb": 221.1} +{"ts": 1794024289.641, "seconds": 5.001, "load1": 1.04, "busy_p": 0.572, "busy_e_free": 0.172, "busy_e_load": 0.154, "busy_all": 0.898, "mine_cpu": 0.081, "foreign_cpu": 0.817, "other_containers_cpu": 0.041, "mem_available_gib": 113.62, "pkg_temp_c": 61.0, "root_avail_gb": 221.1} +{"ts": 1794024294.642, "seconds": 5.001, "load1": 0.96, "busy_p": 0.456, "busy_e_free": 0.184, "busy_e_load": 0.17, "busy_all": 0.81, "mine_cpu": 0.18, "foreign_cpu": 0.63, "other_containers_cpu": 0.049, "mem_available_gib": 113.61, "pkg_temp_c": 63.0, "root_avail_gb": 221.1} +{"ts": 1794024299.656, "seconds": 5.013, "load1": 0.88, "busy_p": 0.566, "busy_e_free": 0.184, "busy_e_load": 0.174, "busy_all": 0.924, "mine_cpu": 0.0, "foreign_cpu": 0.924, "other_containers_cpu": 0.065, "mem_available_gib": 114.45, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} diff --git a/bench/wildct/results/2026-11-07/runs/H52w/wait_s b/bench/wildct/results/2026-11-07/runs/H52w/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H52w/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/campaign.log b/bench/wildct/results/2026-11-07/runs/campaign.log index d00481fc..66cf4dac 100644 --- a/bench/wildct/results/2026-11-07/runs/campaign.log +++ b/bench/wildct/results/2026-11-07/runs/campaign.log @@ -21,3 +21,7 @@ == smoke5 lock taken 2026-11-07T03:34:15Z == smoke5 start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh smoke 2026-11-07T03:34:15Z == smoke5 rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T03:35:07Z +== H52w waiting for the lock 2026-11-07T04:03:54Z +== H52w lock taken 2026-11-07T04:03:54Z +== H52w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h52w 2026-11-07T04:03:54Z +== H52w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:05:00Z diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index d587d431..b251544d 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,17 +1,17 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55 and H59d runs (82 rows). +Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d and H52w runs (82 rows). ## Abstract k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. -- Nineteen hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P. H60 is the newest: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. +- Twenty hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w to wildct, an i7-13700. H52w is the newest: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. - Twenty are REFUTED. Six of them changed shipped defaults. The newest, H28x, is H28 replicated on the Xeon 6741P: eight pods that mmap one 1 GiB model through an image volume again shared one page-cache copy (PSS 132.6 MiB per pod, 1.00x the size on the node against 9.00x for per-pod copies), but they were ready in 0.714 of the copy arm's time, not the 0.5 or less the row needed, so only the memory part replicated [ledger:H28x]. Before it, H7x is H7 run to its full protocol on the Xeon 6741P: at matched 16-bit weights (Qwen3-4B, 16 CPUs), OVMS 2026.4.1 decoded at 1.22x the rate of EAT 1.1.0's stock vLLM CPU engine (v0.28.0) in all five block pairs, so OVMS's decode lead is not only weight precision; vLLM prefilled the long prompt faster [ledger:H7x]. Before it, H58 is the zero-touch thesis on the Xeon 6741P, with the two serving pairs holding about half of every B70. With no flag, `k3d-gpu up` advertised three of the four B70s and no DSA work queue. `k3d-gpu agent up --model` then refused all three models. Gemma 4 26B-A4B and Qwen3.8-27B went to OVMS on the GPU, where their int4 IRs missed the best card's free-memory budget by 0.1 and 0.5 GiB. Qwen3.8-Flash-Next was refused because no DRAM bandwidth was cached. So EAT's banking flow never came up [ledger:H58]. Before it, H52b is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. - 34 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in a KVM guest on the Xeon 6741P) is INCONCLUSIVE because Intel's own install did not finish: enterprise-inference's `cpu: auto` sized the Gemma 4 pod to 76 of the guest's 95.4 allocatable CPUs, so EAT's sandbox router could not be scheduled, twice, and no hold ran. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H55 (SLO-gated admission of an E-core neighbour on the i9-13900T) is INCONCLUSIVE: all 25 runs completed on a busy host, but in one round the controller's run closed three request windows, one short of its gate; not gated, the controller held the neighbour at its lowest quota in four of five rounds, and no arm kept prefill within the row's bar of quiet. H59d (the engine choice on the i9-13900T: llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter against OVMS with OpenVINO's base-model int4 IR) is INCONCLUSIVE because the llama.cpp arm was not deliverable in two of its four holds: in both holds where it followed OVMS, its server was OOMKilled at the fit's 45Gi pod limit (4 x 262,144 tokens), while the same render served in both holds where it ran first. Not gated, in the one performance hold it served it decoded 1.05x OVMS per request at four concurrent requests, and GSM8K read 50 against 49 of 50. So with no backend set, the i9-13900T keeps OVMS. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 9 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H52w, H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 8 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -48,7 +48,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p ## Results -### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P) +### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w to wildct's i7-13700) | Row | Result | Scope limit | |---|---|---| @@ -70,6 +70,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H28 | Eight pods that mmap one 1 GiB model through an image volume shared one page-cache copy: PSS 132.5 MiB per pod and 1.00x the model size on the node, against 1028.5 MiB and 9.00x for pods that copy it into an emptyDir; they were ready in 2980 ms vs 6333 ms (0.47x, CI 0.45-0.55, complete separation), under matched `typical` background load (amendment 27) [ledger:H28] | One host, overlayfs, N = 8, a 1 GiB file read page by page, CPU-resident weights. The time part clears 0.5x by a small margin. On a Xeon 6741P the memory part replicated and the time part did not [ledger:H28x] | | H60 | Qwen3.8-Flash-Next (125B MoE, ggml-org IQ4_NL GGUF in tmpfs) served by llama.cpp v0.8.0 from the Xeon 6741P's DRAM decoded 16.00 tok/s at one stream, 0.312 of the 51.3 tok/s ceiling that 305 GB/s gives at 5.94 GB read per token. A same-hold Gemma 4 QAT Q4_0 control on the same build read 0.379 (48.52 tok/s), so the ratio was 0.823, CI 0.819-0.831, inside the 0.7-1.3 band; GSM8K items 0-49 scored 50/50 [ledger:H60] | One hold, 48 threads on the 48 physical cores, CPU only, greedy, thinking off for Flash-Next (the control opened its thinking block). The control's decode window starts about 3 tokens after its first chunk, so its rate reads about 1% high; timed from the first chunk, the ratio is 0.833. The ceiling uses a fixed STREAM figure; package watts include about 130 W of other tenants' idle draw | | H40 | The agent harness reproduces the BFCL board: bf16 Qwen3-4B-Instruct-2507 on vLLM CUDA scored 88.0% on the fixed 100-id subset in BFCL's own mode (board-weighted 84.9%, band 77.9-91.9%), every category inside its binomial interval [ledger:H40] | A check of the harness, not of the hardware; one 100-id subset, one run, on the NVIDIA dGPU | +| H52w | On wildct (an i7-13700 with two NVIDIA cards and a UHD 770 iGPU), a plain `k3d-gpu up` with the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`) detected every CPU engine present and nothing phantom: the hybrid P/E cores with the balloons policy (`k3d-gpu test`'s P-core pod on CPU 0, its E-core pod on CPU 17) and the launcher's six ISA labels (AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS); no AVX-512, AMX or NUMA label, and no DSA, IAA, QAT, NPU or Gaudi resource or pod. Both GPU vendors were named as detected and opted out, no GPU resource was allocatable and no pod mounted a `/dev` path [ledger:H52w] | One hold, release 0.1.89. Detection with the GPUs opted out: says nothing about a GPU reaching a pod on this host | ### REFUTED (all but H5, H7x, H27, H27x, H28x, H34x, H43, H45, H48, H50, H51 and H52b changed a default) From e8e0bd9f5f0344bf92d132a0256ee949d05f357c Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Fri, 6 Nov 2026 20:30:35 -0800 Subject: [PATCH 2/4] bench(wildct): H52w run review fixes: docs anchor, unrecorded module read, two header lines The technical report's SUPPORTED heading now names H52w, which moved its anchor, so the partner update's link to the old anchor broke the docs build (mkdocs --strict: "does not contain an anchor"). The link now points at the new anchor, the partner update lists H52w among the SUPPORTED rows, and its open-items line no longer counts H52w as pending. The run section said the loaded module count read 224 before the hold and at 04:11Z. No record holds a 04:11Z read; the 224 is the preflight's (03:28Z, host/preflight.txt), and host/h52w.txt has 18 of 18 loaded and kernel.modprobe empty before the hold only. The text now says exactly that, and that no module read after the hold is recorded. Two Reproduce lines now match the records: the shipped archive's bench/ differs from main's at 28b1e70b in RESUME.md and the run records (the runners are equal), and sample.py read / at 221.1-222.2 GB (decimal), where 207G is df -h's figure. No prediction, verdict or measured value changes. --- EXPERIMENTS.md | 11 +++++++---- docs/intel/partner-update-2026-11.md | 5 +++-- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index a3617323..84926709 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -8836,14 +8836,15 @@ accelerators: host inventory: UHD 770 8086:a780 (i915), RTX 4000 SFF Ada 10de k3d-gpu: release 0.1.89 (aur-v0.1.89, .deb sha256 8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc), unpacked in ~/k3dgpu-bench-wildct/k3d-gpu; still the newest release when the hold started bench scripts: main at 85a7bc86, shipped as the archive repo-85a7bc8668d1 (its launcher equals 0.1.89's apart - from VERSION; its bench/ and scripts/ equal main's at 28b1e70b) + from VERSION; its bench/ and scripts/ equal main's at 28b1e70b apart from + bench/wildct/RESUME.md and the run records under bench/wildct/results/) node image: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4, k3s v1.37.1+k3s1, containerd 2.3.4-k3s1; k3d v5.9.0, helm v4.3.1, kubectl v1.37.2 runtime: k3d-gpu test pods busybox:1.37@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec environment: K3D_GPU_VENDORS=auto,-nvidia,-intel, K3D_GPU_K3S_ARGS=, K3D_GPU_CLUSTER=k3dgpu-w-h52, K3D_GPU_RAPL_EXEC="docker exec k3dgpu-w-rapl" (rule W4; H52w reads no energy) host load: load1 0.61-1.14, foreign CPU 0.47-1.77 CPU-s/s, MemAvailable 113.6-114.5 GiB, package 61-67 C, - / 207 GB available (sample.py, 13 samples) + / 221.1-222.2 GB available (sample.py, 13 samples; df -h read 207G before and after) ``` Command, on wildct: RESUME.md step 6's `launch H52w 1800 3 h52w`, which runs `bench/wildct/hold.sh H52w @@ -8900,8 +8901,10 @@ containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the s line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` cluster or container, no process of the hold and no network or volume of ours was left, and -`live.lock` was free. The host's loaded module count read 224 before the hold and at 04:11Z, -with all 18 modules of `/etc/modules-load.d/k3dgpu-wildct.conf` loaded and `kernel.modprobe` empty. +`live.lock` was free. Before the hold (04:03Z) all 18 modules of +`/etc/modules-load.d/k3dgpu-wildct.conf` were loaded and `kernel.modprobe` was empty; the preflight +(03:28Z) counted 224 loaded modules (`host/preflight.txt`). The records hold no module read after +the hold. Rule W7: this was the campaign's first hold, so every later wildct row runs release 0.1.89. Holds used: 1 of at most 12. The next row in the registered order is H18w. diff --git a/docs/intel/partner-update-2026-11.md b/docs/intel/partner-update-2026-11.md index 94f97d17..0e2ab7a6 100644 --- a/docs/intel/partner-update-2026-11.md +++ b/docs/intel/partner-update-2026-11.md @@ -58,7 +58,8 @@ All drafts are unsent and need a maintainer's approval before anything is filed. SUPPORTED rows, each scoped to its host: - On the Xeon 6741P: the CPU engine choice [ledger:H59], DRAM-resident serving of Qwen3.8-Flash-Next [ledger:H60], and H25 replicated [ledger:H25x]. -- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p). +- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-to-wildcts-i7-13700). +- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w]. ## Two results that help EAT @@ -72,7 +73,7 @@ v0.28.0 is past the v0.24.0 CPU attention hang that stalled EAT 1.0.3's stock ar - **The Keycloak argv secret.** F3 stays REFUTED. `k3d-gpu eas up` now patches enterprise-inference's `keycloak_config` role so that neither the admin password nor the client secret is a `kcadm.sh` argument ([State and secrets](eas.md#state-and-secrets)). The patch is checked with a `ps` sampler against a stand-in for kcadm; no live run has checked it yet. A fresh `eas up --agent-toolkit` with F3's secret scan would. - **Intel's default install in a guest (H62).** A re-run needs the `cpu: auto` sizing to leave room for EAT's sandbox router, or a guest shape where it does. - **k3d-gpu gaps found on the Xeon.** `/dev/dri/by-path` for oneCCL across B70s (H53); the fit refusing rather than falling back to the CPU, and no DRAM bandwidth cached on a new host (H58); placing one-card arms on one card (H54b, H54c). -- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. +- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H52w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. - **EAT 1.1.0 through `eat-up.sh`.** Pinned since 2026-10-29 and pre-registered as F4; not run yet. - **EAS upstream moves.** EAS #61 merged on 2026-10-30 (EAS `main` `0f7bc2b`). The pin stays at `97f6157`; moving past #61 needs patch 0002 regenerated. EAS #50 and #52 (OpenShift) are open and would move 0006. EAS #71 (teardown hang) is open; `eas down` uses its workaround. - **ERAG and Arc under EAS.** EAS's Enterprise RAG layer is not wired into `k3d-gpu eas` yet, and EAS has no Intel GPU path on this branch; Arc and Gaudi stay with `k3d-gpu agent up`. From b6c54a728e9217e4e146ed76e85fe8ee775bb7d9 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Fri, 6 Nov 2026 20:35:36 -0800 Subject: [PATCH 3/4] bench(wildct): H18w run on release 0.1.89, SUPPORTED Row 2 of the wildct campaign, run to its registered line: one hold of three bench/sandbox_round.sh runs on cluster k3dgpu-w-sbx with the release node image, the box owner's GPU opt-out and rule W3. In every run `sandbox up` exited 0 with the sandbox pod on CPU 17 (E-cores 16-23); the host's tailscale and LAN sshd addresses were refused from the sandbox with 2 packets on kube-router's drop-log rule and reached from the control pod; pip installed tomli_w from PyPI into /tmp/.local. H18's outcome table reads SUPPORTED. The production snapshot, the firewall rules and fail2ban's sshd jail read the same before and after the hold, and nothing of the hold was left on wildct. Records are under bench/wildct/results/2026-11-07/; RESUME.md names H25w as the next row, and the technical report's counts move with the row. --- EXPERIMENTS.md | 107 +++++++- bench/wildct/RESUME.md | 34 +-- bench/wildct/results/2026-11-07/host/h18w.txt | 231 ++++++++++++++++++ bench/wildct/results/2026-11-07/runs/H18w.out | 4 + .../results/2026-11-07/runs/H18w/hold.log | 48 ++++ .../2026-11-07/runs/H18w/launcher.diff | 0 .../2026-11-07/runs/H18w/out/run-1.txt | 16 ++ .../2026-11-07/runs/H18w/out/run-2.txt | 16 ++ .../2026-11-07/runs/H18w/out/run-3.txt | 16 ++ .../2026-11-07/runs/H18w/out/sandbox-1.log | 30 +++ .../2026-11-07/runs/H18w/out/sandbox-2.log | 30 +++ .../2026-11-07/runs/H18w/out/sandbox-3.log | 30 +++ .../results/2026-11-07/runs/H18w/out/up-1.log | 34 +++ .../results/2026-11-07/runs/H18w/out/up-2.log | 34 +++ .../results/2026-11-07/runs/H18w/out/up-3.log | 33 +++ bench/wildct/results/2026-11-07/runs/H18w/pid | 1 + .../results/2026-11-07/runs/H18w/post.txt | 17 ++ .../results/2026-11-07/runs/H18w/pre.txt | 17 ++ bench/wildct/results/2026-11-07/runs/H18w/rc | 1 + .../results/2026-11-07/runs/H18w/sample.jsonl | 49 ++++ .../results/2026-11-07/runs/H18w/wait_s | 1 + .../results/2026-11-07/runs/campaign.log | 4 + docs/intel/technical-report.md | 11 +- 23 files changed, 744 insertions(+), 20 deletions(-) create mode 100644 bench/wildct/results/2026-11-07/host/h18w.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w.out create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/hold.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/launcher.diff create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/pid create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/post.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/pre.txt create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/rc create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl create mode 100644 bench/wildct/results/2026-11-07/runs/H18w/wait_s diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index a3617323..acfb1597 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -85,7 +85,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57dp | On the i9-13900T P-cores under the shared host's own load, llama.cpp with Gemma 4 QAT Q4_0 and the QAT MTP drafter decodes faster unpinned (8 threads, no `-C`/`--cpu-strict`) than strictly pinned one thread per physical P-core: N=1 ratio >= 1.10. | Written 2026-10-21, after H57d, before any H57dp measurement. SUPPORTED (0.5); Ld-mtp-u/Ld-mtp in [1.0, 1.8] (0.6). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): 2 holds of at most 75 min, H57d's protocol. SUPPORTED: pooled median ratio >= 1.10, CI lower bound > 1.0, faster in both holds. REFUTED: CI upper bound < 1.10. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched, foreign P-core load spread 2.97 > 2.0 CPU-s/s (Od 0.45, Ld-mtp 3.42, Ld-mtp-u 3.11), so the scorer stops before the statistic. Not gated: Ld-mtp-u/Ld-mtp 1.108 (95% CI 1.072-1.142), faster in both holds (1.108, 1.100); Ld-mtp-u 13.39, Ld-mtp 12.09 tok/s pooled N=1; affinity as registered. | INCONCLUSIVE | | H57de | H57da with the unpinned llama.cpp arm: no arm decodes Gemma 4 QAT at N=1 5% or more faster than k3d-gpu's OVMS pick Od on the i9-13900T P-cores. | Written 2026-10-21, before any H57de measurement. REFUTED (0.5); Ld-mtp-u/Od in [1.0, 1.6] (0.5). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): same holds as H57dp. REFUTED: Ld-mtp-u/Od CI lower bound >= 1.05 and faster in both holds. SUPPORTED: CI upper bound < 1.05. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched (the same gate, spread 2.97 > 2.0). Od also ran without the P-core balloon (cpuset 0-15,17-31: the runner's `agent manifest` call named no cluster, so after #183 it missed the hold's kube context), so it was not the registered Od. Not gated: Ld-mtp-u/Od 1.369 (1.345-1.396), Od 9.78 tok/s pooled N=1. | INCONCLUSIVE | | H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. Result ([H52w run](#h52w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign). `up` took 32 s, named nvidia and intel as opted out and printed `Accelerators: hybrid`; the balloons policy pod Running, `k3d-gpu test`'s P-core pod on CPU 0 and its E-core pod on CPU 17; the launcher's ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT and CETSS, no AVX512 or AMX label from any source; no NUMA label; allocatable cpu, ephemeral-storage, hugepages, memory and pods only; no GPU plugin pod, no `/dev` hostPath and no GPU resource (`GPU-SCAN clean`); the node container `gpus=[]`; the inventory unchanged. Every row matches. | SUPPORTED | -| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. | PENDING | +| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. Result ([H18w run](#h18w-run-wildct-i7-13700-2026-11-07)): one hold of 3 runs on release 0.1.89 (rule W7's release for the campaign), a fresh cluster each. In every run `sandbox up` exited 0 with the warm pool's `readyReplicas` 1 and the sandbox pod on CPU 17, inside the E-cores 16-23; from the sandbox, TCP to `100.66.38.121:22` and to `192.168.8.238:22` was refused in 0.11-0.12 s with 2 packets on the policy's drop-log rule, and from the control pod both connected; `pypi.org` resolved, `pypi.org:443` connected and `pip install --user tomli_w==1.2.0` exited 0, importing from `/tmp/.local`. Every run meets (a), (b) and (c). | SUPPORTED | | H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | | H34w | H34's hypothesis on wildct: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, both on the 16-CPU P-core balloon, under matched background load. A replication: original models, runner, scorer and rule. | Written 2026-10-20, before any H34w measurement. At `typical`: median paired B/A decode tok/s 1.35 (1.15-1.6) and median paired B/A package J/token 0.76 (0.6-0.92). SUPPORTED by H34's rule at 0.6, REFUTED 0.25, INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, `typical` only, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h34`, the GPU opt-out in place of H34's `intel,hybrid`, W2, W3, W4. | PENDING | @@ -8906,6 +8906,111 @@ with all 18 modules of `/etc/modules-load.d/k3dgpu-wildct.conf` loaded and `kern Rule W7: this was the campaign's first hold, so every later wildct row runs release 0.1.89. Holds used: 1 of at most 12. The next row in the registered order is H18w. +### H18w run (wildct, i7-13700, 2026-11-07) + +**Reproduce:** + +```text +# provenance +date: 2026-11-07T04:22:23Z to 04:26:30Z (one hold; lock free, no load wait: load1 0.52); + runs 1-3 began 04:22:29Z, 04:23:31Z and 04:25:25Z +hypothesis: H18w (H18's runner and outcome table, as the wildct pre-registration's H18w line lists them) +host: wildct, reached by IP 100.66.38.121: Intel Core i7-13700 (family 6 model 183), 8 P-cores + (cpu_core 0-15) and 8 E-cores (cpu_atom 16-23), 1 NUMA node, 125 GiB RAM, Manjaro; Docker + 29.7.2, live-restore off, cgroup systemd/2 (step 1, 03:32:22Z) +host kernel: 6.6.156-2-MANJARO +accelerators: hybrid=1; UHD 770 (i915), RTX 4000 SFF Ada and RTX 2000 Ada (nvidia), opted out and not + used (rule W8) +network: tailscale0 100.66.38.121, default-route source 192.168.8.238, sshd on 0.0.0.0:22 and [::]:22; + TARGETS as the runner resolved them: 100.66.38.121:22 192.168.8.238:22 pypi.org:443 +k3d-gpu: release 0.1.89 (aur-v0.1.89, .deb sha256 8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc), + rule W7's release since the first hold (H52w) +bench scripts: main at 85a7bc86, shipped as the archive repo-85a7bc8668d1; sandbox_round.sh runs its + scripts/k3d-gpu (equal to 0.1.89's apart from VERSION) and share/ (equal to 0.1.89's) +node image: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4, + k3s v1.37.1+k3s1; k3d v5.9.0, helm v4.3.1, kubectl v1.37.2 +runtime: agent-sandbox v1.0.8 controller and python-runtime-sandbox v1.0.8, both pinned by digest in + share/ (H18 ran v1.0.7); NetworkPolicy enforced by k3s's embedded kube-router +model: none +environment: IMAGE=, K3D_GPU_CLUSTER=k3dgpu-w-sbx, K3D_GPU_VENDORS=auto,-nvidia,-intel, + K3D_GPU_K3S_ARGS=, K3D_GPU_RAPL_EXEC="docker exec k3dgpu-w-rapl" (rule W4) +energy source: intel-rapl:0 energy_uj (package-0), confounds only: package W over 5 s idle before each run +runs: 3 counted, one fresh cluster each, in one hold; no pilot +host load: load1 0.48-1.51, foreign CPU 0.51-1.65 CPU-s/s, MemAvailable 113.4-114.7 GiB, package + 61-69 C, swap used 0 MiB (sample.py, 49 samples; df: / 207 GB available before and after) +``` + +Command, on wildct: RESUME.md step 6's `launch H18w 2100 3 h18w`, which runs `bench/wildct/hold.sh H18w +2100 3 bash bench/wildct/row.sh h18w`, and so `IMAGE= OUT=~/k3dgpu-bench-wildct/runs/H18w/out +K3D_GPU_CLUSTER=k3dgpu-w-sbx K3D_GPU_VENDORS=auto,-nvidia,-intel K3D_GPU_K3S_ARGS= +bench/sandbox_round.sh ` for n = 1, 2, 3, the registered line. Records (each run's key=value block, +`up` and `sandbox up` logs, the hold's samples; the kubeconfigs were not copied): +[bench/wildct/results/2026-11-07/runs/H18w/](bench/wildct/results/2026-11-07/runs/H18w/). The host +before and after the hold: [bench/wildct/results/2026-11-07/host/h18w.txt](bench/wildct/results/2026-11-07/host/h18w.txt). +The scored lines of each run: + +```bash +grep -hE '^(run=|sandbox_up_rc|warm_ready|pod=|probe|policy_chain|dns_pypi|env=|pip_rc)' \ + bench/wildct/results/2026-11-07/runs/H18w/out/run-{1,2,3}.txt +``` + +Each `up` named nvidia and intel as opted out, printed `Accelerators: hybrid` and ended with +`Accelerators schedulable: hybrid=1`. + +| Run | `sandbox up` (s) | sandbox CPUs | sandbox -> 100.66.38.121:22 | control -> same | sandbox -> 192.168.8.238:22 | control -> same | drop-log pkts | pypi.org:443 | pip `tomli_w` | load (1 min) | throttle delta | +|-----|------|----|------|------|------|------|---|------|------|------|------| +| 1 | 12.3 | 17 | refused, 0.12 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 7.4 s | 0.48 | 0 | +| 2 | 69.9 | 17 | refused, 0.12 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 1.2 s | 1.23 | 0 | +| 3 | 15.0 | 17 | refused, 0.11 s | connect | refused, 0.12 s | connect | 2 | connect | ok, 1.7 s | 1.15 | 0 | + +In each run: + +- `sandbox up` exited 0, the warm pool's `readyReplicas` was 1, and its status check printed + `ok: python-ecores-: CPUs 17 (E-cores 16-23)`. The runner's own read of the pod's + `Cpus_allowed_list` was 17, with the host's `cpu_atom` 16-23. +- Every blocked connect printed `NOCONNECT ConnectionRefusedError [Errno 111] Connection refused`. + Each of kube-router's policy chains (one per cluster) counted 2 packets on its NFLOG rule + ("DROP by policy agent-sandbox/python-ecores-network-policy"). Each control connect took 0.08-0.09 s. +- `pypi.org` resolved to 151.101.0.223 (runs 1 and 2) and 151.101.192.223 (run 3), both Fastly. The + sandbox's connect to `pypi.org:443` took 2.33, 0.22 and 0.14 s. +- The sandbox ran with `HOME=/tmp uid=1000`. pip's last line of output was its upgrade notice, and the + import read `/tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py`. + +`sandbox up` wall time: median 15.0 s, spread (max - min) 57.6 s, n = 3. Run 2's warm pool was 67 s +old when `sandbox up` read it ready (9 s and 12 s in runs 1 and 3); each fresh cluster pulls the +controller and runtime images, and the records do not say more. None of the checks is a timing +threshold. Confounds: package power over the 5 s before each run 35.6, 35.3 and 34.6 W; swap used 0 MiB; +the package throttle count delta, summed over the host's 24 counters, 0 in every run. + +**Verdict: SUPPORTED.** Every run meets (a), (b) and (c), the first line of H18's outcome table. On a +second client Raptor Lake host, `k3d-gpu sandbox up` from the release kept each sandbox on an E-core +and behind the default-deny egress policy: the tailscale and LAN addresses were refused from the +sandbox and reached from a control pod without the policy, and PyPI stayed reachable for a `pip install +--user`. The prediction (every run meets (a), (b) and (c), 0.8) held. Scope: as H18's, k3s v1.37.1 +kube-router and IPv4 targets only; the three registered targets; agent-sandbox v1.0.8, the release's. + +Not scored: + +- Rule W2 keeps CPUs 16-19 free for the production services, except for H18w's one 1-CPU sandbox pod + and k3s's own pods. The sandbox pod got CPU 17, inside that range, and ran the probes and the pip + install there (1.2-7.4 s). +- As in H18, `pip_rc` is the exit of `pip install ... | tail -1 && python3 -c 'import tomli_w ...'` in + the pod's `sh`, so pip's own exit reaches it only through the import that follows. The import read + the package from `/tmp/.local`, where only the `--user` install puts it. +- The control pod's six connects to the host's sshd (two per run) left fail2ban's sshd jail as it was: + `Total failed` 0 and `Total banned` 2 before and after the hold. The jail runs the filter's `normal` + mode, with `maxretry` 5. + +Production box. The snapshot before the hold (04:20Z) and after it (04:27Z) read the same: 18 +containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the same port codes. The +line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and +after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` +cluster or container, no process of the hold and no network or volume of ours was left, and +`live.lock` was free. The host's loaded module count read 224 after the hold, with all 18 modules of +`/etc/modules-load.d/k3dgpu-wildct.conf` loaded and `kernel.modprobe` empty. + +Holds used: 2 of at most 12. The next row in the registered order is H25w. + ### H5 amendment A1: stage-1 harness and definitions (2026-10-19) Written 2026-10-19T22:30-07:00, before any H5 measurement. It fixes what the diff --git a/bench/wildct/RESUME.md b/bench/wildct/RESUME.md index 1870ae39..a453a0bc 100644 --- a/bench/wildct/RESUME.md +++ b/bench/wildct/RESUME.md @@ -2,18 +2,24 @@ How to resume the wildct campaign ([EXPERIMENTS.md](../../EXPERIMENTS.md), "Wildct pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5 and R6, and the "H52w -run" section). This file changes no rule. Run it top to bottom, except that step 3b (ship -`origin/main` to wildct) runs first when the `repo` on wildct predates `bench/wildct/nfprobe.sh` -(H10 runs it). Nothing past step 1 runs until every step 1 check passes. - -## Where things stand (2026-11-07, after H52w) - -- **Next: row 2, `H18w` (step 6).** Row 1, `H52w`, ran on 2026-11-07 (04:03:54Z to 04:05:00Z, - rc 0, every line of `out/rc.txt` a pass) and is scored SUPPORTED (EXPERIMENTS.md "H52w run"; - records in `results/2026-11-07/runs/H52w/`, the host before and after in - `results/2026-11-07/host/h52w.txt`). The host's containers, start times, restart counts, port - codes and firewall rules read the same before and after it, and nothing of the hold was left. - Holds used: 1 of at most 12. +run" and "H18w run" sections). This file changes no rule. Run it top to bottom, except that step +3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates +`bench/wildct/nfprobe.sh` (H10 runs it). Nothing past step 1 runs until every step 1 check passes. + +## Where things stand (2026-11-07, after H18w) + +- **Next: row 3, `H25w` (step 6).** Holds used: 2 of at most 12. +- **Row 2, `H18w`,** ran on 2026-11-07 (04:22:23Z to 04:26:30Z, rc 0; `out/run-1..3.txt` each has + `sandbox_up_rc=0` and `pip_rc=0`) and is scored SUPPORTED (EXPERIMENTS.md "H18w run"; records in + `results/2026-11-07/runs/H18w/`, the host before and after in `results/2026-11-07/host/h18w.txt`). + The host's containers, start times, restart counts, port codes and firewall rules read the same + before and after it, fail2ban's sshd jail counters did not change, and nothing of the hold was + left. +- **Row 1, `H52w`,** ran on 2026-11-07 (04:03:54Z to 04:05:00Z, rc 0, every line of `out/rc.txt` a + pass) and is scored SUPPORTED (EXPERIMENTS.md "H52w run"; records in + `results/2026-11-07/runs/H52w/`, the host before and after in `results/2026-11-07/host/h52w.txt`). + The host's containers, start times, restart counts, port codes and firewall rules read the same + before and after it, and nothing of the hold was left. - **Release frozen at 0.1.89 (rule W7).** `H52w`, the campaign's first hold, ran 0.1.89, so every later hold runs it too. Step 3 does not run again, and neither does step 5. - **Kernel modules** (each approved by the box owner on 2026-10-29; `kernel.modprobe` stays empty): @@ -70,8 +76,8 @@ run" section). This file changes no rule. Run it top to bottom, except that step `networks-before.txt` and the verify logs; - `runs/`: `smoke1*`, `smoke2*`, `diag-smoke2*`, `smoke3*` (with `smoke3-node.log`, the node's log, and `smoke3-node-excerpt.txt`), `smoke4*` (with `smoke4-live.txt`), `smoke5*`, - `H52w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones renamed - `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, + `H52w*`, `H18w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones + renamed `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, `install-0.1.89.log`, `setup-images-*.log`, `IMAGE` and `campaign.log`; - `diag/`: `diag-up.sh`, the command `diag-smoke2` ran; `iptprobe-in.sh`, an ad-hoc form of `nfprobe.sh`; the second load's read-only checks `modsnap.sh` (modules and firewall, as diff --git a/bench/wildct/results/2026-11-07/host/h18w.txt b/bench/wildct/results/2026-11-07/host/h18w.txt new file mode 100644 index 00000000..ee0099e8 --- /dev/null +++ b/bench/wildct/results/2026-11-07/host/h18w.txt @@ -0,0 +1,231 @@ +# wildct around the H18w hold, 2026-11-07 (UTC). Read-only reads as andrew, except the firewall +# dumps (iptables-save, ip6tables-save and nft list ruleset) and fail2ban-client status, as root. +# The production snapshot: running containers, published-port HTTP codes from the host, and each +# running container's StartedAt and RestartCount. + +== state before the hold (2026-11-07T04:20Z) +## date 2026-11-07T04:20:28+00:00 +## boot: 2026-10-29 05:11:09 +## kernel: 6.6.156-2-MANJARO +## disk +/dev/nvme0n1p2 3667G 3273G 207G 95% / +## mem +MemTotal: 131524088 kB +MemAvailable: 120085356 kB +## loadavg: 0.43 0.45 0.48 1/3658 32271 +VERSION="0.1.89" # kept in lockstep with packaging/aur PKGBUILD by auto-release.yml +## repo -> repo-85a7bc8668d1 COMMIT=85a7bc8668d18a3233496f0a3a23b09bb63ee20f +## runs/IMAGE: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +## tools +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +## clusters (k3d cluster list; names only) +wtm (the box owner's; exited; not this campaign's). No k3dgpu-w-* cluster. +## our containers +ps rc=0 +## live.lock +-rw-r--r-- 1 andrew andrew 0 Nov 6 23:03 /home/andrew/k3dgpu-bench-wildct/live.lock +lock-free +hold procs rc=1 +H52w pid=40856 alive=no rc=0 +convert pid=24108 alive=no rc=0 +diag-smoke2 pid=17199 alive=no rc=1 +smoke2 pid=32217 alive=no rc=1 +smoke3 pid=62226 alive=no rc=0 +smoke4 pid=48184 alive=no rc=0 +smoke5 pid=31149 alive=no rc=0 +## modules of /etc/modules-load.d/k3dgpu-wildct.conf +loaded: 18 of 18; missing: 0 +modprobe=[] +## newest k3d-gpu release at 04:21Z (gh release list): aur-v0.1.89 (2026-11-07T00:23:41Z), as installed; +## rule W7 froze it at the first hold (H52w) + +== the row's command and targets, read before the launch (row.sh h18w with WILDCT_ROW_PRINT=1) +## row print +IMAGE=cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +OUT=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out +K3D_GPU_CLUSTER=k3dgpu-w-sbx +K3D_GPU_VENDORS=auto,-nvidia,-intel +K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1% +bash /home/andrew/k3dgpu-bench-wildct/repo/bench/sandbox_round.sh (runs 1-3) +## row check +check rc=0 +## TARGETS as sandbox_round.sh resolves them +TARGETS=100.66.38.121:22 192.168.8.238:22 pypi.org:443 +## sshd listen +LISTEN 0 0 0.0.0.0:22 0.0.0.0:* +LISTEN 0 0 *:22 *:* +## launcher vs release +LAUNCHER-MATCH +share diff done +## node image +cryptoandcoffee/k3d-gpu@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 sha256:5811390ea98da96ebe01a3338291d4f462bb8c16e0063c99939944190547b8c8 +## sandbox images in share + image: registry.k8s.io/agent-sandbox/agent-sandbox-controller:v1.0.8@sha256:e1d96312487eba7d8f001e848a9d4e644b7deea8c2a4564ccc318abb729be731 + image: registry.k8s.io/agent-sandbox/python-runtime-sandbox:v1.0.8@sha256:41da84809426098cb04cdafbb41855f15e3604f7455ef53816bad666e532e787 +## cpu_atom, cpu_core +16-23 +0-15 + +== fail2ban (sshd jail: maxretry 5, findtime 1d, filter mode normal) before the hold +Status for the jail: sshd +|- Filter +| |- Currently failed: 0 +| |- Total failed: 0 +| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd + _COMM=sshd-session +`- Actions + |- Currently banned: 2 + |- Total banned: 2 + +== production snapshot before the hold (2026-11-07T04:20Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== production snapshot after the hold (2026-11-07T04:27Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== fail2ban after the hold +Status for the jail: sshd +|- Filter +| |- Currently failed: 0 +| |- Total failed: 0 +| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd + _COMM=sshd-session +`- Actions + |- Currently banned: 2 + |- Total banned: 2 + +== comparison +production snapshot: identical +iptables-save rules, packet counters stripped: identical +ip6tables-save rules, packet counters stripped: identical +nft list ruleset, counters stripped: identical +fail2ban sshd jail counters: identical + +== after-hold checks (RESUME.md step 6) +== after H18w 2026-11-07T04:26:59Z +0 +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z +list rc=0 +ps rc=0 +strays rc=0 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +lock-free +net rc=1 (grep found no k3dgpu network) +vol rc=1 (grep found no k3dgpu volume) + +== host state after the hold +## date 2026-11-07T04:27:50+00:00 +## modules: 224 loaded +modprobe=[] +clusters rc=0 +ours grep rc=1 (1 = none) +lock-free +hold procs rc=1 (1 = none) +/dev/nvme0n1p2 3667G 3273G 207G 95% / +## thermal_throttle files: 24 +## sshd: active diff --git a/bench/wildct/results/2026-11-07/runs/H18w.out b/bench/wildct/results/2026-11-07/runs/H18w.out new file mode 100644 index 00000000..10276bf5 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w.out @@ -0,0 +1,4 @@ +== H18w waiting for the lock 2026-11-07T04:22:23Z +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z diff --git a/bench/wildct/results/2026-11-07/runs/H18w/hold.log b/bench/wildct/results/2026-11-07/runs/H18w/hold.log new file mode 100644 index 00000000..18b4c112 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/hold.log @@ -0,0 +1,48 @@ +run=1 date=2026-11-07T04:22:29Z idle_pkg_w=35.6 load1=0.48 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=12.3 +status: ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-hfgtf cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=2.33 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.10 CONNECT +policy_chain=KUBE-NWPLCY-MPEOAIZRZ6XJTD26 policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=7.4 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 +run=2 date=2026-11-07T04:23:31Z idle_pkg_w=35.3 load1=1.23 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=69.9 +status: ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-cthm6 cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.22 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.14 CONNECT +policy_chain=KUBE-NWPLCY-6TDZP7NK6CZGKVRG policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.2 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 +run=3 date=2026-11-07T04:25:25Z idle_pkg_w=34.6 load1=1.15 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=15.0 +status: ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-7nd5c cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.11 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.14 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.09 CONNECT +policy_chain=KUBE-NWPLCY-WQVD77VIFGWKCLQF policy_drop_pkts=2 +dns_pypi=151.101.192.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.7 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/launcher.diff b/bench/wildct/results/2026-11-07/runs/H18w/launcher.diff new file mode 100644 index 00000000..e69de29b diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt new file mode 100644 index 00000000..d9e6cb4d --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-1.txt @@ -0,0 +1,16 @@ +run=1 date=2026-11-07T04:22:29Z idle_pkg_w=35.6 load1=0.48 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=12.3 +status: ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-hfgtf cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=2.33 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.10 CONNECT +policy_chain=KUBE-NWPLCY-MPEOAIZRZ6XJTD26 policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=7.4 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt new file mode 100644 index 00000000..03eb63fd --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-2.txt @@ -0,0 +1,16 @@ +run=2 date=2026-11-07T04:23:31Z idle_pkg_w=35.3 load1=1.23 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=69.9 +status: ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-cthm6 cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.22 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.14 CONNECT +policy_chain=KUBE-NWPLCY-6TDZP7NK6CZGKVRG policy_drop_pkts=2 +dns_pypi=151.101.0.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.2 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt b/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt new file mode 100644 index 00000000..6b9f6dea --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/run-3.txt @@ -0,0 +1,16 @@ +run=3 date=2026-11-07T04:25:25Z idle_pkg_w=34.6 load1=1.15 swap_used_mib=0 +sandbox_up_rc=0 sandbox_up_s=15.0 +status: ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) +warm_ready=1 +pod=python-ecores-7nd5c cpus=17 host_atom=16-23 host_core=0-15 +probe ns=agent-sandbox dst=100.66.38.121:22 rc=1 s=0.11 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=100.66.38.121:22 rc=0 s=0.08 CONNECT +probe ns=agent-sandbox dst=192.168.8.238:22 rc=1 s=0.12 NOCONNECT ConnectionRefusedError [Errno 111] Connection refused +probe ns=sbx-control dst=192.168.8.238:22 rc=0 s=0.09 CONNECT +probe ns=agent-sandbox dst=pypi.org:443 rc=0 s=0.14 CONNECT +probe ns=sbx-control dst=pypi.org:443 rc=0 s=0.09 CONNECT +policy_chain=KUBE-NWPLCY-WQVD77VIFGWKCLQF policy_drop_pkts=2 +dns_pypi=151.101.192.223 +env=HOME=/tmp uid=1000 +pip_rc=0 pip_s=1.7 [notice] To update, run: pip install --upgrade pip IMPORT /tmp/.local/lib/python3.14/site-packages/tomli_w/__init__.py +throttle_delta=0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log new file mode 100644 index 00000000..d5c87d6b --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-1.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 9s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-hfgtf True DependenciesReady 8s + ok: python-ecores-hfgtf: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log new file mode 100644 index 00000000..2b78e5db --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-2.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 67s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-cthm6 True DependenciesReady 66s + ok: python-ecores-cthm6: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log new file mode 100644 index 00000000..cc4ffad8 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/sandbox-3.log @@ -0,0 +1,30 @@ +==> Installing agent-sandbox v1.0.8 (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox.yml)... +namespace/agent-sandbox-system created +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io created +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io created +serviceaccount/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrole.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller created +clusterrolebinding.rbac.authorization.k8s.io/agent-sandbox-controller-extensions created +service/agent-sandbox-controller created +deployment.apps/agent-sandbox-controller created +customresourcedefinition.apiextensions.k8s.io/sandboxes.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxtemplates.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxclaims.extensions.agents.x-k8s.io condition met +customresourcedefinition.apiextensions.k8s.io/sandboxwarmpools.extensions.agents.x-k8s.io condition met +Waiting for deployment "agent-sandbox-controller" rollout to finish: 0 of 1 updated replicas are available... +deployment "agent-sandbox-controller" successfully rolled out +==> Applying the E-core sandbox profile (/home/andrew/k3dgpu-bench-wildct/repo/share/agent-sandbox-k3d.yml)... +namespace/agent-sandbox created +sandboxtemplate.extensions.agents.x-k8s.io/python-ecores created +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores created +==> Waiting for the python-ecores warm pool (first start pulls the runtime image)... +NAME READY DESIRED AGE +sandboxwarmpool.extensions.agents.x-k8s.io/python-ecores 1 1 12s + +NAME READY REASON AGE +sandbox.agents.x-k8s.io/python-ecores-7nd5c True DependenciesReady 12s + ok: python-ecores-7nd5c: CPUs 17 (E-cores 16-23) diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log new file mode 100644 index 00000000..a4902dda --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-1.log @@ -0,0 +1,34 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-1 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +Waiting for daemon set "nri-resource-policy-balloons" rollout to finish: 0 of 1 updated pods are available... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log new file mode 100644 index 00000000..aca51d04 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-2.log @@ -0,0 +1,34 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-2 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +Waiting for daemon set "nri-resource-policy-balloons" rollout to finish: 0 of 1 updated pods are available... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log b/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log new file mode 100644 index 00000000..0d231735 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/out/up-3.log @@ -0,0 +1,33 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-sbx' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-sbx' +INFO[0000] Created image volume k3d-k3dgpu-w-sbx-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-sbx-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-sbx' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-sbx-server-0' +INFO[0004] All agents already running. +INFO[0004] All helpers already running. +INFO[0004] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0006] Cluster 'k3dgpu-w-sbx' created successfully! +INFO[0006] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0006] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-sbx) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H18w/out/kubeconfig-3 +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-sbx-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-sbx-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H18w/pid b/bench/wildct/results/2026-11-07/runs/H18w/pid new file mode 100644 index 00000000..d8033cb0 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/pid @@ -0,0 +1 @@ +46184 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/post.txt b/bench/wildct/results/2026-11-07/runs/H18w/post.txt new file mode 100644 index 00000000..9957da65 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/post.txt @@ -0,0 +1,17 @@ +2026-11-07T04:26:29Z + 23:26:29 up 8 days, 19:15, 2 users, load average: 1.25, 0.93, 0.67 +%CPU PID USER COMMAND + 2.7 6519 andrew rustdesk + 2.6 1157 root tailscaled + 1.7 8054 andrew firefox + 1.5 7117 andrew rustdesk + 1.4 1 root systemd + 1.4 9806 andrew WebExtensions + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.4 12980 andrew 1password + 0.3 25800 andrew localsearch-ext + total used free shared buff/cache available +Mem: 125 11 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/pre.txt b/bench/wildct/results/2026-11-07/runs/H18w/pre.txt new file mode 100644 index 00000000..554587b2 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/pre.txt @@ -0,0 +1,17 @@ +2026-11-07T04:22:23Z + 23:22:23 up 8 days, 19:11, 2 users, load average: 0.52, 0.47, 0.48 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 1.8 8054 andrew firefox + 1.5 29612 andrew rustdesk + 1.4 1 root systemd + 1.4 9806 andrew WebExtensions + 1.0 9906 911 python3 + 0.8 1184 root gdm + 0.5 7035 andrew rustdesk + 0.4 12980 andrew 1password + 0.3 3313 root dockerd + total used free shared buff/cache available +Mem: 125 10 96 2 21 114 +/dev/nvme0n1p2 3667G 3273G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/rc b/bench/wildct/results/2026-11-07/runs/H18w/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/rc @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl b/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl new file mode 100644 index 00000000..c5d34895 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/sample.jsonl @@ -0,0 +1,49 @@ +{"ts": 1794025349.324, "seconds": 5.0, "load1": 0.48, "busy_p": 0.28, "busy_e_free": 0.146, "busy_e_load": 0.118, "busy_all": 0.544, "mine_cpu": 0.001, "foreign_cpu": 0.543, "other_containers_cpu": 0.047, "mem_available_gib": 114.45, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025354.325, "seconds": 5.001, "load1": 0.52, "busy_p": 0.856, "busy_e_free": 0.3, "busy_e_load": 0.274, "busy_all": 1.43, "mine_cpu": 0.001, "foreign_cpu": 1.429, "other_containers_cpu": 0.072, "mem_available_gib": 114.16, "pkg_temp_c": 64.0, "root_avail_gb": 222.2} +{"ts": 1794025359.326, "seconds": 5.001, "load1": 0.56, "busy_p": 0.752, "busy_e_free": 0.232, "busy_e_load": 0.186, "busy_all": 1.17, "mine_cpu": 0.0, "foreign_cpu": 1.17, "other_containers_cpu": 0.063, "mem_available_gib": 114.05, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025364.327, "seconds": 5.001, "load1": 0.68, "busy_p": 0.48, "busy_e_free": 0.182, "busy_e_load": 0.144, "busy_all": 0.806, "mine_cpu": 0.0, "foreign_cpu": 0.806, "other_containers_cpu": 0.069, "mem_available_gib": 114.0, "pkg_temp_c": 69.0, "root_avail_gb": 222.2} +{"ts": 1794025369.328, "seconds": 5.001, "load1": 0.62, "busy_p": 0.276, "busy_e_free": 0.138, "busy_e_load": 0.12, "busy_all": 0.534, "mine_cpu": 0.0, "foreign_cpu": 0.534, "other_containers_cpu": 0.041, "mem_available_gib": 114.01, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025374.329, "seconds": 5.001, "load1": 0.57, "busy_p": 0.388, "busy_e_free": 0.18, "busy_e_load": 0.136, "busy_all": 0.704, "mine_cpu": 0.0, "foreign_cpu": 0.704, "other_containers_cpu": 0.086, "mem_available_gib": 113.96, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025379.345, "seconds": 5.015, "load1": 0.85, "busy_p": 1.773, "busy_e_free": 0.54, "busy_e_load": 0.399, "busy_all": 2.712, "mine_cpu": 1.906, "foreign_cpu": 0.806, "other_containers_cpu": 0.063, "mem_available_gib": 113.64, "pkg_temp_c": 63.0, "root_avail_gb": 221.2} +{"ts": 1794025384.346, "seconds": 5.001, "load1": 1.02, "busy_p": 1.861, "busy_e_free": 0.588, "busy_e_load": 0.52, "busy_all": 2.969, "mine_cpu": 1.685, "foreign_cpu": 1.284, "other_containers_cpu": 0.094, "mem_available_gib": 113.56, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025389.348, "seconds": 5.001, "load1": 0.94, "busy_p": 0.734, "busy_e_free": 0.236, "busy_e_load": 0.186, "busy_all": 1.156, "mine_cpu": 0.486, "foreign_cpu": 0.67, "other_containers_cpu": 0.069, "mem_available_gib": 113.58, "pkg_temp_c": 63.0, "root_avail_gb": 220.7} +{"ts": 1794025394.349, "seconds": 5.002, "load1": 1.1, "busy_p": 1.31, "busy_e_free": 0.558, "busy_e_load": 0.326, "busy_all": 2.193, "mine_cpu": 1.256, "foreign_cpu": 0.937, "other_containers_cpu": 0.072, "mem_available_gib": 113.49, "pkg_temp_c": 63.0, "root_avail_gb": 220.3} +{"ts": 1794025399.351, "seconds": 5.002, "load1": 1.18, "busy_p": 0.366, "busy_e_free": 0.224, "busy_e_load": 0.13, "busy_all": 0.72, "mine_cpu": 0.194, "foreign_cpu": 0.526, "other_containers_cpu": 0.041, "mem_available_gib": 113.47, "pkg_temp_c": 61.0, "root_avail_gb": 220.3} +{"ts": 1794025404.352, "seconds": 5.001, "load1": 1.08, "busy_p": 0.378, "busy_e_free": 0.138, "busy_e_load": 0.112, "busy_all": 0.628, "mine_cpu": 0.08, "foreign_cpu": 0.548, "other_containers_cpu": 0.083, "mem_available_gib": 114.44, "pkg_temp_c": 65.0, "root_avail_gb": 220.3} +{"ts": 1794025409.367, "seconds": 5.015, "load1": 1.23, "busy_p": 0.467, "busy_e_free": 0.136, "busy_e_load": 0.114, "busy_all": 0.716, "mine_cpu": 0.002, "foreign_cpu": 0.714, "other_containers_cpu": 0.051, "mem_available_gib": 114.43, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025414.368, "seconds": 5.001, "load1": 1.22, "busy_p": 0.624, "busy_e_free": 0.216, "busy_e_load": 0.184, "busy_all": 1.024, "mine_cpu": 0.001, "foreign_cpu": 1.023, "other_containers_cpu": 0.063, "mem_available_gib": 114.26, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794025419.369, "seconds": 5.001, "load1": 1.28, "busy_p": 1.074, "busy_e_free": 0.3, "busy_e_load": 0.222, "busy_all": 1.596, "mine_cpu": 0.0, "foreign_cpu": 1.596, "other_containers_cpu": 0.068, "mem_available_gib": 114.05, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025424.37, "seconds": 5.001, "load1": 1.18, "busy_p": 0.67, "busy_e_free": 0.168, "busy_e_load": 0.144, "busy_all": 0.982, "mine_cpu": 0.0, "foreign_cpu": 0.982, "other_containers_cpu": 0.061, "mem_available_gib": 114.0, "pkg_temp_c": 64.0, "root_avail_gb": 222.2} +{"ts": 1794025429.371, "seconds": 5.001, "load1": 1.08, "busy_p": 0.298, "busy_e_free": 0.12, "busy_e_load": 0.096, "busy_all": 0.514, "mine_cpu": 0.0, "foreign_cpu": 0.514, "other_containers_cpu": 0.044, "mem_available_gib": 114.01, "pkg_temp_c": 68.0, "root_avail_gb": 222.2} +{"ts": 1794025434.372, "seconds": 5.001, "load1": 1.0, "busy_p": 0.338, "busy_e_free": 0.11, "busy_e_load": 0.076, "busy_all": 0.524, "mine_cpu": 0.0, "foreign_cpu": 0.524, "other_containers_cpu": 0.081, "mem_available_gib": 113.98, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025439.386, "seconds": 5.015, "load1": 1.0, "busy_p": 0.54, "busy_e_free": 0.239, "busy_e_load": 0.187, "busy_all": 0.967, "mine_cpu": 0.0, "foreign_cpu": 0.967, "other_containers_cpu": 0.06, "mem_available_gib": 113.9, "pkg_temp_c": 62.0, "root_avail_gb": 222.1} +{"ts": 1794025444.388, "seconds": 5.001, "load1": 1.24, "busy_p": 2.805, "busy_e_free": 0.724, "busy_e_load": 0.648, "busy_all": 4.177, "mine_cpu": 3.156, "foreign_cpu": 1.021, "other_containers_cpu": 0.12, "mem_available_gib": 113.59, "pkg_temp_c": 63.0, "root_avail_gb": 221.2} +{"ts": 1794025449.389, "seconds": 5.001, "load1": 1.3, "busy_p": 1.314, "busy_e_free": 0.42, "busy_e_load": 0.376, "busy_all": 2.109, "mine_cpu": 1.065, "foreign_cpu": 1.044, "other_containers_cpu": 0.075, "mem_available_gib": 113.52, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025454.39, "seconds": 5.001, "load1": 1.19, "busy_p": 0.392, "busy_e_free": 0.132, "busy_e_load": 0.136, "busy_all": 0.66, "mine_cpu": 0.137, "foreign_cpu": 0.523, "other_containers_cpu": 0.054, "mem_available_gib": 113.54, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025459.392, "seconds": 5.001, "load1": 1.1, "busy_p": 0.378, "busy_e_free": 0.126, "busy_e_load": 0.13, "busy_all": 0.634, "mine_cpu": 0.113, "foreign_cpu": 0.521, "other_containers_cpu": 0.042, "mem_available_gib": 113.53, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025464.393, "seconds": 5.001, "load1": 1.09, "busy_p": 0.324, "busy_e_free": 0.168, "busy_e_load": 0.108, "busy_all": 0.6, "mine_cpu": 0.035, "foreign_cpu": 0.565, "other_containers_cpu": 0.091, "mem_available_gib": 113.54, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025469.408, "seconds": 5.015, "load1": 1.0, "busy_p": 0.305, "busy_e_free": 0.122, "busy_e_load": 0.12, "busy_all": 0.546, "mine_cpu": 0.041, "foreign_cpu": 0.505, "other_containers_cpu": 0.054, "mem_available_gib": 113.54, "pkg_temp_c": 62.0, "root_avail_gb": 221.0} +{"ts": 1794025474.409, "seconds": 5.002, "load1": 0.92, "busy_p": 0.31, "busy_e_free": 0.14, "busy_e_load": 0.138, "busy_all": 0.588, "mine_cpu": 0.034, "foreign_cpu": 0.554, "other_containers_cpu": 0.06, "mem_available_gib": 113.56, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025479.411, "seconds": 5.001, "load1": 1.01, "busy_p": 0.558, "busy_e_free": 0.2, "busy_e_load": 0.174, "busy_all": 0.932, "mine_cpu": 0.05, "foreign_cpu": 0.882, "other_containers_cpu": 0.07, "mem_available_gib": 114.71, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025484.412, "seconds": 5.001, "load1": 1.17, "busy_p": 1.14, "busy_e_free": 0.252, "busy_e_load": 0.194, "busy_all": 1.586, "mine_cpu": 0.064, "foreign_cpu": 1.522, "other_containers_cpu": 0.079, "mem_available_gib": 113.4, "pkg_temp_c": 61.0, "root_avail_gb": 221.0} +{"ts": 1794025489.413, "seconds": 5.001, "load1": 1.15, "busy_p": 0.402, "busy_e_free": 0.148, "busy_e_load": 0.126, "busy_all": 0.676, "mine_cpu": 0.048, "foreign_cpu": 0.628, "other_containers_cpu": 0.041, "mem_available_gib": 113.44, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025494.414, "seconds": 5.001, "load1": 1.06, "busy_p": 0.354, "busy_e_free": 0.15, "busy_e_load": 0.1, "busy_all": 0.604, "mine_cpu": 0.037, "foreign_cpu": 0.567, "other_containers_cpu": 0.093, "mem_available_gib": 113.42, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794025499.429, "seconds": 5.014, "load1": 1.38, "busy_p": 0.682, "busy_e_free": 0.207, "busy_e_load": 0.189, "busy_all": 1.079, "mine_cpu": 0.367, "foreign_cpu": 0.712, "other_containers_cpu": 0.055, "mem_available_gib": 113.42, "pkg_temp_c": 62.0, "root_avail_gb": 220.8} +{"ts": 1794025504.43, "seconds": 5.001, "load1": 1.51, "busy_p": 1.036, "busy_e_free": 0.414, "busy_e_load": 0.294, "busy_all": 1.744, "mine_cpu": 1.102, "foreign_cpu": 0.642, "other_containers_cpu": 0.094, "mem_available_gib": 113.4, "pkg_temp_c": 63.0, "root_avail_gb": 220.3} +{"ts": 1794025509.431, "seconds": 5.001, "load1": 1.39, "busy_p": 0.302, "busy_e_free": 0.134, "busy_e_load": 0.14, "busy_all": 0.576, "mine_cpu": 0.046, "foreign_cpu": 0.53, "other_containers_cpu": 0.06, "mem_available_gib": 113.42, "pkg_temp_c": 61.0, "root_avail_gb": 220.3} +{"ts": 1794025514.433, "seconds": 5.002, "load1": 1.27, "busy_p": 0.32, "busy_e_free": 0.14, "busy_e_load": 0.144, "busy_all": 0.604, "mine_cpu": 0.045, "foreign_cpu": 0.559, "other_containers_cpu": 0.062, "mem_available_gib": 113.44, "pkg_temp_c": 62.0, "root_avail_gb": 220.3} +{"ts": 1794025519.434, "seconds": 5.001, "load1": 1.25, "busy_p": 0.71, "busy_e_free": 0.398, "busy_e_load": 0.192, "busy_all": 1.3, "mine_cpu": 0.0, "foreign_cpu": 1.3, "other_containers_cpu": 0.044, "mem_available_gib": 114.44, "pkg_temp_c": 65.0, "root_avail_gb": 220.6} +{"ts": 1794025524.435, "seconds": 5.001, "load1": 1.15, "busy_p": 0.46, "busy_e_free": 0.126, "busy_e_load": 0.086, "busy_all": 0.672, "mine_cpu": 0.001, "foreign_cpu": 0.671, "other_containers_cpu": 0.08, "mem_available_gib": 114.43, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025529.447, "seconds": 5.012, "load1": 1.22, "busy_p": 0.928, "busy_e_free": 0.251, "busy_e_load": 0.223, "busy_all": 1.403, "mine_cpu": 0.001, "foreign_cpu": 1.402, "other_containers_cpu": 0.053, "mem_available_gib": 114.14, "pkg_temp_c": 66.0, "root_avail_gb": 222.2} +{"ts": 1794025534.448, "seconds": 5.001, "load1": 1.2, "busy_p": 0.67, "busy_e_free": 0.222, "busy_e_load": 0.19, "busy_all": 1.082, "mine_cpu": 0.0, "foreign_cpu": 1.082, "other_containers_cpu": 0.07, "mem_available_gib": 114.06, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025539.449, "seconds": 5.001, "load1": 1.35, "busy_p": 0.524, "busy_e_free": 0.206, "busy_e_load": 0.138, "busy_all": 0.868, "mine_cpu": 0.0, "foreign_cpu": 0.868, "other_containers_cpu": 0.063, "mem_available_gib": 113.95, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025544.45, "seconds": 5.001, "load1": 1.24, "busy_p": 0.31, "busy_e_free": 0.146, "busy_e_load": 0.114, "busy_all": 0.57, "mine_cpu": 0.0, "foreign_cpu": 0.57, "other_containers_cpu": 0.064, "mem_available_gib": 113.96, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} +{"ts": 1794025549.451, "seconds": 5.001, "load1": 1.14, "busy_p": 0.316, "busy_e_free": 0.168, "busy_e_load": 0.16, "busy_all": 0.644, "mine_cpu": 0.0, "foreign_cpu": 0.644, "other_containers_cpu": 0.04, "mem_available_gib": 113.86, "pkg_temp_c": 61.0, "root_avail_gb": 222.2} +{"ts": 1794025554.452, "seconds": 5.001, "load1": 1.37, "busy_p": 1.07, "busy_e_free": 0.328, "busy_e_load": 0.256, "busy_all": 1.654, "mine_cpu": 0.0, "foreign_cpu": 1.654, "other_containers_cpu": 0.098, "mem_available_gib": 113.74, "pkg_temp_c": 62.0, "root_avail_gb": 221.7} +{"ts": 1794025559.464, "seconds": 5.013, "load1": 1.34, "busy_p": 0.307, "busy_e_free": 0.128, "busy_e_load": 0.124, "busy_all": 0.559, "mine_cpu": 0.027, "foreign_cpu": 0.532, "other_containers_cpu": 0.047, "mem_available_gib": 113.72, "pkg_temp_c": 61.0, "root_avail_gb": 221.7} +{"ts": 1794025564.466, "seconds": 5.001, "load1": 1.23, "busy_p": 0.61, "busy_e_free": 0.272, "busy_e_load": 0.192, "busy_all": 1.074, "mine_cpu": 0.476, "foreign_cpu": 0.598, "other_containers_cpu": 0.077, "mem_available_gib": 113.69, "pkg_temp_c": 63.0, "root_avail_gb": 221.4} +{"ts": 1794025569.467, "seconds": 5.001, "load1": 1.37, "busy_p": 1.312, "busy_e_free": 0.468, "busy_e_load": 0.338, "busy_all": 2.117, "mine_cpu": 1.125, "foreign_cpu": 0.992, "other_containers_cpu": 0.074, "mem_available_gib": 113.6, "pkg_temp_c": 63.0, "root_avail_gb": 221.3} +{"ts": 1794025574.469, "seconds": 5.002, "load1": 1.34, "busy_p": 1.016, "busy_e_free": 0.364, "busy_e_load": 0.14, "busy_all": 1.519, "mine_cpu": 0.755, "foreign_cpu": 0.764, "other_containers_cpu": 0.057, "mem_available_gib": 113.54, "pkg_temp_c": 63.0, "root_avail_gb": 220.8} +{"ts": 1794025579.47, "seconds": 5.002, "load1": 1.4, "busy_p": 1.729, "busy_e_free": 0.676, "busy_e_load": 0.338, "busy_all": 2.743, "mine_cpu": 2.006, "foreign_cpu": 0.737, "other_containers_cpu": 0.061, "mem_available_gib": 113.49, "pkg_temp_c": 62.0, "root_avail_gb": 220.3} +{"ts": 1794025584.471, "seconds": 5.001, "load1": 1.36, "busy_p": 0.738, "busy_e_free": 0.462, "busy_e_load": 0.316, "busy_all": 1.516, "mine_cpu": 0.0, "foreign_cpu": 1.516, "other_containers_cpu": 0.091, "mem_available_gib": 114.43, "pkg_temp_c": 63.0, "root_avail_gb": 222.2} +{"ts": 1794025589.485, "seconds": 5.013, "load1": 1.25, "busy_p": 0.345, "busy_e_free": 0.15, "busy_e_load": 0.118, "busy_all": 0.612, "mine_cpu": 0.0, "foreign_cpu": 0.612, "other_containers_cpu": 0.051, "mem_available_gib": 114.39, "pkg_temp_c": 62.0, "root_avail_gb": 222.2} diff --git a/bench/wildct/results/2026-11-07/runs/H18w/wait_s b/bench/wildct/results/2026-11-07/runs/H18w/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H18w/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/campaign.log b/bench/wildct/results/2026-11-07/runs/campaign.log index 66cf4dac..896b5dfa 100644 --- a/bench/wildct/results/2026-11-07/runs/campaign.log +++ b/bench/wildct/results/2026-11-07/runs/campaign.log @@ -25,3 +25,7 @@ == H52w lock taken 2026-11-07T04:03:54Z == H52w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h52w 2026-11-07T04:03:54Z == H52w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:05:00Z +== H18w waiting for the lock 2026-11-07T04:22:23Z +== H18w lock taken 2026-11-07T04:22:23Z +== H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z +== H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index b251544d..00c45b63 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,17 +1,17 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d and H52w runs (82 rows). +Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d, H52w and H18w runs (82 rows). ## Abstract k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w and H18w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. -- Twenty hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w to wildct, an i7-13700. H52w is the newest: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. +- 21 hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w and H18w to wildct, an i7-13700. H18w is the newest: H18 replicated on that host. In 3 of 3 runs `k3d-gpu sandbox up` from the release kept the sandbox on an E-core (CPU 17) behind default-deny egress: the host's tailscale and LAN addresses were refused from the sandbox and reached from a control pod, and `pip install --user` from PyPI worked [ledger:H18w]. Before it, H52w: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. - Twenty are REFUTED. Six of them changed shipped defaults. The newest, H28x, is H28 replicated on the Xeon 6741P: eight pods that mmap one 1 GiB model through an image volume again shared one page-cache copy (PSS 132.6 MiB per pod, 1.00x the size on the node against 9.00x for per-pod copies), but they were ready in 0.714 of the copy arm's time, not the 0.5 or less the row needed, so only the memory part replicated [ledger:H28x]. Before it, H7x is H7 run to its full protocol on the Xeon 6741P: at matched 16-bit weights (Qwen3-4B, 16 CPUs), OVMS 2026.4.1 decoded at 1.22x the rate of EAT 1.1.0's stock vLLM CPU engine (v0.28.0) in all five block pairs, so OVMS's decode lead is not only weight precision; vLLM prefilled the long prompt faster [ledger:H7x]. Before it, H58 is the zero-touch thesis on the Xeon 6741P, with the two serving pairs holding about half of every B70. With no flag, `k3d-gpu up` advertised three of the four B70s and no DSA work queue. `k3d-gpu agent up --model` then refused all three models. Gemma 4 26B-A4B and Qwen3.8-27B went to OVMS on the GPU, where their int4 IRs missed the best card's free-memory budget by 0.1 and 0.5 GiB. Qwen3.8-Flash-Next was refused because no DRAM bandwidth was cached. So EAT's banking flow never came up [ledger:H58]. Before it, H52b is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. - 34 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in a KVM guest on the Xeon 6741P) is INCONCLUSIVE because Intel's own install did not finish: enterprise-inference's `cpu: auto` sized the Gemma 4 pod to 76 of the guest's 95.4 allocatable CPUs, so EAT's sandbox router could not be scheduled, twice, and no hold ran. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H55 (SLO-gated admission of an E-core neighbour on the i9-13900T) is INCONCLUSIVE: all 25 runs completed on a busy host, but in one round the controller's run closed three request windows, one short of its gate; not gated, the controller held the neighbour at its lowest quota in four of five rounds, and no arm kept prefill within the row's bar of quiet. H59d (the engine choice on the i9-13900T: llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter against OVMS with OpenVINO's base-model int4 IR) is INCONCLUSIVE because the llama.cpp arm was not deliverable in two of its four holds: in both holds where it followed OVMS, its server was OOMKilled at the fit's 45Gi pod limit (4 x 262,144 tokens), while the same render served in both holds where it ran first. Not gated, in the one performance hold it served it decoded 1.05x OVMS per request at four concurrent requests, and GSM8K read 50 against 49 of 50. So with no backend set, the i9-13900T keeps OVMS. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 8 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H18w, H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 7 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -48,7 +48,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p ## Results -### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w to wildct's i7-13700) +### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w and H18w to wildct's i7-13700) | Row | Result | Scope limit | |---|---|---| @@ -71,6 +71,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H60 | Qwen3.8-Flash-Next (125B MoE, ggml-org IQ4_NL GGUF in tmpfs) served by llama.cpp v0.8.0 from the Xeon 6741P's DRAM decoded 16.00 tok/s at one stream, 0.312 of the 51.3 tok/s ceiling that 305 GB/s gives at 5.94 GB read per token. A same-hold Gemma 4 QAT Q4_0 control on the same build read 0.379 (48.52 tok/s), so the ratio was 0.823, CI 0.819-0.831, inside the 0.7-1.3 band; GSM8K items 0-49 scored 50/50 [ledger:H60] | One hold, 48 threads on the 48 physical cores, CPU only, greedy, thinking off for Flash-Next (the control opened its thinking block). The control's decode window starts about 3 tokens after its first chunk, so its rate reads about 1% high; timed from the first chunk, the ratio is 0.833. The ceiling uses a fixed STREAM figure; package watts include about 130 W of other tenants' idle draw | | H40 | The agent harness reproduces the BFCL board: bf16 Qwen3-4B-Instruct-2507 on vLLM CUDA scored 88.0% on the fixed 100-id subset in BFCL's own mode (board-weighted 84.9%, band 77.9-91.9%), every category inside its binomial interval [ledger:H40] | A check of the harness, not of the hardware; one 100-id subset, one run, on the NVIDIA dGPU | | H52w | On wildct (an i7-13700 with two NVIDIA cards and a UHD 770 iGPU), a plain `k3d-gpu up` with the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`) detected every CPU engine present and nothing phantom: the hybrid P/E cores with the balloons policy (`k3d-gpu test`'s P-core pod on CPU 0, its E-core pod on CPU 17) and the launcher's six ISA labels (AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS); no AVX-512, AMX or NUMA label, and no DSA, IAA, QAT, NPU or Gaudi resource or pod. Both GPU vendors were named as detected and opted out, no GPU resource was allocatable and no pod mounted a `/dev` path [ledger:H52w] | One hold, release 0.1.89. Detection with the GPUs opted out: says nothing about a GPU reaching a pod on this host | +| H18w | H18 replicated on wildct (an i7-13700, E-cores 16-23) with the release node image and the box owner's GPU opt-out: in 3 of 3 runs `k3d-gpu sandbox up` exited 0 with the sandbox pod on CPU 17; TCP from the sandbox to the host's tailscale (`100.66.38.121:22`) and LAN (`192.168.8.238:22`) addresses was refused, with 2 packets on kube-router's drop-log rule, while a control pod without the policy connected to both; `pip install --user tomli_w==1.2.0` from PyPI exited 0 [ledger:H18w] | One hold, release 0.1.89, agent-sandbox v1.0.8, kube-router on k3s v1.37.1, IPv4 targets only | ### REFUTED (all but H5, H7x, H27, H27x, H28x, H34x, H43, H45, H48, H50, H51 and H52b changed a default) From bba4afe50e052949209c59bb2ceea8199bda99e6 Mon Sep 17 00:00:00 2001 From: Andrew Mello Date: Fri, 6 Nov 2026 20:46:02 -0800 Subject: [PATCH 4/4] bench(wildct): H18w run review fixes: docs anchor, partner update, four record readings The technical report's SUPPORTED heading now names H18w, which moved its anchor, so the partner update's link to it failed mkdocs --strict (the Docs build on this PR). The link now uses the new anchor. The partner update also lists H18w with the wildct SUPPORTED rows and no longer counts it among the PENDING wildct rows. In the H18w run section and its ledger row, four readings now say what the records hold. The 207 in "/ 207 GB available" is df -BG's GiB figure; sample.py read 220.3-222.2 GB, and the swap reading is the runner's, not sample.py's. The 0.08-0.09 s control connects are those to the two private addresses (the control pod's pypi.org:443 connects took 0.09-0.14 s). The 18-of-18 module read is from before the hold; after it the records hold the module count (224) and kernel.modprobe. The pip result is the runner's pip_rc, as the section's own note on pip's exit says, and the technical report's row states it the same way. No prediction, measurement or verdict changes. --- EXPERIMENTS.md | 13 ++++++++----- docs/intel/partner-update-2026-11.md | 6 +++--- docs/intel/technical-report.md | 2 +- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index f77d1abd..9c3e3a53 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -85,7 +85,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57dp | On the i9-13900T P-cores under the shared host's own load, llama.cpp with Gemma 4 QAT Q4_0 and the QAT MTP drafter decodes faster unpinned (8 threads, no `-C`/`--cpu-strict`) than strictly pinned one thread per physical P-core: N=1 ratio >= 1.10. | Written 2026-10-21, after H57d, before any H57dp measurement. SUPPORTED (0.5); Ld-mtp-u/Ld-mtp in [1.0, 1.8] (0.6). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): 2 holds of at most 75 min, H57d's protocol. SUPPORTED: pooled median ratio >= 1.10, CI lower bound > 1.0, faster in both holds. REFUTED: CI upper bound < 1.10. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched, foreign P-core load spread 2.97 > 2.0 CPU-s/s (Od 0.45, Ld-mtp 3.42, Ld-mtp-u 3.11), so the scorer stops before the statistic. Not gated: Ld-mtp-u/Ld-mtp 1.108 (95% CI 1.072-1.142), faster in both holds (1.108, 1.100); Ld-mtp-u 13.39, Ld-mtp 12.09 tok/s pooled N=1; affinity as registered. | INCONCLUSIVE | | H57de | H57da with the unpinned llama.cpp arm: no arm decodes Gemma 4 QAT at N=1 5% or more faster than k3d-gpu's OVMS pick Od on the i9-13900T P-cores. | Written 2026-10-21, before any H57de measurement. REFUTED (0.5); Ld-mtp-u/Od in [1.0, 1.6] (0.5). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): same holds as H57dp. REFUTED: Ld-mtp-u/Od CI lower bound >= 1.05 and faster in both holds. SUPPORTED: CI upper bound < 1.05. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched (the same gate, spread 2.97 > 2.0). Od also ran without the P-core balloon (cpuset 0-15,17-31: the runner's `agent manifest` call named no cluster, so after #183 it missed the hold's kube context), so it was not the registered Od. Not gated: Ld-mtp-u/Od 1.369 (1.345-1.396), Od 9.78 tok/s pooled N=1. | INCONCLUSIVE | | H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. Result ([H52w run](#h52w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign). `up` took 32 s, named nvidia and intel as opted out and printed `Accelerators: hybrid`; the balloons policy pod Running, `k3d-gpu test`'s P-core pod on CPU 0 and its E-core pod on CPU 17; the launcher's ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT and CETSS, no AVX512 or AMX label from any source; no NUMA label; allocatable cpu, ephemeral-storage, hugepages, memory and pods only; no GPU plugin pod, no `/dev` hostPath and no GPU resource (`GPU-SCAN clean`); the node container `gpus=[]`; the inventory unchanged. Every row matches. | SUPPORTED | -| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. Result ([H18w run](#h18w-run-wildct-i7-13700-2026-11-07)): one hold of 3 runs on release 0.1.89 (rule W7's release for the campaign), a fresh cluster each. In every run `sandbox up` exited 0 with the warm pool's `readyReplicas` 1 and the sandbox pod on CPU 17, inside the E-cores 16-23; from the sandbox, TCP to `100.66.38.121:22` and to `192.168.8.238:22` was refused in 0.11-0.12 s with 2 packets on the policy's drop-log rule, and from the control pod both connected; `pypi.org` resolved, `pypi.org:443` connected and `pip install --user tomli_w==1.2.0` exited 0, importing from `/tmp/.local`. Every run meets (a), (b) and (c). | SUPPORTED | +| H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. Result ([H18w run](#h18w-run-wildct-i7-13700-2026-11-07)): one hold of 3 runs on release 0.1.89 (rule W7's release for the campaign), a fresh cluster each. In every run `sandbox up` exited 0 with the warm pool's `readyReplicas` 1 and the sandbox pod on CPU 17, inside the E-cores 16-23; from the sandbox, TCP to `100.66.38.121:22` and to `192.168.8.238:22` was refused in 0.11-0.12 s with 2 packets on the policy's drop-log rule, and from the control pod both connected; `pypi.org` resolved, `pypi.org:443` connected and `pip install --user tomli_w==1.2.0` gave `pip_rc=0`, with the import reading `/tmp/.local`. Every run meets (a), (b) and (c). | SUPPORTED | | H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | | H34w | H34's hypothesis on wildct: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, both on the 16-CPU P-core balloon, under matched background load. A replication: original models, runner, scorer and rule. | Written 2026-10-20, before any H34w measurement. At `typical`: median paired B/A decode tok/s 1.35 (1.15-1.6) and median paired B/A package J/token 0.76 (0.6-0.92). SUPPORTED by H34's rule at 0.6, REFUTED 0.25, INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, `typical` only, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h34`, the GPU opt-out in place of H34's `intel,hybrid`, W2, W3, W4. | PENDING | @@ -8940,7 +8940,8 @@ environment: IMAGE=, K3D_GPU_CLUSTER=k3dgpu-w-sbx, K3D_GPU_V energy source: intel-rapl:0 energy_uj (package-0), confounds only: package W over 5 s idle before each run runs: 3 counted, one fresh cluster each, in one hold; no pilot host load: load1 0.48-1.51, foreign CPU 0.51-1.65 CPU-s/s, MemAvailable 113.4-114.7 GiB, package - 61-69 C, swap used 0 MiB (sample.py, 49 samples; df: / 207 GB available before and after) + 61-69 C, / 220.3-222.2 GB available (sample.py, 49 samples; df -BG read 207G before and + after); swap used 0 MiB at each run's start (the runner's read) ``` Command, on wildct: RESUME.md step 6's `launch H18w 2100 3 h18w`, which runs `bench/wildct/hold.sh H18w @@ -8973,7 +8974,8 @@ In each run: `Cpus_allowed_list` was 17, with the host's `cpu_atom` 16-23. - Every blocked connect printed `NOCONNECT ConnectionRefusedError [Errno 111] Connection refused`. Each of kube-router's policy chains (one per cluster) counted 2 packets on its NFLOG rule - ("DROP by policy agent-sandbox/python-ecores-network-policy"). Each control connect took 0.08-0.09 s. + ("DROP by policy agent-sandbox/python-ecores-network-policy"). Each control connect to those two + addresses took 0.08-0.09 s. - `pypi.org` resolved to 151.101.0.223 (runs 1 and 2) and 151.101.192.223 (run 3), both Fastly. The sandbox's connect to `pypi.org:443` took 2.33, 0.22 and 0.14 s. - The sandbox ran with `HOME=/tmp uid=1000`. pip's last line of output was its upgrade notice, and the @@ -9009,8 +9011,9 @@ containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the s line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` cluster or container, no process of the hold and no network or volume of ours was left, and -`live.lock` was free. The host's loaded module count read 224 after the hold, with all 18 modules of -`/etc/modules-load.d/k3dgpu-wildct.conf` loaded and `kernel.modprobe` empty. +`live.lock` was free. Before the hold (04:20Z) all 18 modules of +`/etc/modules-load.d/k3dgpu-wildct.conf` were loaded and `kernel.modprobe` was empty; after it +(04:27Z) the host had 224 modules loaded and `kernel.modprobe` was still empty. Holds used: 2 of at most 12. The next row in the registered order is H25w. diff --git a/docs/intel/partner-update-2026-11.md b/docs/intel/partner-update-2026-11.md index 0e2ab7a6..ad2804c8 100644 --- a/docs/intel/partner-update-2026-11.md +++ b/docs/intel/partner-update-2026-11.md @@ -58,8 +58,8 @@ All drafts are unsent and need a maintainer's approval before anything is filed. SUPPORTED rows, each scoped to its host: - On the Xeon 6741P: the CPU engine choice [ledger:H59], DRAM-resident serving of Qwen3.8-Flash-Next [ledger:H60], and H25 replicated [ledger:H25x]. -- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-to-wildcts-i7-13700). -- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w]. +- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-and-h18w-to-wildcts-i7-13700). +- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w], and H18 replicated, agent sandboxes on its E-cores behind default-deny egress [ledger:H18w]. ## Two results that help EAT @@ -73,7 +73,7 @@ v0.28.0 is past the v0.24.0 CPU attention hang that stalled EAT 1.0.3's stock ar - **The Keycloak argv secret.** F3 stays REFUTED. `k3d-gpu eas up` now patches enterprise-inference's `keycloak_config` role so that neither the admin password nor the client secret is a `kcadm.sh` argument ([State and secrets](eas.md#state-and-secrets)). The patch is checked with a `ps` sampler against a stand-in for kcadm; no live run has checked it yet. A fresh `eas up --agent-toolkit` with F3's secret scan would. - **Intel's default install in a guest (H62).** A re-run needs the `cpu: auto` sizing to leave room for EAT's sandbox router, or a guest shape where it does. - **k3d-gpu gaps found on the Xeon.** `/dev/dri/by-path` for oneCCL across B70s (H53); the fit refusing rather than falling back to the CPU, and no DRAM bandwidth cached on a new host (H58); placing one-card arms on one card (H54b, H54c). -- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H52w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. +- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H18w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. - **EAT 1.1.0 through `eat-up.sh`.** Pinned since 2026-10-29 and pre-registered as F4; not run yet. - **EAS upstream moves.** EAS #61 merged on 2026-10-30 (EAS `main` `0f7bc2b`). The pin stays at `97f6157`; moving past #61 needs patch 0002 regenerated. EAS #50 and #52 (OpenShift) are open and would move 0006. EAS #71 (teardown hang) is open; `eas down` uses its workaround. - **ERAG and Arc under EAS.** EAS's Enterprise RAG layer is not wired into `k3d-gpu eas` yet, and EAS has no Intel GPU path on this branch; Arc and Gaudi stay with `k3d-gpu agent up`. diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index 00c45b63..92b12015 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -71,7 +71,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H60 | Qwen3.8-Flash-Next (125B MoE, ggml-org IQ4_NL GGUF in tmpfs) served by llama.cpp v0.8.0 from the Xeon 6741P's DRAM decoded 16.00 tok/s at one stream, 0.312 of the 51.3 tok/s ceiling that 305 GB/s gives at 5.94 GB read per token. A same-hold Gemma 4 QAT Q4_0 control on the same build read 0.379 (48.52 tok/s), so the ratio was 0.823, CI 0.819-0.831, inside the 0.7-1.3 band; GSM8K items 0-49 scored 50/50 [ledger:H60] | One hold, 48 threads on the 48 physical cores, CPU only, greedy, thinking off for Flash-Next (the control opened its thinking block). The control's decode window starts about 3 tokens after its first chunk, so its rate reads about 1% high; timed from the first chunk, the ratio is 0.833. The ceiling uses a fixed STREAM figure; package watts include about 130 W of other tenants' idle draw | | H40 | The agent harness reproduces the BFCL board: bf16 Qwen3-4B-Instruct-2507 on vLLM CUDA scored 88.0% on the fixed 100-id subset in BFCL's own mode (board-weighted 84.9%, band 77.9-91.9%), every category inside its binomial interval [ledger:H40] | A check of the harness, not of the hardware; one 100-id subset, one run, on the NVIDIA dGPU | | H52w | On wildct (an i7-13700 with two NVIDIA cards and a UHD 770 iGPU), a plain `k3d-gpu up` with the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`) detected every CPU engine present and nothing phantom: the hybrid P/E cores with the balloons policy (`k3d-gpu test`'s P-core pod on CPU 0, its E-core pod on CPU 17) and the launcher's six ISA labels (AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS); no AVX-512, AMX or NUMA label, and no DSA, IAA, QAT, NPU or Gaudi resource or pod. Both GPU vendors were named as detected and opted out, no GPU resource was allocatable and no pod mounted a `/dev` path [ledger:H52w] | One hold, release 0.1.89. Detection with the GPUs opted out: says nothing about a GPU reaching a pod on this host | -| H18w | H18 replicated on wildct (an i7-13700, E-cores 16-23) with the release node image and the box owner's GPU opt-out: in 3 of 3 runs `k3d-gpu sandbox up` exited 0 with the sandbox pod on CPU 17; TCP from the sandbox to the host's tailscale (`100.66.38.121:22`) and LAN (`192.168.8.238:22`) addresses was refused, with 2 packets on kube-router's drop-log rule, while a control pod without the policy connected to both; `pip install --user tomli_w==1.2.0` from PyPI exited 0 [ledger:H18w] | One hold, release 0.1.89, agent-sandbox v1.0.8, kube-router on k3s v1.37.1, IPv4 targets only | +| H18w | H18 replicated on wildct (an i7-13700, E-cores 16-23) with the release node image and the box owner's GPU opt-out: in 3 of 3 runs `k3d-gpu sandbox up` exited 0 with the sandbox pod on CPU 17; TCP from the sandbox to the host's tailscale (`100.66.38.121:22`) and LAN (`192.168.8.238:22`) addresses was refused, with 2 packets on kube-router's drop-log rule, while a control pod without the policy connected to both; `pip install --user tomli_w==1.2.0` from PyPI installed into `/tmp/.local`, where the import found it [ledger:H18w] | One hold, release 0.1.89, agent-sandbox v1.0.8, kube-router on k3s v1.37.1, IPv4 targets only | ### REFUTED (all but H5, H7x, H27, H27x, H28x, H34x, H43, H45, H48, H50, H51 and H52b changed a default)