diff --git a/EXPERIMENTS.md b/EXPERIMENTS.md index 039f8cf4..a2c39658 100644 --- a/EXPERIMENTS.md +++ b/EXPERIMENTS.md @@ -86,7 +86,7 @@ setup too weak to decide). Only `SUPPORTED` backs a claim. | H57de | H57da with the unpinned llama.cpp arm: no arm decodes Gemma 4 QAT at N=1 5% or more faster than k3d-gpu's OVMS pick Od on the i9-13900T P-cores. | Written 2026-10-21, before any H57de measurement. REFUTED (0.5); Ld-mtp-u/Od in [1.0, 1.6] (0.5). | [H57dp pre-registration](#h57dp-pre-registration-llamacpp-thread-pinning-on-the-busy-client-i9-13900t-2026-10-21): same holds as H57dp. REFUTED: Ld-mtp-u/Od CI lower bound >= 1.05 and faster in both holds. SUPPORTED: CI upper bound < 1.05. Else INCONCLUSIVE, or when the load gate fails. Run 2026-10-29 ([H57dp run](#h57dp-run-i9-13900t-2026-10-29)): load unmatched (the same gate, spread 2.97 > 2.0). Od also ran without the P-core balloon (cpuset 0-15,17-31: the runner's `agent manifest` call named no cluster, so after #183 it missed the hold's kube context), so it was not the registered Od. Not gated: Ld-mtp-u/Od 1.369 (1.345-1.396), Od 9.78 tok/s pooled N=1. | INCONCLUSIVE | | H52w | A plain `k3d-gpu up` on wildct, a client Raptor Lake host (i7-13700, 8 P-cores + 8 E-cores, AVX2 and AVX-VNNI, no AVX-512, no AMX, one NUMA node), with only the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`), detects every CPU engine actually present and advertises each usable one, with nothing missing and nothing phantom; the two NVIDIA cards and the Intel iGPU are named as detected and opted out, and nothing in the cluster mounts or advertises a GPU. | Written 2026-10-20 from an inventory read from sysfs, /proc and PCI sysfs, not from k3d-gpu. Hybrid active, balloons policy running, the P-core test pod inside CPUs 0-15 and the E-core one inside 16-23 (0.8); launcher ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS, and no AVX512 or AMX label from any source (0.85); no NUMA, DSA, IAA, QAT, NPU or Gaudi object (0.9); `up` names nvidia and intel as opted out, no GPU resource, no GPU plugin pod, no pod mounting `/dev/dri` or `/dev/nvidia*` (0.85, given the opt-out fix). SUPPORTED (0.6), REFUTED (0.4). | The table in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): `bench/wildct/h52w.sh` under `bench/wildct/hold.sh` (inventory, `detect` as found, `doctor`, `up`, node object, pod GPU scan, `k3d-gpu test`, `down`, inventory again), the installed release. SUPPORTED: every row matches. REFUTED: any row does not. Result ([H52w run](#h52w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign). `up` took 32 s, named nvidia and intel as opted out and printed `Accelerators: hybrid`; the balloons policy pod Running, `k3d-gpu test`'s P-core pod on CPU 0 and its E-core pod on CPU 17; the launcher's ISA labels exactly AVXVNNI, VAES, GFNI, SHA, CETIBT and CETSS, no AVX512 or AMX label from any source; no NUMA label; allocatable cpu, ephemeral-storage, hugepages, memory and pods only; no GPU plugin pod, no `/dev` hostPath and no GPU resource (`GPU-SCAN clean`); the node container `gpus=[]`; the inventory unchanged. Every row matches. | SUPPORTED | | H18w | H18's hypothesis on wildct: `k3d-gpu sandbox up` keeps agent sandboxes on the E-cores and inside a default-deny egress policy on k3d (kube-router), while public egress (PyPI) still works. A replication: H18's runner, image source (the release node image) and parts (a), (b), (c). | Written 2026-10-20, before any H18w measurement. Every run meets (a), (b) and (c) (0.8): sandbox `Cpus_allowed_list` inside 16-23; TCP to `100.66.38.121:22` (tailscale CGNAT) and `192.168.8.238:22` (host LAN) refused from the sandbox with the drop-log counter > 0, connected from the control pod; `pip install --user tomli_w==1.2.0` exits 0. | H18's `bench/sandbox_round.sh`, n = 3 runs in one hold, unchanged except as listed in [the wildct pre-registration](#wildct-pre-registration-client-raptor-lake-i7-13700-2026-10-20): cluster `k3dgpu-w-sbx`, the GPU opt-out, rule W3, the release node image, RAPL through rule W4. Outcome table: H18's. Result ([H18w run](#h18w-run-wildct-i7-13700-2026-11-07)): one hold of 3 runs on release 0.1.89 (rule W7's release for the campaign), a fresh cluster each. In every run `sandbox up` exited 0 with the warm pool's `readyReplicas` 1 and the sandbox pod on CPU 17, inside the E-cores 16-23; from the sandbox, TCP to `100.66.38.121:22` and to `192.168.8.238:22` was refused in 0.11-0.12 s with 2 packets on the policy's drop-log rule, and from the control pod both connected; `pypi.org` resolved, `pypi.org:443` connected and `pip install --user tomli_w==1.2.0` gave `pip_rc=0`, with the import reading `/tmp/.local`. Every run meets (a), (b) and (c). | SUPPORTED | -| H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). | PENDING | +| H25w | H25's hypothesis (T part) on wildct: agents that send a byte-identical system prompt and tools to one OVMS CPU server reuse one prefix KV, so an added agent's first chunk arrives at least 4x sooner with the shipping prefix caching on than with `--enable_prefix_caching=false`. A replication: original model, prefix shape, probe, scorer and thresholds. | Written 2026-10-20, before any H25w measurement. Added-agent median TTFT OFF 10 s (7-15), ON 1.0 s (0.6-1.6), ratio 10x (6-14), CI lower bound >= 4x; c/OFF >= 0.8. SUPPORTED 0.8. | H25's `bench/shared-prefix/run.sh` and `probe.py score` (A3 first-chunk TTFT, A4 amendment-27 load rule), ON OFF interleaved 3+3 rounds at `typical`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h25`, the GPU opt-out, rule W2 (load on E-cores 20-23 only), W3, W4, new seeds (`SEED_BASE=2560`). Both arms keep the CPU Fast Draft the line registers through `K3D_GPU_AGENT_DRAFT=CPU`, since the launcher's default no longer drafts on a CPU class no H34-family row measured: [wildct amendment R7](#wildct-amendment-r7-h25ws-cpu-fast-draft-2026-11-07) (2026-11-07, before any H25w hold). Result ([H25w run](#h25w-run-wildct-i7-13700-2026-11-07)): one hold on release 0.1.89 (rule W7's release for the campaign), load MATCHED (amendment 27), no round void; round 6 (OFF) was cut by the level's 18:30 cap, so OFF has 2 rounds (n = 10) and ON 3 (n = 15). Added agent's first chunk 2.198 s ON vs 31.437 s OFF (14.30x, CI 13.38-14.89), c/OFF 1.00, a1 ON/OFF 0.97, every round pair >= 4x: T SUPPORTED. Outside the point ranges: OFF and ON took 2.9x and 2.2x H25 run 4's times on the i9, the package at 33.6-36.3 W (PL1 35 W). | SUPPORTED | | H28w | H28's hypothesis on wildct: N = 8 agent pods that mmap one 1 GiB model through an image volume share one page-cache copy and become ready in at most half the time of pods that copy it into an emptyDir. A replication (H28 has no model; the 1 GiB blob is kept). | Written 2026-10-20, before any H28w measurement. Memory parts hold (0.95): T PSS <= 160 MiB per pod, model page cache about 1.0x size (T) vs about 9x (C). Time part: T/C round-median ready 0.5 (0.4-0.65), against H28's 0.47 on the i9. SUPPORTED 0.45, REFUTED (time part) 0.4, INCONCLUSIVE or not run (disk) 0.15. | H28's `bench/zero-copy-models/probe.sh` (A1, A2), arms C T T C C T plus the unscored warm-up, `typical` only, scored by `bench/zero-copy-models/score.py`, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h28`, node on the P-cores (`NODE_CPUS=0-15`), W2, W3 (passed to the plain `k3d cluster create`), and a disk gate (W6). | PENDING | | H34w | H34's hypothesis on wildct: same-engine CPU Fast Draft (OVMS 2026.4.1, `OpenVINO/Qwen3-0.6B-int4-ov` draft on the CPU) beats the `OpenVINO/Qwen3-4B-int4-ov` target alone on single-stream decode, both on the 16-CPU P-core balloon, under matched background load. A replication: original models, runner, scorer and rule. | Written 2026-10-20, before any H34w measurement. At `typical`: median paired B/A decode tok/s 1.35 (1.15-1.6) and median paired B/A package J/token 0.76 (0.6-0.92). SUPPORTED by H34's rule at 0.6, REFUTED 0.25, INCONCLUSIVE 0.15. | H34's `bench/hetero-spec/h34.sh` and `h34.py`, n = 12 pairs A B / B A, 5 s load window per request, `typical` only, unchanged except as listed in the wildct pre-registration: cluster `k3dgpu-w-h34`, the GPU opt-out in place of H34's `intel,hybrid`, W2, W3, W4. | PENDING | | H57wa | H57da's hypothesis on wildct (i7-13700, the 16 P-core CPUs, AVX2 and AVX-VNNI, no AVX-512, no AMX): with the Gemma 4 26B-A4B QAT family, k3d-gpu's own CPU pick Od (`agent manifest` auto on the P-cores: OVMS 2026.4.1, `OpenVINO/gemma-4-26b-a4b-it-int4-ov`) is the fastest single-stream arm: no other gate-passing arm (llama.cpp with Google's QAT Q4_0 GGUF, with or without the QAT MTP drafter) decodes at N=1 5% or more faster. A cross-host replication of H57da on a second AMX-less client. | Written 2026-10-20, before any H57d or H57w measurement. N=1 median decode tok/s: Od in [7, 18] (0.5); Ld in [10, 26] (0.5); Ld/Od in [1.1, 1.8] (0.5); every arm's GSM8K 0-49 score >= 0.80 (0.6). REFUTED (0.55), SUPPORTED (0.25), INCONCLUSIVE (0.2). Not gated: TTFT on a 1,000-token prompt, aggregate tok/s at N=4, package J/token. | H57da's protocol, runner, statistic and outcome rules (the H57d pre-registration and its runner amendment), unchanged except as listed in the wildct pre-registration: H57d's runner (amendment R1) with the host knobs listed there: clusters `k3dgpu-w-h57d-`, the GPU opt-out in place of `K3D_GPU_VENDORS=hybrid`, W2-W6, hold-start `load1` <= 12, models downloaded on wildct at H57d's revisions, at most 5 holds of at most 75 min. Od's render names the hold's cluster: [H57d amendment R2](#h57d-amendment-r2-ods-render-names-the-holds-cluster-2026-10-29) (2026-10-29, before any H57w hold). | PENDING | @@ -9060,6 +9060,140 @@ draft). Test: `tests/wildct_hold_test.sh` (row.sh's H25w environment), in CI. `scripts/k3d-gpu` must still equal 0.1.89's apart from the version line, or `hold.sh` refuses the hold (exit 7), and its `share/` must equal 0.1.89's. Rule W7 keeps release 0.1.89. +### H25w run (wildct, i7-13700, 2026-11-07) + +**Reproduce:** + +```text +# provenance +date: 2026-11-07T05:18:20Z to 05:36:58Z (one hold; lock free, no load wait: load1 0.99); + rounds' pre-windows began 05:19:51Z, 05:21:43Z, 05:26:04Z, 05:28:00Z, 05:32:24Z and 05:34:18Z +hypothesis: H25w (H25's T part, A3 first-chunk TTFT, A4 amendment-27 load rule, as the wildct + pre-registration's H25w line lists them; wildct amendment R7) +host: wildct, reached by IP 100.66.38.121: Intel Core i7-13700 (family 6 model 183), 8 P-cores + (cpu_core 0-15) and 8 E-cores (cpu_atom 16-23), 1 NUMA node, 125 GiB RAM, PL1 35 W, Manjaro; + Docker 29.7.2, live-restore off, cgroup systemd/2 (step 1, 03:32:22Z) +host kernel: 6.6.156-2-MANJARO +accelerators: hybrid=1; UHD 770 (i915), RTX 4000 SFF Ada and RTX 2000 Ada (nvidia), opted out and not + used (rule W8) +k3d-gpu: release 0.1.89 (aur-v0.1.89, .deb sha256 8f94d69ccc999e4f67295f6cffea8de6cb5307e2ef71210d3ba090458ba88acc), + rule W7's release since the first hold (H52w); 0.1.90 was cut at 05:25:00Z, during the hold, + from packaging changes (its launcher equals 0.1.89's apart from VERSION) +bench scripts: main at ce4ff16c (amendment R7's merge), shipped as the archive repo-ce4ff16c47ee; run.sh + runs its scripts/k3d-gpu (equal to 0.1.89's apart from VERSION) and share/ (equal to 0.1.89's) +node image: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4, + k3s v1.37.1+k3s1; k3d v5.9.0, helm v4.3.1, kubectl v1.37.2 +runtime: openvino/model_server@sha256:a928903f…6fc58 (2026.4.1), pulled by the pod (k3d image import + found no local copy of the tag@digest ref, as in H25 run 4); `agent manifest` CPU as shipped + (P-cores, 16 CPUs, memory 6Gi) with the CPU Fast Draft OpenVINO/Qwen3-0.6B-int4-ov + (K3D_GPU_AGENT_DRAFT=CPU, amendment R7) in both arms; pod Cpus_allowed_list 0-15 in all rounds +models: OpenVINO/Qwen3-4B-int4-ov b467368d (openvino_model.bin 074214fa…df2f9); draft f864c610 + (28613e99…) staged at /OpenVINO-Qwen3-0.6B-int4-ov; hostPath from a copy in the node +load: bench/load/load.py `typical`, stress-ng 0.22.01 @sha256:2ea9a7c3…1cbe on E-cores 20-23 + (LOAD_CPUS, rule W2; 3 containers, 3 of 3 running in every window) +energy source: intel-rapl:0 energy_uj (package-0) through k3dgpu-w-rapl (rule W4), per round and per window +environment: IMAGE=, MODELS=~/k3dgpu-bench-wildct/models, LEVELS=typical, + CLUSTER=k3dgpu-w-h25, SEED_BASE=2560, HOLD=1440, LOAD_CPUS=20-23, + K3D_GPU_VENDORS=auto,-nvidia,-intel, K3D_GPU_K3S_ARGS=, K3D_GPU_AGENT_DRAFT=CPU, + K3D_GPU_RAPL_EXEC="docker exec k3dgpu-w-rapl" +runs per arm: order ON OFF ON OFF ON OFF, seeds 2561-2566; ON 3 rounds (n = 15 added agents), OFF 2 rounds + (n = 10): round 6 (OFF) was stopped by run.sh's 18:30 level cap and wrote no round file +host load: load1 1.15-11.55, foreign CPU 0.63-8.14 CPU-s/s (p90 1.62), MemAvailable 109.6-114.4 GiB, + package 61-72 C, / 213.1-222.0 GB available (sample.py, 223 samples; df -BG read 207G + before and after) +``` + +Command, on wildct: RESUME.md step 6's `launch H25w 1500 8 h25w`, which runs `bench/wildct/hold.sh H25w +1500 8 bash bench/wildct/row.sh h25w`, and so `IMAGE= MODELS= OUT= +LEVELS=typical CLUSTER=k3dgpu-w-h25 SEED_BASE=2560 HOLD=1440 LOAD_CPUS=20-23 $E K3D_GPU_AGENT_DRAFT=CPU +bench/shared-prefix/run.sh`, the registered line with amendment R7's variable. Records (round files, +load windows, the score, both manifests, the `up`, image import and cache-usage logs, `run.sh`'s output +in `hold.log`, the hold's samples; the kubeconfig was not copied): +[bench/wildct/results/2026-11-07/runs/H25w/](bench/wildct/results/2026-11-07/runs/H25w/). +The host before and after the hold: [bench/wildct/results/2026-11-07/host/h25w.txt](bench/wildct/results/2026-11-07/host/h25w.txt). +The score, from the copied records (identical to the `score.json` written on wildct): + +```bash +python3 bench/shared-prefix/probe.py score \ + --load bench/wildct/results/2026-11-07/runs/H25w/out/typical/load.jsonl \ + bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-*.json +``` + +`up` named nvidia and intel as opted out, printed `Accelerators: hybrid` and ended with +`Accelerators schedulable: hybrid=1`. The agent manifest read "agent: +OpenVINO/Qwen3-4B-int4-ov as 'Qwen3-4B-int4-ov' on CPU, P-cores, 16 CPU(s)" and "agent: Fast Draft +OpenVINO/Qwen3-0.6B-int4-ov on CPU"; `off.yaml` differs from `on.yaml` by the one +`--enable_prefix_caching=false` line. All 35 scored requests returned with no error, `prompt_tokens` +1634 and `completion_tokens` 8, TTFT present in every one; graph.pbtxt read back as intended in all 5 +rounds. No round is void. + +**Load, amendment 27: MATCHED.** Covariates (gating, `TOLERANCE`): + +| Phase | Metric | ON | OFF | spread / limit | +|-------|--------|----|-----|----------------| +| pre (n = 3 per arm) | E-core busy % (CPUs 20-23) | 50.05 | 49.85 | 0.20 / 5.0 ok | +| pre | throttle/min | 0 | 0 | 0 / 10 ok | +| pre | swap used / growth MiB | 0 / 0 | 0 / 0 | ok | +| during (ON 54, OFF 119 windows) | E-core busy % (load CPUs minus pod CPUs 0-15 = 20-23) | 54.24 | 54.92 | 0.68 / 5.0 ok | +| during | swap used / growth MiB | 0 / 0 | 0 / 0 | ok | + +Outcomes during the rounds (reported, never gating): package W 34.87 ON vs 34.87 OFF; throttle/min +0 vs 0; 1-min load 8.43 vs 10.12. OFF's windows include round 6's pre-window and its 28 during windows, +since the level's load record holds every round that started; without them the rule also reads MATCHED +(pre busy spread 0.07, during 0.70; not gated). + +**Pre-registered verdict: T SUPPORTED at `typical`** (`probe.py score`). M is not measured (A3). + +| | ON (shipping default) | OFF (`--enable_prefix_caching=false`) | +|---|---|---| +| added agents a2..a6, median TTFT | 2.198 s (n = 15) | 31.437 s (n = 10) | +| per-round medians | 2.154, 2.204, 2.281 s (spread 0.127 s, 6%) | 31.408, 32.584 s (spread 1.176 s, 4%) | +| cold agent a1, per round | 32.43, 30.87, 30.88 s | 32.47, 31.41 s | +| control c (one token changed near position 16), per round | 32.25, 31.50, 30.09 s | 30.62, 30.61 s | +| package energy per round | 3439, 3406, 3354 J | 8439, 8500 J | + +- Ratio OFF/ON of the added-agent median TTFT: **14.30x**, bootstrap 95% CI 13.38-14.89x (10000 + resamples, seed 25). The lower bound clears the pre-registered 4x. +- Controls: c/OFF median 1.00 (>= 0.8x: one changed token near the start makes a cold request); a1 + ON/OFF 0.97 (inside ±15%). Per-arm spread gate passes. No round is void. +- Cross-round pairs >= 4x: 6 of 6. Derived confidence by the A2 rule: (6+1)/(6+2) = 0.88. +- Round 6 (OFF, seed 2566) started at 05:34:18Z and `run.sh`'s level cap (no new round after 18:30, + the probe under `timeout` of what is left) stopped its probe 1111 s into the run, before it wrote its + round file. + The H25 pre-registration ends a level with a hard abort at 18:30, and rounds not run count as missing + under the arm rule, which turns an arm INCONCLUSIVE only when more than 1 of its 3 rounds is lost; OFF + lost 1. Each of an OFF round's seven scored requests is a cold prefill: about 31 s here (240 s per + round with the warm-up), against 10.9 s on the i9 in H25 run 4. So six rounds did not fit the + level's 18:30. + +Against the prediction (written 2026-10-20, SUPPORTED 0.8): the verdict and both gates held (CI lower +bound 13.38x >= 4x, c/OFF 1.00 >= 0.8). The point ranges did not: OFF 31.44 s against 10 s (7-15), ON +2.20 s against 1.0 s (0.6-1.6), and the ratio 14.30x against 10x (6-14), just above that range. OFF +ran 2.9x and ON 2.2x slower than H25 run 4 on the i9-13900T (OFF 10.87 s, ON 1.02 s). The package read +33.6-36.3 W in every load window of both arms, pre-windows with the pod idle included; wildct's PL1 is +35 W, and the i9 runs PL1 = PL2 = 117 W. The run does not test whether the power limit is the cause. +Per added agent, the shared prefix saved 29.2 s of first-chunk time here, against 9.85 s on the i9. + +Not scored: + +- The ON rounds used 0.40x the OFF rounds' package energy (median 3406 J vs 8469 J for a round's 8 + requests; whole package, the production containers' draw included). +- CPUs 0-15, the pod's 16-CPU balloon, read about 8 CPU-s/s busy through the rounds (sample.py + `busy_p`, median 8.01, max 8.16); the records do not say which threads ran. +- Foreign CPU (everything but this campaign's containers) was above 3 CPU-s/s in four samples: three + during cluster setup (05:18:25-05:18:50Z) and one at cleanup (05:36:52Z); none during a round. + +Production box. The snapshot before the hold (05:18Z) and after it (05:37Z) read the same: 18 +containers Up, each started 2026-10-29T09:11:27Z with restart count 0, and the same port codes. The +line counts of `iptables-save`, `ip6tables-save` and `nft list ruleset` read 180, 53 and 286 before and +after, and their rules were identical with the counters stripped. After the hold, no `k3dgpu-w-*` +cluster or container, no process of the hold and no network or volume of ours was left, and +`live.lock` was free. Before the hold (05:18Z) all 18 modules of +`/etc/modules-load.d/k3dgpu-wildct.conf` were loaded and `kernel.modprobe` was empty; after it +(05:37Z) the host had 224 modules loaded and `kernel.modprobe` was still empty. + +Holds used: 3 of at most 12. The next row in the registered order is H34w. + ### H5 amendment A1: stage-1 harness and definitions (2026-10-19) Written 2026-10-19T22:30-07:00, before any H5 measurement. It fixes what the diff --git a/bench/wildct/RESUME.md b/bench/wildct/RESUME.md index a453a0bc..ba9015c1 100644 --- a/bench/wildct/RESUME.md +++ b/bench/wildct/RESUME.md @@ -1,14 +1,22 @@ # wildct resume pack How to resume the wildct campaign ([EXPERIMENTS.md](../../EXPERIMENTS.md), "Wildct -pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5 and R6, and the "H52w -run" and "H18w run" sections). This file changes no rule. Run it top to bottom, except that step -3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates +pre-registration", rules W1-W8, wildct amendments R1, P1, R2, R3, R4, R5, R6 and R7, and the +"H52w run", "H18w run" and "H25w run" sections). This file changes no rule. Run it top to bottom, +except that step 3b (ship `origin/main` to wildct) runs first when the `repo` on wildct predates `bench/wildct/nfprobe.sh` (H10 runs it). Nothing past step 1 runs until every step 1 check passes. -## Where things stand (2026-11-07, after H18w) +## Where things stand (2026-11-07, after H25w) -- **Next: row 3, `H25w` (step 6).** Holds used: 2 of at most 12. +- **Next: row 4, `H34w` (step 6).** Holds used: 3 of at most 12. +- **Row 3, `H25w`,** ran on 2026-11-07 (05:18:20Z to 05:36:58Z, rc 0; `out/typical/score.json` + written) and is scored SUPPORTED (EXPERIMENTS.md "H25w run"; records in + `results/2026-11-07/runs/H25w/`, the host before and after in `results/2026-11-07/host/h25w.txt`). + It ran from main at `ce4ff16c`, which carries amendment R7 (`row.sh h25w` sets + `K3D_GPU_AGENT_DRAFT=CPU`), shipped by step 3b as `repo-ce4ff16c47ee`; its launcher and `share/` + equal 0.1.89's. Round 6 (OFF) was cut by `run.sh`'s 18:30 level cap, so OFF has 2 rounds. The + host's containers, start times, restart counts, port codes and firewall rules read the same + before and after it, and nothing of the hold was left. - **Row 2, `H18w`,** ran on 2026-11-07 (04:22:23Z to 04:26:30Z, rc 0; `out/run-1..3.txt` each has `sandbox_up_rc=0` and `pip_rc=0`) and is scored SUPPORTED (EXPERIMENTS.md "H18w run"; records in `results/2026-11-07/runs/H18w/`, the host before and after in `results/2026-11-07/host/h18w.txt`). @@ -22,6 +30,8 @@ run" and "H18w run" sections). This file changes no rule. Run it top to bottom, before and after it, and nothing of the hold was left. - **Release frozen at 0.1.89 (rule W7).** `H52w`, the campaign's first hold, ran 0.1.89, so every later hold runs it too. Step 3 does not run again, and neither does step 5. + Release 0.1.90 was cut at 2026-11-07T05:25:00Z, during `H25w`, from packaging changes; its + launcher equals 0.1.89's apart from the version line, and its `share/` equals 0.1.89's. - **Kernel modules** (each approved by the box owner on 2026-10-29; `kernel.modprobe` stays empty): - `vxlan` and `br_netfilter` (amendment R3; `results/2026-10-29/host/modules.txt`); - `xt_comment xt_statistic xt_multiport xt_limit xt_NFLOG xt_physdev xt_pkttype ipt_REJECT @@ -71,12 +81,12 @@ run" and "H18w run" sections). This file changes no rule. Run it top to bottom, `8eb21d5b...`, 321,145,696 bytes, the i9's bytes) and Od's `graph.pbtxt` (`c57ecf35...`, the i9's); both hashed again on 2026-11-07; - `h57/bin/k3d` (the shim) and `h57/weights.sha256`; - - `repo` -> `repo-85a7bc8668d1` (and the older archives); + - `repo` -> `repo-ce4ff16c47ee` (and the older archives, `repo-85a7bc8668d1` among them); - `setup/`: `images-before.txt`, `containers-before.txt`, `volumes-before.txt`, `networks-before.txt` and the verify logs; - `runs/`: `smoke1*`, `smoke2*`, `diag-smoke2*`, `smoke3*` (with `smoke3-node.log`, the node's log, and `smoke3-node-excerpt.txt`), `smoke4*` (with `smoke4-live.txt`), `smoke5*`, - `H52w*`, `H18w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones + `H52w*`, `H18w*`, `H25w*`, `convert/`, `resume-health.txt` (2026-11-07T03:32:22Z; the earlier ones renamed `resume-health.txt.`), `install-0.1.79.log`, `install-0.1.81.log`, `install-0.1.89.log`, `setup-images-*.log`, `IMAGE` and `campaign.log`; - `diag/`: `diag-up.sh`, the command `diag-smoke2` ran; `iptprobe-in.sh`, an ad-hoc form of diff --git a/bench/wildct/results/2026-11-07/host/h25w.txt b/bench/wildct/results/2026-11-07/host/h25w.txt new file mode 100644 index 00000000..d65082bd --- /dev/null +++ b/bench/wildct/results/2026-11-07/host/h25w.txt @@ -0,0 +1,227 @@ +# wildct around the H25w hold, 2026-11-07 (UTC). Read-only reads as andrew, except the firewall +# dumps (iptables-save, ip6tables-save and nft list ruleset), as root. +# The production snapshot: running containers, published-port HTTP codes from the host, and each +# running container's StartedAt and RestartCount. + +== state before the hold (2026-11-07T05:18Z), after step 3b shipped main at ce4ff16c (amendment R7) +## date 2026-11-07T05:18:11+00:00 +## boot: 2026-10-29 05:11:09 +## kernel: 6.6.156-2-MANJARO +## disk +/dev/nvme0n1p2 3667G 3274G 207G 95% / +## mem +MemTotal: 131524088 kB +MemAvailable: 119958892 kB +## loadavg: 0.64 0.46 0.46 1/3707 49457 +VERSION="0.1.89" # kept in lockstep with packaging/aur PKGBUILD by auto-release.yml +## repo -> repo-ce4ff16c47ee COMMIT=ce4ff16c47ee447664abd6ddead869dd1bc80ce8 +## runs/IMAGE: cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +## tools +k3d version v5.9.0 +k3s version v1.35.5-k3s1 (default) +Client Version: v1.37.2 +v4.3.1+gc0d08d0 +## clusters (k3d cluster list; names only) +wtm (the box owner's; exited; not this campaign's). No k3dgpu-w-* cluster. +## our containers +ps rc=0 +## live.lock +-rw-r--r-- 1 andrew andrew 0 Nov 6 23:22 /home/andrew/k3dgpu-bench-wildct/live.lock +lock-free +hold procs rc=1 +H18w pid=46184 alive=no rc=0 +H52w pid=40856 alive=no rc=0 +convert pid=24108 alive=no rc=0 +diag-smoke2 pid=17199 alive=no rc=1 +smoke2 pid=32217 alive=no rc=1 +smoke3 pid=62226 alive=no rc=0 +smoke4 pid=48184 alive=no rc=0 +smoke5 pid=31149 alive=no rc=0 +## modules of /etc/modules-load.d/k3dgpu-wildct.conf +loaded: 18 of 18; missing: 0; 224 modules loaded in all +modprobe=[] +## newest k3d-gpu release at 05:18Z (gh release list): aur-v0.1.89 (2026-11-07T00:23:41Z), as installed; +## rule W7 froze it at the first hold (H52w) + +== the row's command, models and images, read before the launch (row.sh h25w with WILDCT_ROW_PRINT=1) +## row print +IMAGE=cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 +MODELS=/home/andrew/k3dgpu-bench-wildct/models +OUT=/home/andrew/k3dgpu-bench-wildct/runs/H25w/out +LEVELS=typical +CLUSTER=k3dgpu-w-h25 +SEED_BASE=2560 +HOLD=1440 +LOAD_CPUS=20-23 +K3D_GPU_VENDORS=auto,-nvidia,-intel +K3D_GPU_K3S_ARGS=--kubelet-arg=eviction-hard=memory.available<100Mi,nodefs.available<2Gi,imagefs.available<2Gi,nodefs.inodesFree<1% +K3D_GPU_AGENT_DRAFT=CPU +bash /home/andrew/k3dgpu-bench-wildct/repo/bench/shared-prefix/run.sh +## row check +check rc=0 +## launcher vs release +LAUNCHER-MATCH +share diff done +## node image +cryptoandcoffee/k3d-gpu@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4 sha256:5811390ea98da96ebe01a3338291d4f462bb8c16e0063c99939944190547b8c8 +## cpu model +model name : 13th Gen Intel(R) Core(TM) i7-13700 +## staged models (sha256 of openvino_model.bin: target, and the draft staged under it) +074214fa29ab1b479536fc7184ef3036e7b355a5835e8590b61b70a8de1df2f9 OpenVINO/Qwen3-4B-int4-ov/openvino_model.bin +28613e994a6e1aaf54a6fe8677eda35e8e86684e653dde09f3ba960b8e53e152 OpenVINO/Qwen3-4B-int4-ov/OpenVINO-Qwen3-0.6B-int4-ov/openvino_model.bin +## verify logs +/home/andrew/k3dgpu-bench-wildct/setup/dl.log:VERIFY ok +/home/andrew/k3dgpu-bench-wildct/setup/dl.log:VERIFY 1 bad +/home/andrew/k3dgpu-bench-wildct/setup/verify-h57.log:VERIFY ok +newer-than-verify done +## autoquant cache +XDG_STATE_HOME=[] +ls: cannot access '/home/andrew/.local/state/k3d-gpu': No such file or directory +## OVMS image +AGENT_OVMS_IMAGE="openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58" +sha256:6348136997e05a7ee2c2b08e47bb15deb6e5096c62ab3fb821907ded7e2afd7c openvino/model_server@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 +## stress-ng image (load.py's pin): present +ghcr.io/colinianking/stress-ng@sha256:2ea9a7c3828eda1b1ca3d0b0c5a669ec252f9caf97a6a33aedcea6b2a66d1cbe (image sha256:eb124d52...) +## cpu_atom, cpu_core +16-23 +0-15 + +== production snapshot before the hold (2026-11-07T05:18Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== production snapshot after the hold (2026-11-07T05:37Z) +bmd-telemetry-jaeger-1|Up +bmd-telemetry-prometheus-1|Up +feed-my-algorithm-app-1|Up +feed-my-algorithm-controller-1|Up(healthy) +feed-my-algorithm-db-1|Up(healthy) +feed-my-algorithm-redroid-1|Up +feed-my-algorithm-worker-1|Up +know-my-neighbor-db-1|Up(healthy) +proxmox-dc-manager|Up(healthy) +searxng-core|Up +searxng-valkey|Up(healthy) +state-of-my-town-app|Up +state-of-my-town-archivebox|Up(healthy) +state-of-my-town-postgres|Up(healthy) +tos-rate-calculator|Up(healthy) +webapp-adminer-1|Up +webapp-db-1|Up(healthy) +webapp-web-1|Up +port 3000 200 +port 3080 200 +port 3100 200 +port 4317 000 +port 5433 000 +port 5434 000 +port 5544 000 +port 5565 000 +port 8010 404 +port 8443 301 +port 8888 404 +port 8890 200 +port 9090 302 +port 13133 404 +port 16686 200 +/bmd-telemetry-jaeger-1 2026-10-29T09:11:27.91961244Z 0 +/bmd-telemetry-prometheus-1 2026-10-29T09:11:27.949546379Z 0 +/feed-my-algorithm-app-1 2026-10-29T09:11:27.936156879Z 0 +/feed-my-algorithm-controller-1 2026-10-29T09:11:27.939661363Z 0 +/feed-my-algorithm-db-1 2026-10-29T09:11:27.932636735Z 0 +/feed-my-algorithm-redroid-1 2026-10-29T09:11:27.907167381Z 0 +/feed-my-algorithm-worker-1 2026-10-29T09:11:27.938381448Z 0 +/know-my-neighbor-db-1 2026-10-29T09:11:27.946670984Z 0 +/proxmox-dc-manager 2026-10-29T09:11:27.95058685Z 0 +/searxng-core 2026-10-29T09:11:27.943106017Z 0 +/searxng-valkey 2026-10-29T09:11:27.920115646Z 0 +/state-of-my-town-app 2026-10-29T09:11:27.933239949Z 0 +/state-of-my-town-archivebox 2026-10-29T09:11:27.915974301Z 0 +/state-of-my-town-postgres 2026-10-29T09:11:27.948938903Z 0 +/tos-rate-calculator 2026-10-29T09:11:27.934425305Z 0 +/webapp-adminer-1 2026-10-29T09:11:27.945114439Z 0 +/webapp-db-1 2026-10-29T09:11:27.950054666Z 0 +/webapp-web-1 2026-10-29T09:11:27.947765425Z 0 +firewall line counts: iptables-save 180, ip6tables-save 53, nft list ruleset 286 + +== comparison +production snapshot: identical +iptables-save rules, packet counters stripped: identical +ip6tables-save rules, packet counters stripped: identical +nft list ruleset, counters stripped: identical + +== after-hold checks (RESUME.md step 6) +== after H25w 2026-11-07T05:37:17Z +0 +== H25w lock taken 2026-11-07T05:18:20Z +== H25w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h25w 2026-11-07T05:18:20Z +== H25w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T05:36:58Z +list rc=0 +ps rc=0 +strays rc=0 +/dev/nvme0n1p2 3667G 3274G 207G 95% / +lock-free +net rc=1 (grep found no k3dgpu network) +vol rc=1 (grep found no k3dgpu volume) + +== host state after the hold +## date 2026-11-07T05:37:41+00:00 +## modules: 224 loaded +modprobe=[] +clusters rc=0 +ours grep rc=1 (1 = none) +lock-free +hold procs rc=1 (1 = none) +/dev/nvme0n1p2 3667G 3274G 207G 95% / +## thermal_throttle files: 24 +## sshd: active diff --git a/bench/wildct/results/2026-11-07/runs/H25w.out b/bench/wildct/results/2026-11-07/runs/H25w.out new file mode 100644 index 00000000..97ec7252 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w.out @@ -0,0 +1,4 @@ +== H25w waiting for the lock 2026-11-07T05:18:20Z +== H25w lock taken 2026-11-07T05:18:20Z +== H25w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h25w 2026-11-07T05:18:20Z +== H25w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T05:36:58Z diff --git a/bench/wildct/results/2026-11-07/runs/H25w/hold.log b/bench/wildct/results/2026-11-07/runs/H25w/hold.log new file mode 100644 index 00000000..cbd464af --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/hold.log @@ -0,0 +1,177 @@ +load typical: started typical on 20-23 (3 containers, ttl 1500s) +image import failed; the pod pulls it (see /home/andrew/k3dgpu-bench-wildct/runs/H25w/out/import.log) +settle: settled after 5 s +level=typical round=1 arm=ON seed=2561 graph_prefix_caching=true pod_cpus=0-15 load1=2.45 swap_used_mib=0 containers=23 +{"arm": "ON", "seed": 2561, "added_ttft": [2.204154353006743, 2.119715873966925, 2.190626053023152, 2.1099928149487823, 2.153619972988963], "a1": 32.43029022193514, "c": 32.253816806012765, "drss_per_agent_mib": null} +settle: settled after 5 s +level=typical round=2 arm=OFF seed=2562 graph_prefix_caching=false pod_cpus=0-15 load1=7.82 swap_used_mib=0 containers=23 +{"arm": "OFF", "seed": 2562, "added_ttft": [29.978655762970448, 32.30350062099751, 30.520233896910213, 31.40780486294534, 31.465521304053254], "a1": 32.473897791001946, "c": 30.623118146904744, "drss_per_agent_mib": null} +settle: settled after 5 s +level=typical round=3 arm=ON seed=2563 graph_prefix_caching=true pod_cpus=0-15 load1=9.00 swap_used_mib=0 containers=23 +{"arm": "ON", "seed": 2563, "added_ttft": [2.1505636520450935, 2.204036680981517, 2.0910487030632794, 2.3701426689513028, 2.435048700077459], "a1": 30.87193522101734, "c": 31.50250490801409, "drss_per_agent_mib": null} +settle: settled after 5 s +level=typical round=4 arm=OFF seed=2564 graph_prefix_caching=false pod_cpus=0-15 load1=7.78 swap_used_mib=0 containers=23 +{"arm": "OFF", "seed": 2564, "added_ttft": [32.58352233201731, 30.30790936492849, 32.72945735999383, 30.220451082102954, 32.726326987030916], "a1": 31.4112520609051, "c": 30.608316531055607, "drss_per_agent_mib": null} +settle: settled after 5 s +level=typical round=5 arm=ON seed=2565 graph_prefix_caching=true pod_cpus=0-15 load1=8.97 swap_used_mib=0 containers=23 +{"arm": "ON", "seed": 2565, "added_ttft": [2.5247044879943132, 2.4060273899231106, 2.2809814469655976, 2.1814400840085, 2.1977264269953594], "a1": 30.879644831060432, "c": 30.08638923894614, "drss_per_agent_mib": null} +settle: settled after 5 s +level=typical round=6 arm=OFF seed=2566 graph_prefix_caching=false pod_cpus=0-15 load1=8.00 swap_used_mib=0 containers=23 +{ + "n_rounds": { + "ON": 3, + "OFF": 2 + }, + "void": {}, + "median_added_ttft_s": { + "ON": 2.1977264269953594, + "OFF": 31.436663083499297 + }, + "n_added": { + "ON": 15, + "OFF": 10 + }, + "per_round_median_s": { + "ON": [ + 2.153619972988963, + 2.204036680981517, + 2.2809814469655976 + ], + "OFF": [ + 31.40780486294534, + 32.58352233201731 + ] + }, + "spread_s": { + "ON": 0.12736147397663444, + "OFF": 1.1757174690719694 + }, + "ratio": 14.30417484967781, + "ci95": [ + 13.38030782210502, + 14.891222488384075 + ], + "confidence_pairs_ge_4x": 1.0, + "c_on_over_off": 1.0020944279085826, + "a1_on_over_off": 0.9667237191316892, + "prompt_tokens": [ + "1634" + ], + "inconclusive_parts": [], + "T": "SUPPORTED", + "M": "INCONCLUSIVE (not measured: phase B not run, amendment A3)", + "load": { + "matched": true, + "void": false, + "verdict": "MATCHED", + "rule": "amendment-27", + "checks": [ + { + "metric": "busy_pct", + "arms": { + "ON": 50.053, + "OFF": 49.85 + }, + "spread": 0.203, + "limit": 5.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "throttle_per_min", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 10.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "swap_used_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 256.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "swap_delta_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 64.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "busy_pct", + "arms": { + "ON": 54.238, + "OFF": 54.921 + }, + "spread": 0.683, + "limit": 5.0, + "status": "ok", + "phase": "during" + }, + { + "metric": "swap_used_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 256.0, + "status": "ok", + "phase": "during" + }, + { + "metric": "swap_delta_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 64.0, + "status": "ok", + "phase": "during" + } + ], + "outcomes": [ + { + "metric": "pkg_w", + "arms": { + "ON": 34.865, + "OFF": 34.874 + }, + "spread": 0.009 + }, + { + "metric": "throttle_per_min", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0 + }, + { + "metric": "load1", + "arms": { + "ON": 8.432, + "OFF": 10.118 + }, + "spread": 1.686 + } + ], + "unsettled": {} + } +} +elapsed=1111s +cluster k3dgpu-w-h25 deleted at 1114s diff --git a/bench/wildct/results/2026-11-07/runs/H25w/launcher.diff b/bench/wildct/results/2026-11-07/runs/H25w/launcher.diff new file mode 100644 index 00000000..e69de29b diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/import.log b/bench/wildct/results/2026-11-07/runs/H25w/out/import.log new file mode 100644 index 00000000..674e2805 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/import.log @@ -0,0 +1,4 @@ +INFO[0000] Importing image(s) into cluster 'k3dgpu-w-h25' +WARN[0000] Image 'openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58' is not a file and couldn't be found in the container runtime +ERRO[0000] Failed to import image(s) into cluster 'k3dgpu-w-h25': no valid images specified +WARN[0000] At least one error occured while trying to import the image(s) into the selected cluster(s) diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/manifest.err b/bench/wildct/results/2026-11-07/runs/H25w/out/manifest.err new file mode 100644 index 00000000..5548b922 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/manifest.err @@ -0,0 +1,2 @@ +==> agent: OpenVINO/Qwen3-4B-int4-ov as 'Qwen3-4B-int4-ov' on CPU, P-cores, 16 CPU(s) +==> agent: Fast Draft OpenVINO/Qwen3-0.6B-int4-ov on CPU diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/off.yaml b/bench/wildct/results/2026-11-07/runs/H25w/out/off.yaml new file mode 100644 index 00000000..5906c37b --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/off.yaml @@ -0,0 +1,97 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + k3d-gpu.io/cores: performance + annotations: + k3d-gpu.io/agent-device: CPU + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + initContainers: + # --pull downloads the model once and writes its graph.pbtxt (device, + # parsers). On later starts the files are already there and only the + # graph is rewritten, so a device change takes effect with no download. + - name: pull + image: openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3-4B-int4-ov + - --model_repository_path=/models + - --model_name=Qwen3-4B-int4-ov + - --task=text_generation + - --enable_prefix_caching=false + - --target_device=CPU + - --draft_source_model=OpenVINO/Qwen3-0.6B-int4-ov + - --draft_device=CPU + - --tool_parser=hermes3 + - --reasoning_parser=qwen3 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3-4B-int4-ov + - --model_name=Qwen3-4B-int4-ov + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + # No liveness probe: compiling a model for a GPU takes minutes. + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: "16" + memory: 6Gi + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + hostPath: + path: /var/lib/h25-models + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/on.yaml b/bench/wildct/results/2026-11-07/runs/H25w/out/on.yaml new file mode 100644 index 00000000..330188a1 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/on.yaml @@ -0,0 +1,96 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: k3d-gpu-agent +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ovms + namespace: k3d-gpu-agent + labels: + app: ovms +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: ovms + template: + metadata: + labels: + app: ovms + k3d-gpu.io/cores: performance + annotations: + k3d-gpu.io/agent-device: CPU + spec: + securityContext: + runAsUser: 5000 + runAsGroup: 5000 + fsGroup: 5000 + initContainers: + # --pull downloads the model once and writes its graph.pbtxt (device, + # parsers). On later starts the files are already there and only the + # graph is rewritten, so a device change takes effect with no download. + - name: pull + image: openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --pull + - --source_model=OpenVINO/Qwen3-4B-int4-ov + - --model_repository_path=/models + - --model_name=Qwen3-4B-int4-ov + - --task=text_generation + - --target_device=CPU + - --draft_source_model=OpenVINO/Qwen3-0.6B-int4-ov + - --draft_device=CPU + - --tool_parser=hermes3 + - --reasoning_parser=qwen3 + volumeMounts: + - name: models + mountPath: /models + containers: + - name: ovms + image: openvino/model_server:2026.4.1-gpu@sha256:a928903fe429f1e3e27107cc45932d30d95febfd2d1159dc50869c3eac76fc58 + imagePullPolicy: IfNotPresent + args: + - --rest_port=8000 + - --model_path=/models/OpenVINO/Qwen3-4B-int4-ov + - --model_name=Qwen3-4B-int4-ov + - --cache_dir=/models/.ov_cache + - --log_level=INFO + ports: + - name: http + containerPort: 8000 + # No liveness probe: compiling a model for a GPU takes minutes. + readinessProbe: + httpGet: + path: /v2/health/ready + port: http + periodSeconds: 5 + resources: + requests: + cpu: "16" + memory: 6Gi + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + hostPath: + path: /var/lib/h25-models + type: Directory +--- +apiVersion: v1 +kind: Service +metadata: + name: ovms + namespace: k3d-gpu-agent +spec: + selector: + app: ovms + ports: + - name: http + port: 8000 + targetPort: http diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/baseline.jsonl b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/baseline.jsonl new file mode 100644 index 00000000..4f4fa7d1 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/baseline.jsonl @@ -0,0 +1,2 @@ +{"ts": 1794028755.267798, "tag": "baseline", "level": "typical", "seconds": 10.038, "cpus": "20-23", "busy_pct": 49.7, "load1": 2.33, "pkg_w": 34.71, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028765.3551912, "tag": "baseline", "level": "typical", "seconds": 10.029, "cpus": "20-23", "busy_pct": 49.59, "load1": 2.36, "pkg_w": 34.8, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "baseline", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-1.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-1.log new file mode 100644 index 00000000..c0604e80 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-1.log @@ -0,0 +1,3 @@ +[2026-11-07 05:20:33.896][179][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:20:46.545][179][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:21:14.883][179][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 61.5% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-2.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-2.log new file mode 100644 index 00000000..d6574638 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-2.log @@ -0,0 +1,3 @@ +[2026-11-07 05:24:09.990][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 15.4% of 135.3 MB; +[2026-11-07 05:24:47.059][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 30.8% of 135.3 MB; +[2026-11-07 05:25:25.646][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 46.2% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-3.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-3.log new file mode 100644 index 00000000..9a191b63 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-3.log @@ -0,0 +1,3 @@ +[2026-11-07 05:26:46.061][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:26:58.735][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:27:27.202][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 61.5% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-4.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-4.log new file mode 100644 index 00000000..51a759b1 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-4.log @@ -0,0 +1,3 @@ +[2026-11-07 05:30:33.167][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 30.8% of 135.3 MB; +[2026-11-07 05:31:11.581][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 46.2% of 135.3 MB; +[2026-11-07 05:31:49.336][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 61.5% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-5.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-5.log new file mode 100644 index 00000000..fbdee478 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-5.log @@ -0,0 +1,3 @@ +[2026-11-07 05:33:05.553][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:33:19.200][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:33:45.800][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 61.5% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-6.log b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-6.log new file mode 100644 index 00000000..71e04fb5 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/cache-usage-6.log @@ -0,0 +1,3 @@ +[2026-11-07 05:35:33.895][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:36:06.685][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 100.0% of 135.3 MB; +[2026-11-07 05:36:43.914][183][llm_executor][info][llm_executor.hpp:104] All requests: 0; Scheduled requests: 1; Cache type: dynamic, cache usage: 15.4% of 135.3 MB; diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/load.jsonl b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/load.jsonl new file mode 100644 index 00000000..40ab5de1 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/load.jsonl @@ -0,0 +1,179 @@ +{"ts": 1794028791.3480086, "tag": "ON", "level": "typical", "seconds": 5.031, "cpus": "20-23", "busy_pct": 50.34, "load1": 2.45, "pkg_w": 34.73, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794028798.6312087, "tag": "ON", "level": "typical", "seconds": 5.07, "cpus": "20-23", "busy_pct": 53.81, "load1": 2.74, "pkg_w": 35.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028803.850955, "tag": "ON", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 53.26, "load1": 3.0, "pkg_w": 35.05, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028809.1356454, "tag": "ON", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 55.05, "load1": 5.13, "pkg_w": 34.63, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028814.4142466, "tag": "ON", "level": "typical", "seconds": 5.094, "cpus": "20-23", "busy_pct": 55.89, "load1": 6.24, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028819.6613698, "tag": "ON", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 55.45, "load1": 6.87, "pkg_w": 34.74, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028824.9261122, "tag": "ON", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 54.59, "load1": 7.2, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028830.1569216, "tag": "ON", "level": "typical", "seconds": 5.076, "cpus": "20-23", "busy_pct": 54.07, "load1": 7.66, "pkg_w": 34.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028835.3508394, "tag": "ON", "level": "typical", "seconds": 5.052, "cpus": "20-23", "busy_pct": 52.93, "load1": 7.45, "pkg_w": 34.36, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028840.528806, "tag": "ON", "level": "typical", "seconds": 5.066, "cpus": "20-23", "busy_pct": 52.27, "load1": 6.93, "pkg_w": 35.39, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028845.720832, "tag": "ON", "level": "typical", "seconds": 5.064, "cpus": "20-23", "busy_pct": 52.29, "load1": 7.02, "pkg_w": 34.96, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028850.8787403, "tag": "ON", "level": "typical", "seconds": 5.05, "cpus": "20-23", "busy_pct": 52.23, "load1": 7.58, "pkg_w": 34.53, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028856.0360377, "tag": "ON", "level": "typical", "seconds": 5.071, "cpus": "20-23", "busy_pct": 51.71, "load1": 7.45, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028861.255587, "tag": "ON", "level": "typical", "seconds": 5.089, "cpus": "20-23", "busy_pct": 55.52, "load1": 7.58, "pkg_w": 35.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028866.53661, "tag": "ON", "level": "typical", "seconds": 5.103, "cpus": "20-23", "busy_pct": 55.89, "load1": 7.69, "pkg_w": 34.72, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028871.7877822, "tag": "ON", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 54.27, "load1": 7.95, "pkg_w": 34.75, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028877.0546658, "tag": "ON", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 56.36, "load1": 8.2, "pkg_w": 34.87, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028882.3088727, "tag": "ON", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 53.19, "load1": 8.58, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028887.5486739, "tag": "ON", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 54.89, "load1": 8.78, "pkg_w": 34.96, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028903.639574, "tag": "OFF", "level": "typical", "seconds": 5.029, "cpus": "20-23", "busy_pct": 49.8, "load1": 7.82, "pkg_w": 34.72, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794028910.9159746, "tag": "OFF", "level": "typical", "seconds": 5.059, "cpus": "20-23", "busy_pct": 54.87, "load1": 7.81, "pkg_w": 36.06, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028916.1262696, "tag": "OFF", "level": "typical", "seconds": 5.085, "cpus": "20-23", "busy_pct": 52.39, "load1": 7.51, "pkg_w": 35.11, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028921.3745306, "tag": "OFF", "level": "typical", "seconds": 5.089, "cpus": "20-23", "busy_pct": 55.13, "load1": 7.95, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028926.6658983, "tag": "OFF", "level": "typical", "seconds": 5.101, "cpus": "20-23", "busy_pct": 55.76, "load1": 8.35, "pkg_w": 34.71, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028931.9389665, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 56.27, "load1": 8.8, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028937.2051969, "tag": "OFF", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 56.15, "load1": 9.3, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028942.444667, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 55.56, "load1": 9.44, "pkg_w": 34.9, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028947.6370556, "tag": "OFF", "level": "typical", "seconds": 5.051, "cpus": "20-23", "busy_pct": 53.48, "load1": 10.04, "pkg_w": 33.91, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028952.8397706, "tag": "OFF", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 54.19, "load1": 10.04, "pkg_w": 35.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028958.0707412, "tag": "OFF", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 53.97, "load1": 10.04, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028963.3095155, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 53.96, "load1": 10.19, "pkg_w": 34.91, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028968.555378, "tag": "OFF", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 57.26, "load1": 10.42, "pkg_w": 34.98, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028973.8026438, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 55.63, "load1": 10.38, "pkg_w": 34.84, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028978.9779394, "tag": "OFF", "level": "typical", "seconds": 5.036, "cpus": "20-23", "busy_pct": 54.69, "load1": 10.51, "pkg_w": 33.7, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028984.164862, "tag": "OFF", "level": "typical", "seconds": 5.094, "cpus": "20-23", "busy_pct": 55.46, "load1": 10.16, "pkg_w": 36.33, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028989.4282372, "tag": "OFF", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 56.96, "load1": 10.23, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028994.670558, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 54.7, "load1": 10.21, "pkg_w": 34.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794028999.9201198, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 59.5, "load1": 10.19, "pkg_w": 34.74, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029005.158455, "tag": "OFF", "level": "typical", "seconds": 5.085, "cpus": "20-23", "busy_pct": 53.47, "load1": 10.34, "pkg_w": 34.94, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029010.3867495, "tag": "OFF", "level": "typical", "seconds": 5.073, "cpus": "20-23", "busy_pct": 54.96, "load1": 10.47, "pkg_w": 34.87, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029015.6317685, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 54.89, "load1": 10.43, "pkg_w": 35.03, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029020.86631, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 55.63, "load1": 10.64, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029026.103627, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 55.18, "load1": 10.43, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029031.3399684, "tag": "OFF", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 57.59, "load1": 10.31, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029036.5674462, "tag": "OFF", "level": "typical", "seconds": 5.076, "cpus": "20-23", "busy_pct": 54.16, "load1": 10.37, "pkg_w": 34.93, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029041.8431513, "tag": "OFF", "level": "typical", "seconds": 5.11, "cpus": "20-23", "busy_pct": 55.23, "load1": 10.18, "pkg_w": 34.97, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029047.077729, "tag": "OFF", "level": "typical", "seconds": 5.069, "cpus": "20-23", "busy_pct": 54.48, "load1": 10.32, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029052.3344288, "tag": "OFF", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 56.76, "load1": 10.46, "pkg_w": 34.95, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029057.6448064, "tag": "OFF", "level": "typical", "seconds": 5.126, "cpus": "20-23", "busy_pct": 53.72, "load1": 10.5, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029062.87808, "tag": "OFF", "level": "typical", "seconds": 5.076, "cpus": "20-23", "busy_pct": 56.55, "load1": 10.7, "pkg_w": 34.9, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029068.1190658, "tag": "OFF", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 55.01, "load1": 10.97, "pkg_w": 34.91, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029073.354496, "tag": "OFF", "level": "typical", "seconds": 5.085, "cpus": "20-23", "busy_pct": 54.72, "load1": 10.89, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029078.5540812, "tag": "OFF", "level": "typical", "seconds": 5.057, "cpus": "20-23", "busy_pct": 54.67, "load1": 10.98, "pkg_w": 34.68, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029083.7557364, "tag": "OFF", "level": "typical", "seconds": 5.085, "cpus": "20-23", "busy_pct": 53.67, "load1": 10.26, "pkg_w": 35.41, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029089.008236, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 55.06, "load1": 10.32, "pkg_w": 34.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029094.2483068, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 54.04, "load1": 10.64, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029099.4866583, "tag": "OFF", "level": "typical", "seconds": 5.083, "cpus": "20-23", "busy_pct": 56.14, "load1": 10.75, "pkg_w": 34.95, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029104.745427, "tag": "OFF", "level": "typical", "seconds": 5.094, "cpus": "20-23", "busy_pct": 55.07, "load1": 10.93, "pkg_w": 34.91, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029110.0204935, "tag": "OFF", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 56.99, "load1": 11.25, "pkg_w": 34.9, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029115.2877078, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 54.32, "load1": 11.55, "pkg_w": 35.05, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029120.5608816, "tag": "OFF", "level": "typical", "seconds": 5.098, "cpus": "20-23", "busy_pct": 57.29, "load1": 11.19, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029125.8039117, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 54.07, "load1": 11.09, "pkg_w": 34.87, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029131.0364246, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 55.08, "load1": 11.09, "pkg_w": 34.91, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029136.3013039, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 53.96, "load1": 11.08, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029141.53706, "tag": "OFF", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 54.99, "load1": 10.99, "pkg_w": 34.9, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029164.321386, "tag": "ON", "level": "typical", "seconds": 5.032, "cpus": "20-23", "busy_pct": 50.07, "load1": 9.0, "pkg_w": 34.99, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794029171.6440773, "tag": "ON", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 53.94, "load1": 9.16, "pkg_w": 35.8, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029176.9103758, "tag": "ON", "level": "typical", "seconds": 5.105, "cpus": "20-23", "busy_pct": 54.68, "load1": 8.58, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029182.156391, "tag": "ON", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 56.31, "load1": 8.7, "pkg_w": 34.57, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029187.4075334, "tag": "ON", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 54.23, "load1": 8.72, "pkg_w": 34.66, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029192.6804712, "tag": "ON", "level": "typical", "seconds": 5.12, "cpus": "20-23", "busy_pct": 55.5, "load1": 8.66, "pkg_w": 34.75, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029197.92128, "tag": "ON", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 55.07, "load1": 8.61, "pkg_w": 34.7, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029203.1589706, "tag": "ON", "level": "typical", "seconds": 5.076, "cpus": "20-23", "busy_pct": 55.36, "load1": 8.72, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029208.359539, "tag": "ON", "level": "typical", "seconds": 5.058, "cpus": "20-23", "busy_pct": 53.96, "load1": 8.66, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029213.5575984, "tag": "ON", "level": "typical", "seconds": 5.066, "cpus": "20-23", "busy_pct": 55.41, "load1": 8.77, "pkg_w": 35.07, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029218.766599, "tag": "ON", "level": "typical", "seconds": 5.067, "cpus": "20-23", "busy_pct": 53.4, "load1": 8.95, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029223.8911908, "tag": "ON", "level": "typical", "seconds": 5.036, "cpus": "20-23", "busy_pct": 51.09, "load1": 8.47, "pkg_w": 34.13, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029229.0345333, "tag": "ON", "level": "typical", "seconds": 5.065, "cpus": "20-23", "busy_pct": 53.21, "load1": 8.03, "pkg_w": 35.36, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029234.2592883, "tag": "ON", "level": "typical", "seconds": 5.098, "cpus": "20-23", "busy_pct": 55.4, "load1": 8.64, "pkg_w": 35.11, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029239.528748, "tag": "ON", "level": "typical", "seconds": 5.083, "cpus": "20-23", "busy_pct": 54.87, "load1": 8.75, "pkg_w": 34.65, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029244.7579687, "tag": "ON", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 55.16, "load1": 9.01, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029250.0108597, "tag": "ON", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 53.83, "load1": 9.09, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029255.2568405, "tag": "ON", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 54.82, "load1": 9.41, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029260.4960387, "tag": "ON", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 55.21, "load1": 9.53, "pkg_w": 34.73, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029280.6212485, "tag": "OFF", "level": "typical", "seconds": 5.04, "cpus": "20-23", "busy_pct": 50.17, "load1": 7.78, "pkg_w": 35.0, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794029287.9250047, "tag": "OFF", "level": "typical", "seconds": 5.07, "cpus": "20-23", "busy_pct": 53.49, "load1": 7.56, "pkg_w": 35.65, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029293.168232, "tag": "OFF", "level": "typical", "seconds": 5.092, "cpus": "20-23", "busy_pct": 53.25, "load1": 7.84, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029298.4301596, "tag": "OFF", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 54.17, "load1": 8.25, "pkg_w": 34.43, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029303.6702275, "tag": "OFF", "level": "typical", "seconds": 5.085, "cpus": "20-23", "busy_pct": 55.97, "load1": 8.55, "pkg_w": 34.73, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029308.9212763, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 54.89, "load1": 8.67, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029314.150715, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 54.96, "load1": 8.85, "pkg_w": 34.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029319.3912272, "tag": "OFF", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 54.78, "load1": 9.6, "pkg_w": 34.79, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029324.6256456, "tag": "OFF", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 53.08, "load1": 9.55, "pkg_w": 34.27, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029329.8337893, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 55.75, "load1": 9.5, "pkg_w": 35.32, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029335.0713892, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 54.94, "load1": 9.62, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029340.3193643, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 53.58, "load1": 9.74, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029345.5813565, "tag": "OFF", "level": "typical", "seconds": 5.093, "cpus": "20-23", "busy_pct": 56.0, "load1": 9.84, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029350.8599644, "tag": "OFF", "level": "typical", "seconds": 5.105, "cpus": "20-23", "busy_pct": 56.11, "load1": 9.85, "pkg_w": 34.71, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029356.1347547, "tag": "OFF", "level": "typical", "seconds": 5.095, "cpus": "20-23", "busy_pct": 54.75, "load1": 9.86, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029361.391021, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 53.63, "load1": 10.11, "pkg_w": 34.84, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029366.6317677, "tag": "OFF", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 55.15, "load1": 10.27, "pkg_w": 34.71, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029371.863955, "tag": "OFF", "level": "typical", "seconds": 5.083, "cpus": "20-23", "busy_pct": 55.62, "load1": 10.24, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029377.132961, "tag": "OFF", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 54.74, "load1": 10.14, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029382.3561997, "tag": "OFF", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 54.61, "load1": 10.21, "pkg_w": 34.75, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029387.591189, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 54.13, "load1": 10.2, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029392.8336673, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 54.23, "load1": 9.94, "pkg_w": 34.99, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029398.0614154, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 54.89, "load1": 10.02, "pkg_w": 34.8, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029403.3206947, "tag": "OFF", "level": "typical", "seconds": 5.092, "cpus": "20-23", "busy_pct": 55.22, "load1": 10.18, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029408.5723035, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 55.7, "load1": 10.41, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029413.8452487, "tag": "OFF", "level": "typical", "seconds": 5.098, "cpus": "20-23", "busy_pct": 55.76, "load1": 10.7, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029419.1218002, "tag": "OFF", "level": "typical", "seconds": 5.089, "cpus": "20-23", "busy_pct": 54.83, "load1": 10.8, "pkg_w": 34.67, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029424.3637092, "tag": "OFF", "level": "typical", "seconds": 5.087, "cpus": "20-23", "busy_pct": 55.18, "load1": 11.39, "pkg_w": 34.69, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029429.592808, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 52.96, "load1": 10.95, "pkg_w": 35.11, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029434.8432813, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 53.79, "load1": 11.04, "pkg_w": 34.75, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029440.0830407, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 55.07, "load1": 11.44, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029445.344019, "tag": "OFF", "level": "typical", "seconds": 5.107, "cpus": "20-23", "busy_pct": 53.85, "load1": 11.4, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029450.5824766, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 55.25, "load1": 11.13, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029455.8181453, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 54.09, "load1": 11.28, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029461.064865, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 55.07, "load1": 11.5, "pkg_w": 35.07, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029466.3365715, "tag": "OFF", "level": "typical", "seconds": 5.094, "cpus": "20-23", "busy_pct": 55.23, "load1": 11.46, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029471.621255, "tag": "OFF", "level": "typical", "seconds": 5.098, "cpus": "20-23", "busy_pct": 57.53, "load1": 11.26, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029476.8778076, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 54.23, "load1": 11.16, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029482.128467, "tag": "OFF", "level": "typical", "seconds": 5.081, "cpus": "20-23", "busy_pct": 54.39, "load1": 10.99, "pkg_w": 34.68, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029487.3861415, "tag": "OFF", "level": "typical", "seconds": 5.083, "cpus": "20-23", "busy_pct": 55.64, "load1": 10.99, "pkg_w": 34.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029492.6205018, "tag": "OFF", "level": "typical", "seconds": 5.072, "cpus": "20-23", "busy_pct": 53.88, "load1": 10.83, "pkg_w": 34.11, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029497.839929, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 54.09, "load1": 10.76, "pkg_w": 35.52, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029503.0730953, "tag": "OFF", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 54.59, "load1": 10.86, "pkg_w": 34.78, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029508.3089147, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 53.19, "load1": 10.71, "pkg_w": 34.79, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029513.5486126, "tag": "OFF", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 54.34, "load1": 10.81, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029518.777333, "tag": "OFF", "level": "typical", "seconds": 5.073, "cpus": "20-23", "busy_pct": 54.39, "load1": 10.99, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029523.9938881, "tag": "OFF", "level": "typical", "seconds": 5.057, "cpus": "20-23", "busy_pct": 54.86, "load1": 10.99, "pkg_w": 34.5, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029544.331504, "tag": "ON", "level": "typical", "seconds": 5.027, "cpus": "20-23", "busy_pct": 49.75, "load1": 8.97, "pkg_w": 34.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794029551.6110318, "tag": "ON", "level": "typical", "seconds": 5.063, "cpus": "20-23", "busy_pct": 53.87, "load1": 8.59, "pkg_w": 35.72, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029556.8238528, "tag": "ON", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 53.86, "load1": 8.7, "pkg_w": 35.02, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029562.0616927, "tag": "ON", "level": "typical", "seconds": 5.083, "cpus": "20-23", "busy_pct": 53.74, "load1": 8.8, "pkg_w": 34.62, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029567.2968783, "tag": "ON", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 52.85, "load1": 9.3, "pkg_w": 34.74, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029572.5340228, "tag": "ON", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 56.34, "load1": 9.52, "pkg_w": 34.79, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029577.7734327, "tag": "ON", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 53.33, "load1": 9.72, "pkg_w": 34.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029583.0266452, "tag": "ON", "level": "typical", "seconds": 5.096, "cpus": "20-23", "busy_pct": 54.88, "load1": 9.98, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029588.2664843, "tag": "ON", "level": "typical", "seconds": 5.073, "cpus": "20-23", "busy_pct": 52.84, "load1": 10.06, "pkg_w": 34.8, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029593.488517, "tag": "ON", "level": "typical", "seconds": 5.071, "cpus": "20-23", "busy_pct": 53.19, "load1": 9.9, "pkg_w": 34.95, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029598.7061775, "tag": "ON", "level": "typical", "seconds": 5.067, "cpus": "20-23", "busy_pct": 52.44, "load1": 9.98, "pkg_w": 34.73, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029603.8704298, "tag": "ON", "level": "typical", "seconds": 5.033, "cpus": "20-23", "busy_pct": 52.3, "load1": 9.43, "pkg_w": 33.93, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029609.0230434, "tag": "ON", "level": "typical", "seconds": 5.075, "cpus": "20-23", "busy_pct": 53.23, "load1": 9.55, "pkg_w": 35.41, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029614.2378185, "tag": "ON", "level": "typical", "seconds": 5.089, "cpus": "20-23", "busy_pct": 57.26, "load1": 9.59, "pkg_w": 35.51, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029619.4662523, "tag": "ON", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 56.5, "load1": 9.95, "pkg_w": 34.7, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029624.7095559, "tag": "ON", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 54.77, "load1": 9.96, "pkg_w": 34.89, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029629.9480822, "tag": "ON", "level": "typical", "seconds": 5.079, "cpus": "20-23", "busy_pct": 53.76, "load1": 9.96, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029635.198177, "tag": "ON", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 54.6, "load1": 10.2, "pkg_w": 34.83, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029640.3831153, "tag": "ON", "level": "typical", "seconds": 5.046, "cpus": "20-23", "busy_pct": 53.95, "load1": 9.63, "pkg_w": 33.64, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029658.630547, "tag": "OFF", "level": "typical", "seconds": 5.033, "cpus": "20-23", "busy_pct": 49.58, "load1": 8.0, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "pre", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3, "settled": true, "settle_s": 5} +{"ts": 1794029665.9363177, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 53.65, "load1": 7.72, "pkg_w": 35.49, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029671.1479893, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 53.36, "load1": 8.06, "pkg_w": 35.11, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029676.3936048, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 54.81, "load1": 8.06, "pkg_w": 34.59, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029681.635575, "tag": "OFF", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 55.36, "load1": 8.37, "pkg_w": 34.8, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029686.8625863, "tag": "OFF", "level": "typical", "seconds": 5.076, "cpus": "20-23", "busy_pct": 53.41, "load1": 8.5, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029692.0900183, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 57.09, "load1": 8.78, "pkg_w": 34.84, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029697.3337648, "tag": "OFF", "level": "typical", "seconds": 5.075, "cpus": "20-23", "busy_pct": 54.88, "load1": 8.32, "pkg_w": 34.79, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029702.5852365, "tag": "OFF", "level": "typical", "seconds": 5.09, "cpus": "20-23", "busy_pct": 53.91, "load1": 8.61, "pkg_w": 35.1, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029707.8567262, "tag": "OFF", "level": "typical", "seconds": 5.093, "cpus": "20-23", "busy_pct": 56.12, "load1": 8.73, "pkg_w": 34.79, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029713.1381257, "tag": "OFF", "level": "typical", "seconds": 5.095, "cpus": "20-23", "busy_pct": 56.06, "load1": 8.91, "pkg_w": 34.76, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029718.3923826, "tag": "OFF", "level": "typical", "seconds": 5.074, "cpus": "20-23", "busy_pct": 54.8, "load1": 9.16, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029723.6499267, "tag": "OFF", "level": "typical", "seconds": 5.097, "cpus": "20-23", "busy_pct": 55.57, "load1": 9.54, "pkg_w": 34.92, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029728.8913546, "tag": "OFF", "level": "typical", "seconds": 5.078, "cpus": "20-23", "busy_pct": 54.13, "load1": 9.98, "pkg_w": 34.92, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029734.1031384, "tag": "OFF", "level": "typical", "seconds": 5.062, "cpus": "20-23", "busy_pct": 54.65, "load1": 10.14, "pkg_w": 34.55, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029739.2980227, "tag": "OFF", "level": "typical", "seconds": 5.075, "cpus": "20-23", "busy_pct": 54.79, "load1": 9.73, "pkg_w": 35.45, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029744.5570261, "tag": "OFF", "level": "typical", "seconds": 5.098, "cpus": "20-23", "busy_pct": 56.58, "load1": 10.59, "pkg_w": 34.88, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029749.8046293, "tag": "OFF", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 54.0, "load1": 10.62, "pkg_w": 34.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029755.0490015, "tag": "OFF", "level": "typical", "seconds": 5.086, "cpus": "20-23", "busy_pct": 53.8, "load1": 10.73, "pkg_w": 34.85, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029760.3226526, "tag": "OFF", "level": "typical", "seconds": 5.101, "cpus": "20-23", "busy_pct": 55.54, "load1": 10.67, "pkg_w": 34.98, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029765.5696595, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 55.79, "load1": 10.7, "pkg_w": 34.81, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029770.8358035, "tag": "OFF", "level": "typical", "seconds": 5.091, "cpus": "20-23", "busy_pct": 53.77, "load1": 10.64, "pkg_w": 35.02, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029776.106248, "tag": "OFF", "level": "typical", "seconds": 5.088, "cpus": "20-23", "busy_pct": 55.61, "load1": 10.51, "pkg_w": 34.82, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029781.3591142, "tag": "OFF", "level": "typical", "seconds": 5.077, "cpus": "20-23", "busy_pct": 55.25, "load1": 10.63, "pkg_w": 34.77, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029786.6046288, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 54.74, "load1": 10.5, "pkg_w": 34.92, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029791.8426316, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 56.22, "load1": 10.54, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029797.0786796, "tag": "OFF", "level": "typical", "seconds": 5.08, "cpus": "20-23", "busy_pct": 56.13, "load1": 10.74, "pkg_w": 34.86, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029802.3195133, "tag": "OFF", "level": "typical", "seconds": 5.082, "cpus": "20-23", "busy_pct": 52.64, "load1": 10.68, "pkg_w": 35.0, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} +{"ts": 1794029807.5552034, "tag": "OFF", "level": "typical", "seconds": 5.084, "cpus": "20-23", "busy_pct": 53.42, "load1": 10.79, "pkg_w": 34.84, "throttle_per_min": 0.0, "swap_used_mib": 0.0, "swap_delta_mib": 0.0, "phase": "during", "busy_cpus": "20-23", "load_expected": 3, "load_running": 3} diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-1.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-1.json new file mode 100644 index 00000000..47d042fd --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-1.json @@ -0,0 +1 @@ +{"seed": 2561, "arm": "ON", "warmup": {"ttft": 5.328356486978009, "total": 7.147476432961412, "text_len": 17, "prompt_tokens": 38, "completion_tokens": 4, "error": null}, "S": [{"ttft": 32.43029022193514, "total": 34.33885428193025, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a1"}, {"ttft": 2.204154353006743, "total": 4.202276110998355, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a2"}, {"ttft": 2.119715873966925, "total": 4.130007716012187, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a3"}, {"ttft": 2.190626053023152, "total": 4.196175363031216, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a4"}, {"ttft": 2.1099928149487823, "total": 4.109039952978492, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a5"}, {"ttft": 2.153619972988963, "total": 4.150863546994515, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a6"}, {"ttft": 32.253816806012765, "total": 34.13455553899985, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "c"}], "B": [], "mem": [], "drss_per_agent_mib": null, "swap_in_burst": false, "graph_ok": true, "energy_uj": ["96415533998", "99854046008"], "throttle_delta": 0} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-2.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-2.json new file mode 100644 index 00000000..37b113e3 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-2.json @@ -0,0 +1 @@ +{"seed": 2562, "arm": "OFF", "warmup": {"ttft": 5.466046908055432, "total": 7.385430850088596, "text_len": 17, "prompt_tokens": 38, "completion_tokens": 4, "error": null}, "S": [{"ttft": 32.473897791001946, "total": 34.35398981894832, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a1"}, {"ttft": 29.978655762970448, "total": 31.886813127901405, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a2"}, {"ttft": 32.30350062099751, "total": 34.21355658606626, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a3"}, {"ttft": 30.520233896910213, "total": 32.51761358289514, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a4"}, {"ttft": 31.40780486294534, "total": 33.30285911494866, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a5"}, {"ttft": 31.465521304053254, "total": 33.45478583802469, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a6"}, {"ttft": 30.623118146904744, "total": 32.50655829790048, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "c"}], "B": [], "mem": [], "drss_per_agent_mib": null, "swap_in_burst": false, "graph_ok": true, "energy_uj": ["100337566133", "108776485752"], "throttle_delta": 0} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-3.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-3.json new file mode 100644 index 00000000..cf753605 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-3.json @@ -0,0 +1 @@ +{"seed": 2563, "arm": "ON", "warmup": {"ttft": 6.035486338078044, "total": 7.914590311003849, "text_len": 17, "prompt_tokens": 37, "completion_tokens": 4, "error": null}, "S": [{"ttft": 30.87193522101734, "total": 32.7513375839917, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a1"}, {"ttft": 2.1505636520450935, "total": 4.2066242130240425, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a2"}, {"ttft": 2.204036680981517, "total": 4.227393485954963, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a3"}, {"ttft": 2.0910487030632794, "total": 4.116710575995967, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a4"}, {"ttft": 2.3701426689513028, "total": 4.508218331960961, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a5"}, {"ttft": 2.435048700077459, "total": 4.593893364071846, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a6"}, {"ttft": 31.50250490801409, "total": 33.405896416981705, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "c"}], "B": [], "mem": [], "drss_per_agent_mib": null, "swap_in_burst": false, "graph_ok": true, "energy_uj": ["109444707247", "112851067362"], "throttle_delta": 0} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-4.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-4.json new file mode 100644 index 00000000..87d575d5 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-4.json @@ -0,0 +1 @@ +{"seed": 2564, "arm": "OFF", "warmup": {"ttft": 6.039172212011181, "total": 7.967303453013301, "text_len": 17, "prompt_tokens": 37, "completion_tokens": 4, "error": null}, "S": [{"ttft": 31.4112520609051, "total": 33.31402681698091, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a1"}, {"ttft": 32.58352233201731, "total": 34.5518281069817, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a2"}, {"ttft": 30.30790936492849, "total": 32.206166236894205, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a3"}, {"ttft": 32.72945735999383, "total": 34.62810139893554, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a4"}, {"ttft": 30.220451082102954, "total": 32.11941855400801, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a5"}, {"ttft": 32.726326987030916, "total": 34.63763544103131, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a6"}, {"ttft": 30.608316531055607, "total": 32.6283833560301, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "c"}], "B": [], "mem": [], "drss_per_agent_mib": null, "swap_in_burst": false, "graph_ok": true, "energy_uj": ["113498732818", "121998512878"], "throttle_delta": 0} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-5.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-5.json new file mode 100644 index 00000000..4a8dcde1 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/round-5.json @@ -0,0 +1 @@ +{"seed": 2565, "arm": "ON", "warmup": {"ttft": 5.471553103066981, "total": 7.296789239975624, "text_len": 17, "prompt_tokens": 38, "completion_tokens": 4, "error": null}, "S": [{"ttft": 30.879644831060432, "total": 32.91791498300154, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a1"}, {"ttft": 2.5247044879943132, "total": 4.66277340403758, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a2"}, {"ttft": 2.4060273899231106, "total": 4.531224192935042, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a3"}, {"ttft": 2.2809814469655976, "total": 4.31706422904972, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a4"}, {"ttft": 2.1814400840085, "total": 4.183711685007438, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a5"}, {"ttft": 2.1977264269953594, "total": 4.213904211996123, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "a6"}, {"ttft": 30.08638923894614, "total": 31.955919418949634, "text_len": 0, "prompt_tokens": 1634, "completion_tokens": 8, "error": null, "agent": "c"}], "B": [], "mem": [], "drss_per_agent_mib": null, "swap_in_burst": false, "graph_ok": true, "energy_uj": ["122684922070", "126038902864"], "throttle_delta": 0} \ No newline at end of file diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/typical/score.json b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/score.json new file mode 100644 index 00000000..717a35df --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/typical/score.json @@ -0,0 +1,156 @@ +{ + "n_rounds": { + "ON": 3, + "OFF": 2 + }, + "void": {}, + "median_added_ttft_s": { + "ON": 2.1977264269953594, + "OFF": 31.436663083499297 + }, + "n_added": { + "ON": 15, + "OFF": 10 + }, + "per_round_median_s": { + "ON": [ + 2.153619972988963, + 2.204036680981517, + 2.2809814469655976 + ], + "OFF": [ + 31.40780486294534, + 32.58352233201731 + ] + }, + "spread_s": { + "ON": 0.12736147397663444, + "OFF": 1.1757174690719694 + }, + "ratio": 14.30417484967781, + "ci95": [ + 13.38030782210502, + 14.891222488384075 + ], + "confidence_pairs_ge_4x": 1.0, + "c_on_over_off": 1.0020944279085826, + "a1_on_over_off": 0.9667237191316892, + "prompt_tokens": [ + "1634" + ], + "inconclusive_parts": [], + "T": "SUPPORTED", + "M": "INCONCLUSIVE (not measured: phase B not run, amendment A3)", + "load": { + "matched": true, + "void": false, + "verdict": "MATCHED", + "rule": "amendment-27", + "checks": [ + { + "metric": "busy_pct", + "arms": { + "ON": 50.053, + "OFF": 49.85 + }, + "spread": 0.203, + "limit": 5.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "throttle_per_min", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 10.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "swap_used_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 256.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "swap_delta_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 64.0, + "status": "ok", + "phase": "pre" + }, + { + "metric": "busy_pct", + "arms": { + "ON": 54.238, + "OFF": 54.921 + }, + "spread": 0.683, + "limit": 5.0, + "status": "ok", + "phase": "during" + }, + { + "metric": "swap_used_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 256.0, + "status": "ok", + "phase": "during" + }, + { + "metric": "swap_delta_mib", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0, + "limit": 64.0, + "status": "ok", + "phase": "during" + } + ], + "outcomes": [ + { + "metric": "pkg_w", + "arms": { + "ON": 34.865, + "OFF": 34.874 + }, + "spread": 0.009 + }, + { + "metric": "throttle_per_min", + "arms": { + "ON": 0.0, + "OFF": 0.0 + }, + "spread": 0.0 + }, + { + "metric": "load1", + "arms": { + "ON": 8.432, + "OFF": 10.118 + }, + "spread": 1.686 + } + ], + "unsettled": {} + } +} diff --git a/bench/wildct/results/2026-11-07/runs/H25w/out/up.log b/bench/wildct/results/2026-11-07/runs/H25w/out/up.log new file mode 100644 index 00000000..fdebf50c --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/out/up.log @@ -0,0 +1,34 @@ +==> opted out by K3D_GPU_VENDORS: nvidia intel (detected here: nvidia intel); not deployed, not advertised +==> Accelerators: hybrid +==> Experimental features (auto): dra image-volume +==> Creating k3d cluster 'k3dgpu-w-h25' with image cryptoandcoffee/k3d-gpu:0.1.89@sha256:2e53b5114e081d0b86f034edd6c2550eab2156dc3e2f6acaa1a42b99297cd2f4... +INFO[0000] Prep: Network +INFO[0000] Created network 'k3d-k3dgpu-w-h25' +INFO[0000] Created image volume k3d-k3dgpu-w-h25-images +INFO[0000] Starting new tools node... +INFO[0000] Starting node 'k3d-k3dgpu-w-h25-tools' +INFO[0001] Creating node 'k3d-k3dgpu-w-h25-server-0' +INFO[0001] Using the k3d-tools node to gather environment information +INFO[0001] HostIP: using network gateway 172.19.0.1 address +INFO[0001] Starting cluster 'k3dgpu-w-h25' +INFO[0001] Starting servers... +INFO[0001] Starting node 'k3d-k3dgpu-w-h25-server-0' +INFO[0005] All agents already running. +INFO[0005] All helpers already running. +INFO[0005] Injecting records for hostAliases (incl. host.k3d.internal) and for 1 network members into CoreDNS configmap... +INFO[0007] Cluster 'k3dgpu-w-h25' created successfully! +INFO[0007] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false +INFO[0007] You can now use it like this: +export KUBECONFIG=$(k3d kubeconfig write k3dgpu-w-h25) +kubectl cluster-info +==> kubectl for this cluster: export KUBECONFIG=/home/andrew/k3dgpu-bench-wildct/runs/H25w/out/kubeconfig +==> Waiting for nodes Ready... +node/k3d-k3dgpu-w-h25-server-0 condition met +==> Waiting for cluster DNS (CoreDNS)... +node/k3d-k3dgpu-w-h25-server-0 annotated +==> hybrid device plugin auto-deployed by k3s. +balloonspolicy.config.nri/default patched +==> Verifying accelerators are schedulable... +Waiting for daemon set "nri-resource-policy-balloons" rollout to finish: 0 of 1 updated pods are available... +daemon set "nri-resource-policy-balloons" successfully rolled out +==> Accelerators schedulable: hybrid=1. Try: k3d-gpu test diff --git a/bench/wildct/results/2026-11-07/runs/H25w/pid b/bench/wildct/results/2026-11-07/runs/H25w/pid new file mode 100644 index 00000000..fb64f077 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/pid @@ -0,0 +1 @@ +52241 diff --git a/bench/wildct/results/2026-11-07/runs/H25w/post.txt b/bench/wildct/results/2026-11-07/runs/H25w/post.txt new file mode 100644 index 00000000..1ea7143a --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/post.txt @@ -0,0 +1,17 @@ +2026-11-07T05:36:58Z + 00:36:58 up 8 days, 20:25, 2 users, load average: 10.18, 9.66, 6.90 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 1.7 6519 andrew rustdesk + 1.6 40172 andrew localsearch-ext + 1.4 1 root systemd + 1.0 9906 911 python3 + 1.0 8054 andrew firefox + 0.9 9806 andrew WebExtensions + 0.9 1184 root gdm + 0.7 7117 andrew rustdesk + 0.3 52654 root kworker/u48:5-flush-259:0 + total used free shared buff/cache available +Mem: 125 11 93 2 24 114 +/dev/nvme0n1p2 3667G 3274G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H25w/pre.txt b/bench/wildct/results/2026-11-07/runs/H25w/pre.txt new file mode 100644 index 00000000..c4b0ceb9 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/pre.txt @@ -0,0 +1,17 @@ +2026-11-07T05:18:20Z + 00:18:20 up 8 days, 20:07, 2 users, load average: 0.99, 0.55, 0.49 +%CPU PID USER COMMAND + 2.6 1157 root tailscaled + 1.4 1 root systemd + 1.2 6519 andrew rustdesk + 1.0 8054 andrew firefox + 1.0 9906 911 python3 + 0.9 9806 andrew WebExtensions + 0.8 1184 root gdm + 0.5 7117 andrew rustdesk + 0.3 3313 root dockerd + 0.3 5486 root nxrunner.bin + total used free shared buff/cache available +Mem: 125 11 93 2 23 114 +/dev/nvme0n1p2 3667G 3274G 207G 95% / +bmd-telemetry-jaeger-1 bmd-telemetry-prometheus-1 feed-my-algorithm-app-1 feed-my-algorithm-controller-1 feed-my-algorithm-db-1 feed-my-algorithm-redroid-1 feed-my-algorithm-worker-1 know-my-neighbor-db-1 proxmox-dc-manager searxng-core searxng-valkey state-of-my-town-app state-of-my-town-archivebox state-of-my-town-postgres tos-rate-calculator webapp-adminer-1 webapp-db-1 webapp-web-1 diff --git a/bench/wildct/results/2026-11-07/runs/H25w/rc b/bench/wildct/results/2026-11-07/runs/H25w/rc new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/rc @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/H25w/sample.jsonl b/bench/wildct/results/2026-11-07/runs/H25w/sample.jsonl new file mode 100644 index 00000000..4eb89684 --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/sample.jsonl @@ -0,0 +1,223 @@ +{"ts": 1794028705.542, "seconds": 5.0, "load1": 1.15, "busy_p": 1.044, "busy_e_free": 0.682, "busy_e_load": 1.954, "busy_all": 3.68, "mine_cpu": 0.0, "foreign_cpu": 3.68, "other_containers_cpu": 0.056, "mem_available_gib": 113.66, "pkg_temp_c": 62.0, "root_avail_gb": 221.8} +{"ts": 1794028710.543, "seconds": 5.001, "load1": 1.3, "busy_p": 0.86, "busy_e_free": 0.492, "busy_e_load": 2.035, "busy_all": 3.387, "mine_cpu": 0.0, "foreign_cpu": 3.387, "other_containers_cpu": 0.065, "mem_available_gib": 113.6, "pkg_temp_c": 61.0, "root_avail_gb": 221.8} +{"ts": 1794028715.544, "seconds": 5.001, "load1": 1.51, "busy_p": 0.512, "busy_e_free": 0.464, "busy_e_load": 2.014, "busy_all": 2.989, "mine_cpu": 0.0, "foreign_cpu": 2.989, "other_containers_cpu": 0.102, "mem_available_gib": 113.57, "pkg_temp_c": 64.0, "root_avail_gb": 221.9} +{"ts": 1794028720.546, "seconds": 5.001, "load1": 1.55, "busy_p": 0.268, "busy_e_free": 0.356, "busy_e_load": 1.981, "busy_all": 2.605, "mine_cpu": 0.0, "foreign_cpu": 2.605, "other_containers_cpu": 0.059, "mem_available_gib": 113.55, "pkg_temp_c": 62.0, "root_avail_gb": 221.8} +{"ts": 1794028725.547, "seconds": 5.001, "load1": 1.59, "busy_p": 0.268, "busy_e_free": 0.39, "busy_e_load": 1.992, "busy_all": 2.649, "mine_cpu": 0.0, "foreign_cpu": 2.649, "other_containers_cpu": 0.115, "mem_available_gib": 113.52, "pkg_temp_c": 64.0, "root_avail_gb": 221.8} +{"ts": 1794028730.548, "seconds": 5.001, "load1": 2.18, "busy_p": 1.206, "busy_e_free": 0.68, "busy_e_load": 2.054, "busy_all": 3.939, "mine_cpu": 0.0, "foreign_cpu": 3.939, "other_containers_cpu": 0.07, "mem_available_gib": 113.36, "pkg_temp_c": 62.0, "root_avail_gb": 221.3} +{"ts": 1794028735.569, "seconds": 5.021, "load1": 2.25, "busy_p": 1.756, "busy_e_free": 0.844, "busy_e_load": 2.149, "busy_all": 4.75, "mine_cpu": 3.739, "foreign_cpu": 1.011, "other_containers_cpu": 0.078, "mem_available_gib": 113.19, "pkg_temp_c": 63.0, "root_avail_gb": 220.8} +{"ts": 1794028740.57, "seconds": 5.001, "load1": 2.07, "busy_p": 0.312, "busy_e_free": 0.396, "busy_e_load": 1.984, "busy_all": 2.691, "mine_cpu": 2.061, "foreign_cpu": 0.63, "other_containers_cpu": 0.061, "mem_available_gib": 113.19, "pkg_temp_c": 63.0, "root_avail_gb": 221.0} +{"ts": 1794028745.572, "seconds": 5.001, "load1": 2.22, "busy_p": 1.47, "busy_e_free": 1.138, "busy_e_load": 2.209, "busy_all": 4.817, "mine_cpu": 2.239, "foreign_cpu": 2.578, "other_containers_cpu": 0.122, "mem_available_gib": 113.14, "pkg_temp_c": 63.0, "root_avail_gb": 217.6} +{"ts": 1794028750.573, "seconds": 5.001, "load1": 2.45, "busy_p": 0.358, "busy_e_free": 0.428, "busy_e_load": 1.995, "busy_all": 2.781, "mine_cpu": 2.064, "foreign_cpu": 0.717, "other_containers_cpu": 0.065, "mem_available_gib": 113.18, "pkg_temp_c": 63.0, "root_avail_gb": 217.5} +{"ts": 1794028755.575, "seconds": 5.001, "load1": 2.33, "busy_p": 0.314, "busy_e_free": 0.444, "busy_e_load": 1.983, "busy_all": 2.741, "mine_cpu": 1.977, "foreign_cpu": 0.764, "other_containers_cpu": 0.116, "mem_available_gib": 113.2, "pkg_temp_c": 64.0, "root_avail_gb": 217.6} +{"ts": 1794028760.576, "seconds": 5.001, "load1": 2.3, "busy_p": 0.254, "busy_e_free": 0.388, "busy_e_load": 1.976, "busy_all": 2.617, "mine_cpu": 1.991, "foreign_cpu": 0.626, "other_containers_cpu": 0.069, "mem_available_gib": 113.17, "pkg_temp_c": 64.0, "root_avail_gb": 217.5} +{"ts": 1794028765.594, "seconds": 5.018, "load1": 2.36, "busy_p": 0.323, "busy_e_free": 0.454, "busy_e_load": 2.017, "busy_all": 2.794, "mine_cpu": 2.03, "foreign_cpu": 0.764, "other_containers_cpu": 0.064, "mem_available_gib": 113.14, "pkg_temp_c": 63.0, "root_avail_gb": 217.5} +{"ts": 1794028770.595, "seconds": 5.001, "load1": 2.33, "busy_p": 0.556, "busy_e_free": 0.5, "busy_e_load": 2.025, "busy_all": 3.081, "mine_cpu": 2.307, "foreign_cpu": 0.774, "other_containers_cpu": 0.06, "mem_available_gib": 113.14, "pkg_temp_c": 62.0, "root_avail_gb": 217.3} +{"ts": 1794028775.597, "seconds": 5.001, "load1": 2.38, "busy_p": 0.954, "busy_e_free": 0.55, "busy_e_load": 2.017, "busy_all": 3.521, "mine_cpu": 2.602, "foreign_cpu": 0.919, "other_containers_cpu": 0.097, "mem_available_gib": 113.15, "pkg_temp_c": 62.0, "root_avail_gb": 216.7} +{"ts": 1794028780.598, "seconds": 5.001, "load1": 2.35, "busy_p": 1.406, "busy_e_free": 0.696, "busy_e_load": 2.065, "busy_all": 4.167, "mine_cpu": 3.419, "foreign_cpu": 0.748, "other_containers_cpu": 0.076, "mem_available_gib": 112.98, "pkg_temp_c": 63.0, "root_avail_gb": 216.1} +{"ts": 1794028785.6, "seconds": 5.001, "load1": 2.41, "busy_p": 1.646, "busy_e_free": 0.59, "busy_e_load": 2.003, "busy_all": 4.239, "mine_cpu": 3.462, "foreign_cpu": 0.777, "other_containers_cpu": 0.141, "mem_available_gib": 112.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028790.601, "seconds": 5.001, "load1": 2.45, "busy_p": 0.412, "busy_e_free": 0.554, "busy_e_load": 2.017, "busy_all": 2.983, "mine_cpu": 2.215, "foreign_cpu": 0.768, "other_containers_cpu": 0.072, "mem_available_gib": 112.52, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028795.619, "seconds": 5.018, "load1": 2.74, "busy_p": 3.197, "busy_e_free": 0.648, "busy_e_load": 2.083, "busy_all": 5.927, "mine_cpu": 4.97, "foreign_cpu": 0.957, "other_containers_cpu": 0.098, "mem_available_gib": 111.75, "pkg_temp_c": 62.0, "root_avail_gb": 213.3} +{"ts": 1794028800.623, "seconds": 5.004, "load1": 3.0, "busy_p": 6.005, "busy_e_free": 0.745, "busy_e_load": 2.092, "busy_all": 8.843, "mine_cpu": 7.857, "foreign_cpu": 0.986, "other_containers_cpu": 0.13, "mem_available_gib": 110.0, "pkg_temp_c": 62.0, "root_avail_gb": 213.3} +{"ts": 1794028805.625, "seconds": 5.002, "load1": 4.36, "busy_p": 7.849, "busy_e_free": 0.968, "busy_e_load": 2.185, "busy_all": 11.001, "mine_cpu": 9.492, "foreign_cpu": 1.509, "other_containers_cpu": 0.243, "mem_available_gib": 109.86, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028810.63, "seconds": 5.005, "load1": 5.13, "busy_p": 8.026, "busy_e_free": 1.029, "busy_e_load": 2.18, "busy_all": 11.234, "mine_cpu": 9.74, "foreign_cpu": 1.494, "other_containers_cpu": 0.197, "mem_available_gib": 109.82, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028815.636, "seconds": 5.005, "load1": 6.24, "busy_p": 8.001, "busy_e_free": 1.203, "busy_e_load": 2.281, "busy_all": 11.485, "mine_cpu": 9.739, "foreign_cpu": 1.746, "other_containers_cpu": 0.416, "mem_available_gib": 109.79, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028820.641, "seconds": 5.005, "load1": 6.87, "busy_p": 8.072, "busy_e_free": 0.951, "busy_e_load": 2.2, "busy_all": 11.223, "mine_cpu": 9.814, "foreign_cpu": 1.409, "other_containers_cpu": 0.201, "mem_available_gib": 109.75, "pkg_temp_c": 62.0, "root_avail_gb": 213.4} +{"ts": 1794028825.683, "seconds": 5.042, "load1": 7.2, "busy_p": 8.062, "busy_e_free": 1.01, "busy_e_load": 2.138, "busy_all": 11.21, "mine_cpu": 9.758, "foreign_cpu": 1.452, "other_containers_cpu": 0.172, "mem_available_gib": 109.71, "pkg_temp_c": 62.0, "root_avail_gb": 213.2} +{"ts": 1794028830.688, "seconds": 5.005, "load1": 7.66, "busy_p": 8.0, "busy_e_free": 0.871, "busy_e_load": 2.144, "busy_all": 11.015, "mine_cpu": 9.766, "foreign_cpu": 1.249, "other_containers_cpu": 0.16, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028835.692, "seconds": 5.005, "load1": 7.45, "busy_p": 6.238, "busy_e_free": 0.781, "busy_e_load": 2.106, "busy_all": 9.125, "mine_cpu": 7.866, "foreign_cpu": 1.259, "other_containers_cpu": 0.186, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028840.696, "seconds": 5.004, "load1": 6.93, "busy_p": 5.714, "busy_e_free": 0.763, "busy_e_load": 2.104, "busy_all": 8.582, "mine_cpu": 7.328, "foreign_cpu": 1.254, "other_containers_cpu": 0.149, "mem_available_gib": 109.62, "pkg_temp_c": 62.0, "root_avail_gb": 213.3} +{"ts": 1794028845.7, "seconds": 5.004, "load1": 7.02, "busy_p": 5.605, "busy_e_free": 0.773, "busy_e_load": 2.076, "busy_all": 8.455, "mine_cpu": 7.307, "foreign_cpu": 1.148, "other_containers_cpu": 0.191, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028850.704, "seconds": 5.004, "load1": 7.58, "busy_p": 5.678, "busy_e_free": 0.693, "busy_e_load": 2.084, "busy_all": 8.456, "mine_cpu": 7.356, "foreign_cpu": 1.1, "other_containers_cpu": 0.112, "mem_available_gib": 109.63, "pkg_temp_c": 62.0, "root_avail_gb": 213.4} +{"ts": 1794028855.731, "seconds": 5.027, "load1": 7.45, "busy_p": 4.573, "busy_e_free": 0.742, "busy_e_load": 2.053, "busy_all": 7.368, "mine_cpu": 6.232, "foreign_cpu": 1.136, "other_containers_cpu": 0.127, "mem_available_gib": 109.62, "pkg_temp_c": 62.0, "root_avail_gb": 213.4} +{"ts": 1794028860.733, "seconds": 5.002, "load1": 7.58, "busy_p": 7.955, "busy_e_free": 0.86, "busy_e_load": 2.187, "busy_all": 11.001, "mine_cpu": 9.701, "foreign_cpu": 1.3, "other_containers_cpu": 0.153, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794028865.739, "seconds": 5.005, "load1": 7.69, "busy_p": 8.006, "busy_e_free": 1.123, "busy_e_load": 2.222, "busy_all": 11.35, "mine_cpu": 9.738, "foreign_cpu": 1.612, "other_containers_cpu": 0.288, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794028870.745, "seconds": 5.006, "load1": 7.95, "busy_p": 7.992, "busy_e_free": 1.093, "busy_e_load": 2.183, "busy_all": 11.268, "mine_cpu": 9.74, "foreign_cpu": 1.528, "other_containers_cpu": 0.209, "mem_available_gib": 109.63, "pkg_temp_c": 62.0, "root_avail_gb": 213.1} +{"ts": 1794028875.751, "seconds": 5.006, "load1": 8.2, "busy_p": 8.102, "busy_e_free": 1.057, "busy_e_load": 2.279, "busy_all": 11.438, "mine_cpu": 9.757, "foreign_cpu": 1.681, "other_containers_cpu": 0.375, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028880.756, "seconds": 5.006, "load1": 8.58, "busy_p": 8.117, "busy_e_free": 1.059, "busy_e_load": 2.148, "busy_all": 11.323, "mine_cpu": 9.751, "foreign_cpu": 1.572, "other_containers_cpu": 0.219, "mem_available_gib": 109.63, "pkg_temp_c": 62.0, "root_avail_gb": 213.3} +{"ts": 1794028885.802, "seconds": 5.045, "load1": 8.78, "busy_p": 8.081, "busy_e_free": 0.832, "busy_e_load": 2.168, "busy_all": 11.081, "mine_cpu": 9.806, "foreign_cpu": 1.275, "other_containers_cpu": 0.177, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028890.807, "seconds": 5.005, "load1": 8.88, "busy_p": 5.283, "busy_e_free": 0.915, "busy_e_load": 2.164, "busy_all": 8.362, "mine_cpu": 6.898, "foreign_cpu": 1.464, "other_containers_cpu": 0.137, "mem_available_gib": 113.03, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028895.809, "seconds": 5.002, "load1": 8.32, "busy_p": 1.157, "busy_e_free": 0.582, "busy_e_load": 2.003, "busy_all": 3.742, "mine_cpu": 3.043, "foreign_cpu": 0.699, "other_containers_cpu": 0.118, "mem_available_gib": 112.63, "pkg_temp_c": 63.0, "root_avail_gb": 214.4} +{"ts": 1794028900.812, "seconds": 5.003, "load1": 7.82, "busy_p": 0.96, "busy_e_free": 0.522, "busy_e_load": 2.029, "busy_all": 3.51, "mine_cpu": 2.778, "foreign_cpu": 0.732, "other_containers_cpu": 0.07, "mem_available_gib": 112.53, "pkg_temp_c": 65.0, "root_avail_gb": 213.1} +{"ts": 1794028905.814, "seconds": 5.002, "load1": 7.27, "busy_p": 0.342, "busy_e_free": 0.474, "busy_e_load": 2.013, "busy_all": 2.829, "mine_cpu": 1.992, "foreign_cpu": 0.837, "other_containers_cpu": 0.117, "mem_available_gib": 112.48, "pkg_temp_c": 65.0, "root_avail_gb": 213.4} +{"ts": 1794028910.816, "seconds": 5.002, "load1": 7.81, "busy_p": 7.281, "busy_e_free": 0.804, "busy_e_load": 2.167, "busy_all": 10.252, "mine_cpu": 9.178, "foreign_cpu": 1.074, "other_containers_cpu": 0.127, "mem_available_gib": 110.91, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028915.847, "seconds": 5.032, "load1": 7.51, "busy_p": 6.227, "busy_e_free": 0.741, "busy_e_load": 2.089, "busy_all": 9.057, "mine_cpu": 7.94, "foreign_cpu": 1.117, "other_containers_cpu": 0.118, "mem_available_gib": 109.95, "pkg_temp_c": 62.0, "root_avail_gb": 213.1} +{"ts": 1794028920.853, "seconds": 5.005, "load1": 7.95, "busy_p": 7.94, "busy_e_free": 0.911, "busy_e_load": 2.166, "busy_all": 11.016, "mine_cpu": 9.688, "foreign_cpu": 1.328, "other_containers_cpu": 0.206, "mem_available_gib": 109.89, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794028925.858, "seconds": 5.005, "load1": 8.35, "busy_p": 8.085, "busy_e_free": 1.049, "busy_e_load": 2.238, "busy_all": 11.372, "mine_cpu": 9.716, "foreign_cpu": 1.656, "other_containers_cpu": 0.283, "mem_available_gib": 109.83, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028930.863, "seconds": 5.005, "load1": 8.8, "busy_p": 7.994, "busy_e_free": 0.989, "busy_e_load": 2.234, "busy_all": 11.216, "mine_cpu": 9.658, "foreign_cpu": 1.558, "other_containers_cpu": 0.207, "mem_available_gib": 109.78, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028935.869, "seconds": 5.005, "load1": 9.3, "busy_p": 8.052, "busy_e_free": 1.175, "busy_e_load": 2.268, "busy_all": 11.494, "mine_cpu": 9.721, "foreign_cpu": 1.773, "other_containers_cpu": 0.398, "mem_available_gib": 109.75, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794028940.874, "seconds": 5.005, "load1": 9.44, "busy_p": 8.047, "busy_e_free": 0.907, "busy_e_load": 2.214, "busy_all": 11.168, "mine_cpu": 9.746, "foreign_cpu": 1.422, "other_containers_cpu": 0.185, "mem_available_gib": 109.71, "pkg_temp_c": 62.0, "root_avail_gb": 213.1} +{"ts": 1794028945.919, "seconds": 5.045, "load1": 10.04, "busy_p": 7.985, "busy_e_free": 0.952, "busy_e_load": 2.218, "busy_all": 11.154, "mine_cpu": 9.693, "foreign_cpu": 1.461, "other_containers_cpu": 0.184, "mem_available_gib": 109.66, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028950.923, "seconds": 5.004, "load1": 10.04, "busy_p": 6.386, "busy_e_free": 0.797, "busy_e_load": 2.084, "busy_all": 9.268, "mine_cpu": 8.109, "foreign_cpu": 1.159, "other_containers_cpu": 0.17, "mem_available_gib": 109.66, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028955.928, "seconds": 5.005, "load1": 10.04, "busy_p": 8.013, "busy_e_free": 0.969, "busy_e_load": 2.198, "busy_all": 11.18, "mine_cpu": 9.727, "foreign_cpu": 1.453, "other_containers_cpu": 0.236, "mem_available_gib": 109.67, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028960.934, "seconds": 5.006, "load1": 10.19, "busy_p": 8.1, "busy_e_free": 0.887, "busy_e_load": 2.121, "busy_all": 11.108, "mine_cpu": 9.755, "foreign_cpu": 1.353, "other_containers_cpu": 0.193, "mem_available_gib": 109.66, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028965.938, "seconds": 5.004, "load1": 10.42, "busy_p": 8.075, "busy_e_free": 0.967, "busy_e_load": 2.22, "busy_all": 11.263, "mine_cpu": 9.775, "foreign_cpu": 1.488, "other_containers_cpu": 0.274, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794028970.943, "seconds": 5.005, "load1": 10.38, "busy_p": 8.148, "busy_e_free": 1.053, "busy_e_load": 2.29, "busy_all": 11.49, "mine_cpu": 9.965, "foreign_cpu": 1.525, "other_containers_cpu": 0.229, "mem_available_gib": 109.6, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028975.987, "seconds": 5.044, "load1": 10.51, "busy_p": 8.021, "busy_e_free": 0.848, "busy_e_load": 2.201, "busy_all": 11.07, "mine_cpu": 9.774, "foreign_cpu": 1.296, "other_containers_cpu": 0.186, "mem_available_gib": 109.6, "pkg_temp_c": 62.0, "root_avail_gb": 213.2} +{"ts": 1794028980.992, "seconds": 5.005, "load1": 9.91, "busy_p": 6.402, "busy_e_free": 0.717, "busy_e_load": 2.156, "busy_all": 9.275, "mine_cpu": 8.092, "foreign_cpu": 1.183, "other_containers_cpu": 0.135, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794028985.997, "seconds": 5.005, "load1": 10.16, "busy_p": 7.927, "busy_e_free": 1.037, "busy_e_load": 2.228, "busy_all": 11.192, "mine_cpu": 9.65, "foreign_cpu": 1.542, "other_containers_cpu": 0.267, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794028991.004, "seconds": 5.006, "load1": 10.23, "busy_p": 7.96, "busy_e_free": 1.019, "busy_e_load": 2.303, "busy_all": 11.282, "mine_cpu": 9.735, "foreign_cpu": 1.547, "other_containers_cpu": 0.23, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794028996.009, "seconds": 5.006, "load1": 10.21, "busy_p": 8.12, "busy_e_free": 0.995, "busy_e_load": 2.217, "busy_all": 11.333, "mine_cpu": 9.82, "foreign_cpu": 1.513, "other_containers_cpu": 0.179, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029001.016, "seconds": 5.006, "load1": 10.19, "busy_p": 8.144, "busy_e_free": 0.973, "busy_e_load": 2.293, "busy_all": 11.41, "mine_cpu": 9.775, "foreign_cpu": 1.635, "other_containers_cpu": 0.36, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029006.06, "seconds": 5.045, "load1": 10.34, "busy_p": 8.073, "busy_e_free": 0.942, "busy_e_load": 2.149, "busy_all": 11.164, "mine_cpu": 9.81, "foreign_cpu": 1.354, "other_containers_cpu": 0.184, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029011.066, "seconds": 5.005, "load1": 10.47, "busy_p": 8.034, "busy_e_free": 1.067, "busy_e_load": 2.242, "busy_all": 11.342, "mine_cpu": 9.856, "foreign_cpu": 1.486, "other_containers_cpu": 0.168, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029016.071, "seconds": 5.005, "load1": 10.43, "busy_p": 6.429, "busy_e_free": 0.931, "busy_e_load": 2.17, "busy_all": 9.53, "mine_cpu": 8.141, "foreign_cpu": 1.389, "other_containers_cpu": 0.217, "mem_available_gib": 109.66, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029021.076, "seconds": 5.005, "load1": 10.64, "busy_p": 8.081, "busy_e_free": 0.945, "busy_e_load": 2.2, "busy_all": 11.226, "mine_cpu": 9.857, "foreign_cpu": 1.369, "other_containers_cpu": 0.232, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029026.082, "seconds": 5.005, "load1": 10.43, "busy_p": 8.105, "busy_e_free": 0.861, "busy_e_load": 2.21, "busy_all": 11.176, "mine_cpu": 9.811, "foreign_cpu": 1.365, "other_containers_cpu": 0.179, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029031.087, "seconds": 5.005, "load1": 10.31, "busy_p": 8.159, "busy_e_free": 1.151, "busy_e_load": 2.304, "busy_all": 11.614, "mine_cpu": 10.012, "foreign_cpu": 1.602, "other_containers_cpu": 0.332, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029036.131, "seconds": 5.044, "load1": 10.37, "busy_p": 8.063, "busy_e_free": 0.934, "busy_e_load": 2.167, "busy_all": 11.164, "mine_cpu": 9.784, "foreign_cpu": 1.38, "other_containers_cpu": 0.192, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029041.136, "seconds": 5.005, "load1": 10.18, "busy_p": 8.04, "busy_e_free": 0.887, "busy_e_load": 2.204, "busy_all": 11.131, "mine_cpu": 9.778, "foreign_cpu": 1.353, "other_containers_cpu": 0.171, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029046.141, "seconds": 5.005, "load1": 10.32, "busy_p": 6.465, "busy_e_free": 0.957, "busy_e_load": 2.192, "busy_all": 9.614, "mine_cpu": 8.15, "foreign_cpu": 1.464, "other_containers_cpu": 0.288, "mem_available_gib": 109.66, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029051.143, "seconds": 5.002, "load1": 10.46, "busy_p": 7.923, "busy_e_free": 1.012, "busy_e_load": 2.299, "busy_all": 11.233, "mine_cpu": 9.709, "foreign_cpu": 1.524, "other_containers_cpu": 0.195, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029056.148, "seconds": 5.005, "load1": 10.5, "busy_p": 7.997, "busy_e_free": 0.979, "busy_e_load": 2.166, "busy_all": 11.142, "mine_cpu": 9.743, "foreign_cpu": 1.399, "other_containers_cpu": 0.187, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029061.154, "seconds": 5.006, "load1": 10.7, "busy_p": 8.043, "busy_e_free": 1.085, "busy_e_load": 2.231, "busy_all": 11.359, "mine_cpu": 9.803, "foreign_cpu": 1.556, "other_containers_cpu": 0.391, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029066.196, "seconds": 5.042, "load1": 10.97, "busy_p": 8.118, "busy_e_free": 0.946, "busy_e_load": 2.212, "busy_all": 11.276, "mine_cpu": 9.846, "foreign_cpu": 1.43, "other_containers_cpu": 0.171, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029071.201, "seconds": 5.005, "load1": 10.89, "busy_p": 8.108, "busy_e_free": 0.955, "busy_e_load": 2.178, "busy_all": 11.241, "mine_cpu": 9.837, "foreign_cpu": 1.404, "other_containers_cpu": 0.181, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029076.205, "seconds": 5.005, "load1": 10.98, "busy_p": 8.126, "busy_e_free": 1.039, "busy_e_load": 2.246, "busy_all": 11.411, "mine_cpu": 9.816, "foreign_cpu": 1.595, "other_containers_cpu": 0.253, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029081.21, "seconds": 5.005, "load1": 10.26, "busy_p": 6.442, "busy_e_free": 0.771, "busy_e_load": 2.128, "busy_all": 9.341, "mine_cpu": 8.125, "foreign_cpu": 1.216, "other_containers_cpu": 0.145, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029086.215, "seconds": 5.005, "load1": 10.32, "busy_p": 8.083, "busy_e_free": 0.909, "busy_e_load": 2.208, "busy_all": 11.2, "mine_cpu": 9.799, "foreign_cpu": 1.401, "other_containers_cpu": 0.159, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029091.22, "seconds": 5.005, "load1": 10.69, "busy_p": 8.088, "busy_e_free": 1.165, "busy_e_load": 2.222, "busy_all": 11.475, "mine_cpu": 9.837, "foreign_cpu": 1.638, "other_containers_cpu": 0.375, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029096.262, "seconds": 5.042, "load1": 10.64, "busy_p": 8.092, "busy_e_free": 0.952, "busy_e_load": 2.184, "busy_all": 11.228, "mine_cpu": 9.796, "foreign_cpu": 1.432, "other_containers_cpu": 0.157, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029101.267, "seconds": 5.005, "load1": 10.75, "busy_p": 8.04, "busy_e_free": 0.895, "busy_e_load": 2.228, "busy_all": 11.162, "mine_cpu": 9.808, "foreign_cpu": 1.354, "other_containers_cpu": 0.208, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029106.273, "seconds": 5.006, "load1": 10.93, "busy_p": 8.075, "busy_e_free": 1.023, "busy_e_load": 2.21, "busy_all": 11.307, "mine_cpu": 9.724, "foreign_cpu": 1.583, "other_containers_cpu": 0.271, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029111.278, "seconds": 5.006, "load1": 11.25, "busy_p": 8.033, "busy_e_free": 1.021, "busy_e_load": 2.248, "busy_all": 11.302, "mine_cpu": 9.679, "foreign_cpu": 1.623, "other_containers_cpu": 0.231, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029116.284, "seconds": 5.006, "load1": 11.55, "busy_p": 6.393, "busy_e_free": 0.823, "busy_e_load": 2.182, "busy_all": 9.398, "mine_cpu": 8.114, "foreign_cpu": 1.284, "other_containers_cpu": 0.182, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029121.289, "seconds": 5.005, "load1": 11.19, "busy_p": 8.131, "busy_e_free": 1.109, "busy_e_load": 2.254, "busy_all": 11.494, "mine_cpu": 9.892, "foreign_cpu": 1.602, "other_containers_cpu": 0.347, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029126.332, "seconds": 5.043, "load1": 11.09, "busy_p": 8.051, "busy_e_free": 1.009, "busy_e_load": 2.191, "busy_all": 11.251, "mine_cpu": 9.83, "foreign_cpu": 1.421, "other_containers_cpu": 0.153, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029131.337, "seconds": 5.005, "load1": 11.09, "busy_p": 7.98, "busy_e_free": 0.945, "busy_e_load": 2.18, "busy_all": 11.105, "mine_cpu": 9.799, "foreign_cpu": 1.306, "other_containers_cpu": 0.203, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029136.342, "seconds": 5.005, "load1": 11.08, "busy_p": 8.086, "busy_e_free": 1.029, "busy_e_load": 2.13, "busy_all": 11.244, "mine_cpu": 9.789, "foreign_cpu": 1.455, "other_containers_cpu": 0.205, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029141.347, "seconds": 5.005, "load1": 10.99, "busy_p": 8.05, "busy_e_free": 0.975, "busy_e_load": 2.182, "busy_all": 11.206, "mine_cpu": 9.833, "foreign_cpu": 1.373, "other_containers_cpu": 0.199, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029146.352, "seconds": 5.005, "load1": 11.15, "busy_p": 5.229, "busy_e_free": 0.835, "busy_e_load": 2.17, "busy_all": 8.234, "mine_cpu": 6.832, "foreign_cpu": 1.402, "other_containers_cpu": 0.142, "mem_available_gib": 112.99, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029151.355, "seconds": 5.002, "load1": 10.42, "busy_p": 1.357, "busy_e_free": 0.522, "busy_e_load": 2.025, "busy_all": 3.904, "mine_cpu": 3.086, "foreign_cpu": 0.818, "other_containers_cpu": 0.134, "mem_available_gib": 112.6, "pkg_temp_c": 63.0, "root_avail_gb": 214.8} +{"ts": 1794029156.377, "seconds": 5.022, "load1": 9.67, "busy_p": 1.123, "busy_e_free": 0.456, "busy_e_load": 2.017, "busy_all": 3.596, "mine_cpu": 2.862, "foreign_cpu": 0.734, "other_containers_cpu": 0.062, "mem_available_gib": 112.5, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029161.379, "seconds": 5.002, "load1": 9.69, "busy_p": 0.276, "busy_e_free": 0.468, "busy_e_load": 1.993, "busy_all": 2.737, "mine_cpu": 2.01, "foreign_cpu": 0.727, "other_containers_cpu": 0.068, "mem_available_gib": 112.49, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029166.382, "seconds": 5.003, "load1": 9.0, "busy_p": 0.298, "busy_e_free": 0.5, "busy_e_load": 2.019, "busy_all": 2.817, "mine_cpu": 2.005, "foreign_cpu": 0.812, "other_containers_cpu": 0.097, "mem_available_gib": 112.48, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029171.384, "seconds": 5.002, "load1": 9.16, "busy_p": 7.063, "busy_e_free": 0.814, "busy_e_load": 2.141, "busy_all": 10.017, "mine_cpu": 8.861, "foreign_cpu": 1.156, "other_containers_cpu": 0.15, "mem_available_gib": 111.02, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029176.389, "seconds": 5.005, "load1": 8.58, "busy_p": 6.134, "busy_e_free": 0.757, "busy_e_load": 2.164, "busy_all": 9.055, "mine_cpu": 7.931, "foreign_cpu": 1.124, "other_containers_cpu": 0.122, "mem_available_gib": 109.91, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029181.395, "seconds": 5.006, "load1": 8.7, "busy_p": 8.058, "busy_e_free": 1.043, "busy_e_load": 2.227, "busy_all": 11.328, "mine_cpu": 9.697, "foreign_cpu": 1.631, "other_containers_cpu": 0.386, "mem_available_gib": 109.83, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029186.438, "seconds": 5.043, "load1": 8.72, "busy_p": 8.096, "busy_e_free": 0.861, "busy_e_load": 2.179, "busy_all": 11.136, "mine_cpu": 9.756, "foreign_cpu": 1.38, "other_containers_cpu": 0.156, "mem_available_gib": 109.8, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029191.443, "seconds": 5.005, "load1": 8.66, "busy_p": 8.027, "busy_e_free": 0.901, "busy_e_load": 2.22, "busy_all": 11.148, "mine_cpu": 9.751, "foreign_cpu": 1.397, "other_containers_cpu": 0.195, "mem_available_gib": 109.76, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029196.448, "seconds": 5.005, "load1": 8.61, "busy_p": 8.054, "busy_e_free": 0.993, "busy_e_load": 2.186, "busy_all": 11.232, "mine_cpu": 9.838, "foreign_cpu": 1.394, "other_containers_cpu": 0.248, "mem_available_gib": 109.72, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029201.453, "seconds": 5.005, "load1": 8.72, "busy_p": 8.112, "busy_e_free": 0.927, "busy_e_load": 2.218, "busy_all": 11.257, "mine_cpu": 9.814, "foreign_cpu": 1.443, "other_containers_cpu": 0.193, "mem_available_gib": 109.67, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029206.458, "seconds": 5.005, "load1": 8.66, "busy_p": 7.522, "busy_e_free": 0.949, "busy_e_load": 2.202, "busy_all": 10.673, "mine_cpu": 9.223, "foreign_cpu": 1.45, "other_containers_cpu": 0.17, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029211.461, "seconds": 5.003, "load1": 8.77, "busy_p": 4.701, "busy_e_free": 0.839, "busy_e_load": 2.199, "busy_all": 7.739, "mine_cpu": 6.508, "foreign_cpu": 1.231, "other_containers_cpu": 0.188, "mem_available_gib": 109.6, "pkg_temp_c": 66.0, "root_avail_gb": 213.3} +{"ts": 1794029216.49, "seconds": 5.03, "load1": 8.95, "busy_p": 5.663, "busy_e_free": 0.712, "busy_e_load": 2.137, "busy_all": 8.512, "mine_cpu": 7.366, "foreign_cpu": 1.146, "other_containers_cpu": 0.143, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029221.493, "seconds": 5.003, "load1": 8.47, "busy_p": 5.655, "busy_e_free": 0.692, "busy_e_load": 2.097, "busy_all": 8.444, "mine_cpu": 7.37, "foreign_cpu": 1.074, "other_containers_cpu": 0.165, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029226.496, "seconds": 5.003, "load1": 8.03, "busy_p": 5.715, "busy_e_free": 0.814, "busy_e_load": 2.081, "busy_all": 8.609, "mine_cpu": 7.35, "foreign_cpu": 1.259, "other_containers_cpu": 0.19, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029231.5, "seconds": 5.004, "load1": 8.35, "busy_p": 6.161, "busy_e_free": 0.935, "busy_e_load": 2.172, "busy_all": 9.268, "mine_cpu": 7.94, "foreign_cpu": 1.328, "other_containers_cpu": 0.164, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029236.505, "seconds": 5.005, "load1": 8.64, "busy_p": 8.02, "busy_e_free": 1.015, "busy_e_load": 2.214, "busy_all": 11.249, "mine_cpu": 9.748, "foreign_cpu": 1.501, "other_containers_cpu": 0.222, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029241.51, "seconds": 5.005, "load1": 8.75, "busy_p": 8.019, "busy_e_free": 0.883, "busy_e_load": 2.152, "busy_all": 11.054, "mine_cpu": 9.747, "foreign_cpu": 1.307, "other_containers_cpu": 0.198, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029246.551, "seconds": 5.041, "load1": 9.01, "busy_p": 8.067, "busy_e_free": 1.045, "busy_e_load": 2.226, "busy_all": 11.338, "mine_cpu": 9.773, "foreign_cpu": 1.565, "other_containers_cpu": 0.296, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029251.557, "seconds": 5.005, "load1": 9.09, "busy_p": 8.084, "busy_e_free": 0.943, "busy_e_load": 2.134, "busy_all": 11.161, "mine_cpu": 9.797, "foreign_cpu": 1.364, "other_containers_cpu": 0.22, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029256.561, "seconds": 5.005, "load1": 9.41, "busy_p": 8.124, "busy_e_free": 0.973, "busy_e_load": 2.23, "busy_all": 11.327, "mine_cpu": 9.794, "foreign_cpu": 1.533, "other_containers_cpu": 0.221, "mem_available_gib": 109.6, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029261.566, "seconds": 5.005, "load1": 9.53, "busy_p": 7.488, "busy_e_free": 0.875, "busy_e_load": 2.184, "busy_all": 10.547, "mine_cpu": 9.209, "foreign_cpu": 1.338, "other_containers_cpu": 0.199, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029266.569, "seconds": 5.003, "load1": 9.17, "busy_p": 1.007, "busy_e_free": 0.654, "busy_e_load": 2.029, "busy_all": 3.69, "mine_cpu": 2.684, "foreign_cpu": 1.006, "other_containers_cpu": 0.078, "mem_available_gib": 112.8, "pkg_temp_c": 64.0, "root_avail_gb": 215.5} +{"ts": 1794029271.571, "seconds": 5.002, "load1": 8.76, "busy_p": 1.609, "busy_e_free": 0.52, "busy_e_load": 2.053, "busy_all": 4.182, "mine_cpu": 3.37, "foreign_cpu": 0.812, "other_containers_cpu": 0.083, "mem_available_gib": 112.47, "pkg_temp_c": 66.0, "root_avail_gb": 213.3} +{"ts": 1794029276.59, "seconds": 5.019, "load1": 8.37, "busy_p": 0.379, "busy_e_free": 0.416, "busy_e_load": 2.018, "busy_all": 2.813, "mine_cpu": 2.012, "foreign_cpu": 0.801, "other_containers_cpu": 0.105, "mem_available_gib": 112.5, "pkg_temp_c": 66.0, "root_avail_gb": 213.2} +{"ts": 1794029281.593, "seconds": 5.003, "load1": 7.78, "busy_p": 0.322, "busy_e_free": 0.444, "busy_e_load": 2.009, "busy_all": 2.775, "mine_cpu": 2.01, "foreign_cpu": 0.765, "other_containers_cpu": 0.067, "mem_available_gib": 112.5, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029286.596, "seconds": 5.003, "load1": 7.56, "busy_p": 5.643, "busy_e_free": 0.696, "busy_e_load": 2.105, "busy_all": 8.443, "mine_cpu": 7.37, "foreign_cpu": 1.073, "other_containers_cpu": 0.204, "mem_available_gib": 111.31, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029291.6, "seconds": 5.004, "load1": 7.84, "busy_p": 6.057, "busy_e_free": 0.747, "busy_e_load": 2.106, "busy_all": 8.91, "mine_cpu": 7.718, "foreign_cpu": 1.192, "other_containers_cpu": 0.135, "mem_available_gib": 109.89, "pkg_temp_c": 65.0, "root_avail_gb": 213.3} +{"ts": 1794029296.605, "seconds": 5.005, "load1": 8.25, "busy_p": 7.978, "busy_e_free": 1.035, "busy_e_load": 2.202, "busy_all": 11.215, "mine_cpu": 9.719, "foreign_cpu": 1.496, "other_containers_cpu": 0.182, "mem_available_gib": 109.85, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029301.611, "seconds": 5.006, "load1": 8.55, "busy_p": 7.997, "busy_e_free": 0.981, "busy_e_load": 2.211, "busy_all": 11.189, "mine_cpu": 9.74, "foreign_cpu": 1.449, "other_containers_cpu": 0.243, "mem_available_gib": 109.8, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029306.657, "seconds": 5.047, "load1": 8.67, "busy_p": 8.067, "busy_e_free": 1.028, "busy_e_load": 2.211, "busy_all": 11.306, "mine_cpu": 9.937, "foreign_cpu": 1.369, "other_containers_cpu": 0.297, "mem_available_gib": 109.75, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029311.662, "seconds": 5.005, "load1": 8.85, "busy_p": 8.126, "busy_e_free": 0.975, "busy_e_load": 2.222, "busy_all": 11.323, "mine_cpu": 9.797, "foreign_cpu": 1.526, "other_containers_cpu": 0.199, "mem_available_gib": 109.71, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029316.667, "seconds": 5.005, "load1": 8.87, "busy_p": 8.088, "busy_e_free": 0.915, "busy_e_load": 2.162, "busy_all": 11.165, "mine_cpu": 9.733, "foreign_cpu": 1.432, "other_containers_cpu": 0.235, "mem_available_gib": 109.68, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029321.672, "seconds": 5.005, "load1": 9.6, "busy_p": 8.014, "busy_e_free": 0.971, "busy_e_load": 2.184, "busy_all": 11.169, "mine_cpu": 9.792, "foreign_cpu": 1.377, "other_containers_cpu": 0.176, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029326.678, "seconds": 5.005, "load1": 9.55, "busy_p": 6.365, "busy_e_free": 0.821, "busy_e_load": 2.138, "busy_all": 9.324, "mine_cpu": 8.114, "foreign_cpu": 1.21, "other_containers_cpu": 0.146, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029331.683, "seconds": 5.005, "load1": 9.5, "busy_p": 8.078, "busy_e_free": 1.029, "busy_e_load": 2.26, "busy_all": 11.367, "mine_cpu": 9.861, "foreign_cpu": 1.506, "other_containers_cpu": 0.213, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029336.726, "seconds": 5.043, "load1": 9.62, "busy_p": 8.063, "busy_e_free": 1.101, "busy_e_load": 2.134, "busy_all": 11.297, "mine_cpu": 9.727, "foreign_cpu": 1.57, "other_containers_cpu": 0.291, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029341.731, "seconds": 5.005, "load1": 9.74, "busy_p": 8.072, "busy_e_free": 1.019, "busy_e_load": 2.198, "busy_all": 11.288, "mine_cpu": 9.783, "foreign_cpu": 1.505, "other_containers_cpu": 0.219, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029346.736, "seconds": 5.006, "load1": 9.84, "busy_p": 8.027, "busy_e_free": 1.091, "busy_e_load": 2.243, "busy_all": 11.361, "mine_cpu": 9.792, "foreign_cpu": 1.569, "other_containers_cpu": 0.226, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029351.743, "seconds": 5.006, "load1": 9.85, "busy_p": 8.054, "busy_e_free": 1.119, "busy_e_load": 2.221, "busy_all": 11.394, "mine_cpu": 9.777, "foreign_cpu": 1.617, "other_containers_cpu": 0.22, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029356.748, "seconds": 5.006, "load1": 9.86, "busy_p": 8.043, "busy_e_free": 0.923, "busy_e_load": 2.199, "busy_all": 11.165, "mine_cpu": 9.707, "foreign_cpu": 1.458, "other_containers_cpu": 0.212, "mem_available_gib": 109.64, "pkg_temp_c": 65.0, "root_avail_gb": 213.1} +{"ts": 1794029361.754, "seconds": 5.005, "load1": 10.11, "busy_p": 6.463, "busy_e_free": 0.833, "busy_e_load": 2.124, "busy_all": 9.42, "mine_cpu": 8.131, "foreign_cpu": 1.289, "other_containers_cpu": 0.165, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029366.801, "seconds": 5.047, "load1": 10.27, "busy_p": 7.993, "busy_e_free": 0.941, "busy_e_load": 2.223, "busy_all": 11.157, "mine_cpu": 9.737, "foreign_cpu": 1.42, "other_containers_cpu": 0.305, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029371.806, "seconds": 5.005, "load1": 10.24, "busy_p": 8.078, "busy_e_free": 0.999, "busy_e_load": 2.21, "busy_all": 11.287, "mine_cpu": 9.792, "foreign_cpu": 1.495, "other_containers_cpu": 0.222, "mem_available_gib": 109.59, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029376.811, "seconds": 5.005, "load1": 10.14, "busy_p": 8.05, "busy_e_free": 0.881, "busy_e_load": 2.152, "busy_all": 11.083, "mine_cpu": 9.778, "foreign_cpu": 1.305, "other_containers_cpu": 0.214, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029381.815, "seconds": 5.005, "load1": 10.21, "busy_p": 8.06, "busy_e_free": 0.991, "busy_e_load": 2.206, "busy_all": 11.257, "mine_cpu": 9.838, "foreign_cpu": 1.419, "other_containers_cpu": 0.21, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029386.821, "seconds": 5.005, "load1": 10.2, "busy_p": 8.054, "busy_e_free": 0.945, "busy_e_load": 2.122, "busy_all": 11.12, "mine_cpu": 9.766, "foreign_cpu": 1.354, "other_containers_cpu": 0.173, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029391.825, "seconds": 5.004, "load1": 9.94, "busy_p": 6.474, "busy_e_free": 0.945, "busy_e_load": 2.184, "busy_all": 9.604, "mine_cpu": 8.258, "foreign_cpu": 1.346, "other_containers_cpu": 0.155, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029396.861, "seconds": 5.036, "load1": 10.02, "busy_p": 8.072, "busy_e_free": 1.042, "busy_e_load": 2.174, "busy_all": 11.288, "mine_cpu": 9.749, "foreign_cpu": 1.539, "other_containers_cpu": 0.326, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029401.866, "seconds": 5.005, "load1": 10.18, "busy_p": 8.095, "busy_e_free": 0.993, "busy_e_load": 2.244, "busy_all": 11.332, "mine_cpu": 9.754, "foreign_cpu": 1.578, "other_containers_cpu": 0.197, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029406.872, "seconds": 5.006, "load1": 10.41, "busy_p": 8.071, "busy_e_free": 1.041, "busy_e_load": 2.215, "busy_all": 11.327, "mine_cpu": 9.747, "foreign_cpu": 1.58, "other_containers_cpu": 0.25, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029411.878, "seconds": 5.006, "load1": 10.7, "busy_p": 8.04, "busy_e_free": 1.089, "busy_e_load": 2.231, "busy_all": 11.36, "mine_cpu": 9.758, "foreign_cpu": 1.602, "other_containers_cpu": 0.275, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029416.884, "seconds": 5.005, "load1": 10.8, "busy_p": 8.057, "busy_e_free": 1.019, "busy_e_load": 2.186, "busy_all": 11.262, "mine_cpu": 9.795, "foreign_cpu": 1.467, "other_containers_cpu": 0.183, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029421.889, "seconds": 5.005, "load1": 10.9, "busy_p": 8.041, "busy_e_free": 0.991, "busy_e_load": 2.164, "busy_all": 11.196, "mine_cpu": 9.81, "foreign_cpu": 1.386, "other_containers_cpu": 0.209, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029426.932, "seconds": 5.043, "load1": 11.39, "busy_p": 6.485, "busy_e_free": 0.878, "busy_e_load": 2.179, "busy_all": 9.542, "mine_cpu": 8.165, "foreign_cpu": 1.377, "other_containers_cpu": 0.25, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029431.936, "seconds": 5.004, "load1": 10.95, "busy_p": 8.007, "busy_e_free": 0.973, "busy_e_load": 2.146, "busy_all": 11.127, "mine_cpu": 9.734, "foreign_cpu": 1.393, "other_containers_cpu": 0.183, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029436.941, "seconds": 5.005, "load1": 11.04, "busy_p": 8.072, "busy_e_free": 0.815, "busy_e_load": 2.196, "busy_all": 11.083, "mine_cpu": 9.722, "foreign_cpu": 1.361, "other_containers_cpu": 0.212, "mem_available_gib": 109.6, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029441.945, "seconds": 5.005, "load1": 11.44, "busy_p": 8.055, "busy_e_free": 0.971, "busy_e_load": 2.178, "busy_all": 11.204, "mine_cpu": 9.688, "foreign_cpu": 1.516, "other_containers_cpu": 0.241, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029446.95, "seconds": 5.005, "load1": 11.4, "busy_p": 8.04, "busy_e_free": 0.853, "busy_e_load": 2.162, "busy_all": 11.055, "mine_cpu": 9.778, "foreign_cpu": 1.277, "other_containers_cpu": 0.158, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029451.955, "seconds": 5.004, "load1": 11.13, "busy_p": 8.139, "busy_e_free": 0.989, "busy_e_load": 2.252, "busy_all": 11.38, "mine_cpu": 9.785, "foreign_cpu": 1.595, "other_containers_cpu": 0.213, "mem_available_gib": 109.66, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029457.005, "seconds": 5.05, "load1": 11.28, "busy_p": 7.382, "busy_e_free": 0.982, "busy_e_load": 2.155, "busy_all": 10.519, "mine_cpu": 9.02, "foreign_cpu": 1.499, "other_containers_cpu": 0.281, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029462.006, "seconds": 5.001, "load1": 11.5, "busy_p": 7.188, "busy_e_free": 0.864, "busy_e_load": 2.225, "busy_all": 10.277, "mine_cpu": 8.856, "foreign_cpu": 1.421, "other_containers_cpu": 0.199, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029467.011, "seconds": 5.005, "load1": 11.46, "busy_p": 8.066, "busy_e_free": 1.109, "busy_e_load": 2.2, "busy_all": 11.375, "mine_cpu": 9.689, "foreign_cpu": 1.686, "other_containers_cpu": 0.249, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029472.017, "seconds": 5.006, "load1": 11.26, "busy_p": 8.093, "busy_e_free": 0.979, "busy_e_load": 2.283, "busy_all": 11.355, "mine_cpu": 9.754, "foreign_cpu": 1.601, "other_containers_cpu": 0.232, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029477.021, "seconds": 5.004, "load1": 11.16, "busy_p": 7.983, "busy_e_free": 0.925, "busy_e_load": 2.158, "busy_all": 11.067, "mine_cpu": 9.673, "foreign_cpu": 1.394, "other_containers_cpu": 0.162, "mem_available_gib": 109.64, "pkg_temp_c": 65.0, "root_avail_gb": 213.2} +{"ts": 1794029482.025, "seconds": 5.005, "load1": 10.99, "busy_p": 8.02, "busy_e_free": 0.939, "busy_e_load": 2.146, "busy_all": 11.106, "mine_cpu": 9.692, "foreign_cpu": 1.414, "other_containers_cpu": 0.211, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029487.073, "seconds": 5.048, "load1": 10.99, "busy_p": 8.07, "busy_e_free": 0.903, "busy_e_load": 2.177, "busy_all": 11.151, "mine_cpu": 9.784, "foreign_cpu": 1.367, "other_containers_cpu": 0.195, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029492.078, "seconds": 5.004, "load1": 10.83, "busy_p": 6.656, "busy_e_free": 0.957, "busy_e_load": 2.204, "busy_all": 9.817, "mine_cpu": 8.294, "foreign_cpu": 1.523, "other_containers_cpu": 0.298, "mem_available_gib": 109.65, "pkg_temp_c": 66.0, "root_avail_gb": 213.2} +{"ts": 1794029497.079, "seconds": 5.001, "load1": 10.76, "busy_p": 7.838, "busy_e_free": 0.852, "busy_e_load": 2.165, "busy_all": 10.855, "mine_cpu": 9.522, "foreign_cpu": 1.333, "other_containers_cpu": 0.188, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029502.083, "seconds": 5.004, "load1": 10.86, "busy_p": 8.065, "busy_e_free": 0.971, "busy_e_load": 2.154, "busy_all": 11.191, "mine_cpu": 9.767, "foreign_cpu": 1.424, "other_containers_cpu": 0.203, "mem_available_gib": 109.66, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029507.087, "seconds": 5.004, "load1": 10.71, "busy_p": 8.056, "busy_e_free": 0.891, "busy_e_load": 2.132, "busy_all": 11.08, "mine_cpu": 9.771, "foreign_cpu": 1.309, "other_containers_cpu": 0.155, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029512.091, "seconds": 5.004, "load1": 10.81, "busy_p": 8.129, "busy_e_free": 1.111, "busy_e_load": 2.224, "busy_all": 11.465, "mine_cpu": 9.82, "foreign_cpu": 1.645, "other_containers_cpu": 0.221, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029517.135, "seconds": 5.043, "load1": 10.99, "busy_p": 8.056, "busy_e_free": 0.892, "busy_e_load": 2.12, "busy_all": 11.068, "mine_cpu": 9.789, "foreign_cpu": 1.279, "other_containers_cpu": 0.151, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029522.138, "seconds": 5.004, "load1": 10.99, "busy_p": 8.094, "busy_e_free": 1.071, "busy_e_load": 2.238, "busy_all": 11.404, "mine_cpu": 9.734, "foreign_cpu": 1.67, "other_containers_cpu": 0.381, "mem_available_gib": 109.65, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029527.143, "seconds": 5.005, "load1": 10.59, "busy_p": 3.107, "busy_e_free": 0.873, "busy_e_load": 2.106, "busy_all": 6.086, "mine_cpu": 4.653, "foreign_cpu": 1.433, "other_containers_cpu": 0.18, "mem_available_gib": 113.03, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029532.144, "seconds": 5.001, "load1": 10.06, "busy_p": 1.518, "busy_e_free": 0.502, "busy_e_load": 2.044, "busy_all": 4.063, "mine_cpu": 3.223, "foreign_cpu": 0.84, "other_containers_cpu": 0.086, "mem_available_gib": 112.59, "pkg_temp_c": 64.0, "root_avail_gb": 213.8} +{"ts": 1794029537.146, "seconds": 5.001, "load1": 9.58, "busy_p": 0.94, "busy_e_free": 0.47, "busy_e_load": 1.985, "busy_all": 3.395, "mine_cpu": 2.636, "foreign_cpu": 0.759, "other_containers_cpu": 0.066, "mem_available_gib": 112.51, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029542.147, "seconds": 5.001, "load1": 8.97, "busy_p": 0.336, "busy_e_free": 0.466, "busy_e_load": 2.002, "busy_all": 2.803, "mine_cpu": 2.015, "foreign_cpu": 0.788, "other_containers_cpu": 0.067, "mem_available_gib": 112.52, "pkg_temp_c": 65.0, "root_avail_gb": 213.4} +{"ts": 1794029547.164, "seconds": 5.017, "load1": 8.81, "busy_p": 1.188, "busy_e_free": 0.516, "busy_e_load": 2.031, "busy_all": 3.735, "mine_cpu": 2.939, "foreign_cpu": 0.796, "other_containers_cpu": 0.085, "mem_available_gib": 112.22, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029552.166, "seconds": 5.002, "load1": 8.59, "busy_p": 7.367, "busy_e_free": 0.928, "busy_e_load": 2.145, "busy_all": 10.439, "mine_cpu": 9.18, "foreign_cpu": 1.259, "other_containers_cpu": 0.264, "mem_available_gib": 110.34, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029557.169, "seconds": 5.003, "load1": 8.7, "busy_p": 6.27, "busy_e_free": 0.831, "busy_e_load": 2.151, "busy_all": 9.252, "mine_cpu": 8.037, "foreign_cpu": 1.215, "other_containers_cpu": 0.158, "mem_available_gib": 109.9, "pkg_temp_c": 62.0, "root_avail_gb": 213.2} +{"ts": 1794029562.173, "seconds": 5.004, "load1": 8.8, "busy_p": 8.012, "busy_e_free": 0.973, "busy_e_load": 2.152, "busy_all": 11.138, "mine_cpu": 9.689, "foreign_cpu": 1.449, "other_containers_cpu": 0.24, "mem_available_gib": 109.85, "pkg_temp_c": 62.0, "root_avail_gb": 213.1} +{"ts": 1794029567.18, "seconds": 5.007, "load1": 9.3, "busy_p": 8.063, "busy_e_free": 0.897, "busy_e_load": 2.105, "busy_all": 11.064, "mine_cpu": 9.781, "foreign_cpu": 1.283, "other_containers_cpu": 0.155, "mem_available_gib": 109.78, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029572.184, "seconds": 5.004, "load1": 9.52, "busy_p": 8.112, "busy_e_free": 1.275, "busy_e_load": 2.264, "busy_all": 11.651, "mine_cpu": 10.138, "foreign_cpu": 1.513, "other_containers_cpu": 0.219, "mem_available_gib": 109.74, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029577.227, "seconds": 5.043, "load1": 9.72, "busy_p": 8.011, "busy_e_free": 0.88, "busy_e_load": 2.118, "busy_all": 11.009, "mine_cpu": 9.711, "foreign_cpu": 1.298, "other_containers_cpu": 0.15, "mem_available_gib": 109.69, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029582.23, "seconds": 5.004, "load1": 9.98, "busy_p": 8.126, "busy_e_free": 1.077, "busy_e_load": 2.242, "busy_all": 11.445, "mine_cpu": 9.786, "foreign_cpu": 1.659, "other_containers_cpu": 0.35, "mem_available_gib": 109.68, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029587.235, "seconds": 5.004, "load1": 10.06, "busy_p": 6.271, "busy_e_free": 0.903, "busy_e_load": 2.094, "busy_all": 9.268, "mine_cpu": 7.881, "foreign_cpu": 1.387, "other_containers_cpu": 0.228, "mem_available_gib": 109.62, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029592.238, "seconds": 5.003, "load1": 9.9, "busy_p": 5.621, "busy_e_free": 0.806, "busy_e_load": 2.095, "busy_all": 8.521, "mine_cpu": 7.279, "foreign_cpu": 1.242, "other_containers_cpu": 0.175, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029597.241, "seconds": 5.004, "load1": 9.98, "busy_p": 5.706, "busy_e_free": 0.751, "busy_e_load": 2.144, "busy_all": 8.602, "mine_cpu": 7.333, "foreign_cpu": 1.269, "other_containers_cpu": 0.109, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029602.245, "seconds": 5.004, "load1": 9.43, "busy_p": 5.67, "busy_e_free": 0.769, "busy_e_load": 2.118, "busy_all": 8.558, "mine_cpu": 7.339, "foreign_cpu": 1.219, "other_containers_cpu": 0.155, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029607.284, "seconds": 5.039, "load1": 9.55, "busy_p": 5.7, "busy_e_free": 0.742, "busy_e_load": 2.092, "busy_all": 8.533, "mine_cpu": 7.345, "foreign_cpu": 1.188, "other_containers_cpu": 0.108, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029612.288, "seconds": 5.004, "load1": 9.59, "busy_p": 6.35, "busy_e_free": 0.869, "busy_e_load": 2.27, "busy_all": 9.49, "mine_cpu": 8.112, "foreign_cpu": 1.378, "other_containers_cpu": 0.319, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029617.292, "seconds": 5.004, "load1": 9.86, "busy_p": 8.114, "busy_e_free": 0.941, "busy_e_load": 2.196, "busy_all": 11.251, "mine_cpu": 9.875, "foreign_cpu": 1.376, "other_containers_cpu": 0.221, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029622.297, "seconds": 5.004, "load1": 9.95, "busy_p": 8.127, "busy_e_free": 0.979, "busy_e_load": 2.268, "busy_all": 11.374, "mine_cpu": 9.83, "foreign_cpu": 1.544, "other_containers_cpu": 0.223, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029627.302, "seconds": 5.005, "load1": 9.96, "busy_p": 8.066, "busy_e_free": 0.925, "busy_e_load": 2.152, "busy_all": 11.143, "mine_cpu": 9.784, "foreign_cpu": 1.359, "other_containers_cpu": 0.179, "mem_available_gib": 109.64, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029632.306, "seconds": 5.005, "load1": 9.96, "busy_p": 8.052, "busy_e_free": 1.029, "busy_e_load": 2.198, "busy_all": 11.279, "mine_cpu": 9.827, "foreign_cpu": 1.452, "other_containers_cpu": 0.202, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029637.35, "seconds": 5.044, "load1": 10.2, "busy_p": 8.05, "busy_e_free": 0.96, "busy_e_load": 2.185, "busy_all": 11.194, "mine_cpu": 9.852, "foreign_cpu": 1.342, "other_containers_cpu": 0.152, "mem_available_gib": 109.62, "pkg_temp_c": 62.0, "root_avail_gb": 213.1} +{"ts": 1794029642.355, "seconds": 5.005, "load1": 9.63, "busy_p": 3.936, "busy_e_free": 0.889, "busy_e_load": 2.146, "busy_all": 6.971, "mine_cpu": 5.582, "foreign_cpu": 1.389, "other_containers_cpu": 0.214, "mem_available_gib": 113.0, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029647.358, "seconds": 5.003, "load1": 9.1, "busy_p": 1.527, "busy_e_free": 0.578, "busy_e_load": 2.021, "busy_all": 4.126, "mine_cpu": 3.326, "foreign_cpu": 0.8, "other_containers_cpu": 0.105, "mem_available_gib": 112.58, "pkg_temp_c": 63.0, "root_avail_gb": 213.7} +{"ts": 1794029652.36, "seconds": 5.002, "load1": 8.61, "busy_p": 0.692, "busy_e_free": 0.474, "busy_e_load": 1.985, "busy_all": 3.15, "mine_cpu": 2.404, "foreign_cpu": 0.746, "other_containers_cpu": 0.103, "mem_available_gib": 112.51, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029657.362, "seconds": 5.002, "load1": 8.0, "busy_p": 0.362, "busy_e_free": 0.464, "busy_e_load": 1.991, "busy_all": 2.817, "mine_cpu": 2.02, "foreign_cpu": 0.797, "other_containers_cpu": 0.063, "mem_available_gib": 112.51, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029662.365, "seconds": 5.002, "load1": 7.52, "busy_p": 2.553, "busy_e_free": 0.628, "busy_e_load": 2.053, "busy_all": 5.233, "mine_cpu": 4.243, "foreign_cpu": 0.99, "other_containers_cpu": 0.14, "mem_available_gib": 111.93, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029667.4, "seconds": 5.035, "load1": 7.72, "busy_p": 6.963, "busy_e_free": 0.808, "busy_e_load": 2.133, "busy_all": 9.905, "mine_cpu": 8.799, "foreign_cpu": 1.106, "other_containers_cpu": 0.11, "mem_available_gib": 110.02, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029672.403, "seconds": 5.003, "load1": 8.06, "busy_p": 6.874, "busy_e_free": 0.963, "busy_e_load": 2.169, "busy_all": 10.006, "mine_cpu": 8.595, "foreign_cpu": 1.411, "other_containers_cpu": 0.302, "mem_available_gib": 109.87, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029677.408, "seconds": 5.005, "load1": 8.06, "busy_p": 8.069, "busy_e_free": 1.023, "busy_e_load": 2.208, "busy_all": 11.3, "mine_cpu": 9.77, "foreign_cpu": 1.53, "other_containers_cpu": 0.243, "mem_available_gib": 109.81, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029682.413, "seconds": 5.005, "load1": 8.37, "busy_p": 8.06, "busy_e_free": 0.923, "busy_e_load": 2.21, "busy_all": 11.194, "mine_cpu": 9.775, "foreign_cpu": 1.419, "other_containers_cpu": 0.22, "mem_available_gib": 109.77, "pkg_temp_c": 64.0, "root_avail_gb": 213.4} +{"ts": 1794029687.418, "seconds": 5.005, "load1": 8.5, "busy_p": 8.068, "busy_e_free": 0.941, "busy_e_load": 2.138, "busy_all": 11.147, "mine_cpu": 9.758, "foreign_cpu": 1.389, "other_containers_cpu": 0.138, "mem_available_gib": 109.73, "pkg_temp_c": 64.0, "root_avail_gb": 213.3} +{"ts": 1794029692.423, "seconds": 5.005, "load1": 8.78, "busy_p": 8.16, "busy_e_free": 1.135, "busy_e_load": 2.268, "busy_all": 11.562, "mine_cpu": 10.111, "foreign_cpu": 1.451, "other_containers_cpu": 0.204, "mem_available_gib": 109.68, "pkg_temp_c": 65.0, "root_avail_gb": 213.2} +{"ts": 1794029697.466, "seconds": 5.043, "load1": 8.32, "busy_p": 8.011, "busy_e_free": 0.817, "busy_e_load": 2.205, "busy_all": 11.033, "mine_cpu": 9.738, "foreign_cpu": 1.295, "other_containers_cpu": 0.152, "mem_available_gib": 109.64, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029702.472, "seconds": 5.005, "load1": 8.61, "busy_p": 6.387, "busy_e_free": 0.957, "busy_e_load": 2.134, "busy_all": 9.478, "mine_cpu": 8.09, "foreign_cpu": 1.388, "other_containers_cpu": 0.347, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029707.478, "seconds": 5.006, "load1": 8.73, "busy_p": 8.08, "busy_e_free": 1.107, "busy_e_load": 2.243, "busy_all": 11.43, "mine_cpu": 9.719, "foreign_cpu": 1.711, "other_containers_cpu": 0.251, "mem_available_gib": 109.63, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029712.484, "seconds": 5.006, "load1": 8.91, "busy_p": 8.075, "busy_e_free": 1.071, "busy_e_load": 2.261, "busy_all": 11.407, "mine_cpu": 9.843, "foreign_cpu": 1.564, "other_containers_cpu": 0.27, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029717.49, "seconds": 5.007, "load1": 9.16, "busy_p": 8.073, "busy_e_free": 1.077, "busy_e_load": 2.211, "busy_all": 11.361, "mine_cpu": 9.918, "foreign_cpu": 1.443, "other_containers_cpu": 0.191, "mem_available_gib": 109.6, "pkg_temp_c": 64.0, "root_avail_gb": 213.1} +{"ts": 1794029722.496, "seconds": 5.005, "load1": 9.54, "busy_p": 8.123, "busy_e_free": 0.993, "busy_e_load": 2.226, "busy_all": 11.342, "mine_cpu": 9.822, "foreign_cpu": 1.52, "other_containers_cpu": 0.216, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029727.539, "seconds": 5.043, "load1": 9.98, "busy_p": 8.094, "busy_e_free": 0.944, "busy_e_load": 2.171, "busy_all": 11.208, "mine_cpu": 9.776, "foreign_cpu": 1.432, "other_containers_cpu": 0.155, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029732.544, "seconds": 5.005, "load1": 10.14, "busy_p": 8.072, "busy_e_free": 1.007, "busy_e_load": 2.234, "busy_all": 11.313, "mine_cpu": 9.784, "foreign_cpu": 1.529, "other_containers_cpu": 0.216, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029737.549, "seconds": 5.005, "load1": 9.73, "busy_p": 6.398, "busy_e_free": 0.975, "busy_e_load": 2.12, "busy_all": 9.493, "mine_cpu": 8.122, "foreign_cpu": 1.371, "other_containers_cpu": 0.277, "mem_available_gib": 109.63, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029742.554, "seconds": 5.005, "load1": 10.55, "busy_p": 8.103, "busy_e_free": 0.885, "busy_e_load": 2.268, "busy_all": 11.256, "mine_cpu": 9.781, "foreign_cpu": 1.475, "other_containers_cpu": 0.253, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029747.558, "seconds": 5.004, "load1": 10.59, "busy_p": 8.131, "busy_e_free": 0.941, "busy_e_load": 2.186, "busy_all": 11.259, "mine_cpu": 9.805, "foreign_cpu": 1.454, "other_containers_cpu": 0.18, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029752.563, "seconds": 5.005, "load1": 10.62, "busy_p": 8.044, "busy_e_free": 0.965, "busy_e_load": 2.196, "busy_all": 11.206, "mine_cpu": 9.821, "foreign_cpu": 1.385, "other_containers_cpu": 0.168, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029757.605, "seconds": 5.042, "load1": 10.73, "busy_p": 7.989, "busy_e_free": 0.932, "busy_e_load": 2.138, "busy_all": 11.059, "mine_cpu": 9.765, "foreign_cpu": 1.294, "other_containers_cpu": 0.177, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029762.609, "seconds": 5.005, "load1": 10.67, "busy_p": 8.106, "busy_e_free": 0.959, "busy_e_load": 2.258, "busy_all": 11.323, "mine_cpu": 9.761, "foreign_cpu": 1.562, "other_containers_cpu": 0.245, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029767.615, "seconds": 5.006, "load1": 10.7, "busy_p": 6.584, "busy_e_free": 1.043, "busy_e_load": 2.175, "busy_all": 9.803, "mine_cpu": 8.19, "foreign_cpu": 1.613, "other_containers_cpu": 0.374, "mem_available_gib": 109.62, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029772.618, "seconds": 5.002, "load1": 10.64, "busy_p": 7.876, "busy_e_free": 0.952, "busy_e_load": 2.209, "busy_all": 11.037, "mine_cpu": 9.592, "foreign_cpu": 1.445, "other_containers_cpu": 0.209, "mem_available_gib": 109.61, "pkg_temp_c": 64.0, "root_avail_gb": 213.2} +{"ts": 1794029777.623, "seconds": 5.006, "load1": 10.51, "busy_p": 8.023, "busy_e_free": 0.953, "busy_e_load": 2.215, "busy_all": 11.191, "mine_cpu": 9.712, "foreign_cpu": 1.479, "other_containers_cpu": 0.229, "mem_available_gib": 109.6, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029782.628, "seconds": 5.005, "load1": 10.63, "busy_p": 8.084, "busy_e_free": 0.831, "busy_e_load": 2.164, "busy_all": 11.079, "mine_cpu": 9.76, "foreign_cpu": 1.319, "other_containers_cpu": 0.14, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.3} +{"ts": 1794029787.671, "seconds": 5.042, "load1": 10.5, "busy_p": 8.084, "busy_e_free": 0.982, "busy_e_load": 2.239, "busy_all": 11.304, "mine_cpu": 9.805, "foreign_cpu": 1.499, "other_containers_cpu": 0.217, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.2} +{"ts": 1794029792.676, "seconds": 5.005, "load1": 10.54, "busy_p": 8.129, "busy_e_free": 0.879, "busy_e_load": 2.222, "busy_all": 11.23, "mine_cpu": 9.783, "foreign_cpu": 1.447, "other_containers_cpu": 0.178, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029797.681, "seconds": 5.005, "load1": 10.74, "busy_p": 8.162, "busy_e_free": 1.077, "busy_e_load": 2.238, "busy_all": 11.477, "mine_cpu": 9.795, "foreign_cpu": 1.682, "other_containers_cpu": 0.353, "mem_available_gib": 109.65, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029802.686, "seconds": 5.005, "load1": 10.68, "busy_p": 6.452, "busy_e_free": 0.823, "busy_e_load": 2.142, "busy_all": 9.417, "mine_cpu": 8.121, "foreign_cpu": 1.296, "other_containers_cpu": 0.202, "mem_available_gib": 109.61, "pkg_temp_c": 63.0, "root_avail_gb": 213.1} +{"ts": 1794029807.69, "seconds": 5.004, "load1": 10.79, "busy_p": 8.053, "busy_e_free": 0.925, "busy_e_load": 2.118, "busy_all": 11.097, "mine_cpu": 9.739, "foreign_cpu": 1.358, "other_containers_cpu": 0.177, "mem_available_gib": 109.6, "pkg_temp_c": 63.0, "root_avail_gb": 213.4} +{"ts": 1794029812.694, "seconds": 5.004, "load1": 10.8, "busy_p": 6.67, "busy_e_free": 1.149, "busy_e_load": 2.262, "busy_all": 10.082, "mine_cpu": 1.942, "foreign_cpu": 8.14, "other_containers_cpu": 0.138, "mem_available_gib": 114.01, "pkg_temp_c": 64.0, "root_avail_gb": 213.6} +{"ts": 1794029817.716, "seconds": 5.021, "load1": 10.18, "busy_p": 0.498, "busy_e_free": 0.225, "busy_e_load": 0.504, "busy_all": 1.227, "mine_cpu": 0.0, "foreign_cpu": 1.227, "other_containers_cpu": 0.065, "mem_available_gib": 114.41, "pkg_temp_c": 72.0, "root_avail_gb": 222.0} diff --git a/bench/wildct/results/2026-11-07/runs/H25w/wait_s b/bench/wildct/results/2026-11-07/runs/H25w/wait_s new file mode 100644 index 00000000..573541ac --- /dev/null +++ b/bench/wildct/results/2026-11-07/runs/H25w/wait_s @@ -0,0 +1 @@ +0 diff --git a/bench/wildct/results/2026-11-07/runs/campaign.log b/bench/wildct/results/2026-11-07/runs/campaign.log index 896b5dfa..8930ab37 100644 --- a/bench/wildct/results/2026-11-07/runs/campaign.log +++ b/bench/wildct/results/2026-11-07/runs/campaign.log @@ -29,3 +29,7 @@ == H18w lock taken 2026-11-07T04:22:23Z == H18w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h18w 2026-11-07T04:22:24Z == H18w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T04:26:30Z +== H25w waiting for the lock 2026-11-07T05:18:20Z +== H25w lock taken 2026-11-07T05:18:20Z +== H25w start: bash /home/andrew/k3dgpu-bench-wildct/repo/bench/wildct/row.sh h25w 2026-11-07T05:18:20Z +== H25w rc=0; clusters deleted: none; now: ; / 207 GB available 2026-11-07T05:36:58Z diff --git a/docs/intel/partner-update-2026-11.md b/docs/intel/partner-update-2026-11.md index ad2804c8..4e4d8fcb 100644 --- a/docs/intel/partner-update-2026-11.md +++ b/docs/intel/partner-update-2026-11.md @@ -58,8 +58,8 @@ All drafts are unsent and need a maintainer's approval before anything is filed. SUPPORTED rows, each scoped to its host: - On the Xeon 6741P: the CPU engine choice [ledger:H59], DRAM-resident serving of Qwen3.8-Flash-Next [ledger:H60], and H25 replicated [ledger:H25x]. -- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-and-h18w-to-wildcts-i7-13700). -- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w], and H18 replicated, agent sandboxes on its E-cores behind default-deny egress [ledger:H18w]. +- On the i9-13900T: sixteen rows, among them EAT `release-1.0.3-hotfix` deployed and served on a default k3d-gpu cluster, n=1 [ledger:H1b], EAT's NRI balloons chart installing over a k3d-gpu cluster [ledger:H14], and agent sandboxes behind default-deny egress [ledger:H18]. The full list is in the [technical report](technical-report.md#supported-quotable-scoped-to-the-i9-13900t-h25x-h59-and-h60-to-the-xeon-6741p-h52w-h18w-and-h25w-to-wildcts-i7-13700). +- On wildct, an i7-13700 client host: detection of its hybrid cores and ISA extensions with its GPUs opted out [ledger:H52w], H18 replicated, agent sandboxes on its E-cores behind default-deny egress [ledger:H18w], and H25 replicated, an added agent's first chunk 14.3x sooner with OVMS's shipped prefix caching on [ledger:H25w]. ## Two results that help EAT @@ -73,7 +73,7 @@ v0.28.0 is past the v0.24.0 CPU attention hang that stalled EAT 1.0.3's stock ar - **The Keycloak argv secret.** F3 stays REFUTED. `k3d-gpu eas up` now patches enterprise-inference's `keycloak_config` role so that neither the admin password nor the client secret is a `kcadm.sh` argument ([State and secrets](eas.md#state-and-secrets)). The patch is checked with a `ps` sampler against a stand-in for kcadm; no live run has checked it yet. A fresh `eas up --agent-toolkit` with F3's secret scan would. - **Intel's default install in a guest (H62).** A re-run needs the `cpu: auto` sizing to leave room for EAT's sandbox router, or a guest shape where it does. - **k3d-gpu gaps found on the Xeon.** `/dev/dri/by-path` for oneCCL across B70s (H53); the fit refusing rather than falling back to the CPU, and no DRAM bandwidth cached on a new host (H58); placing one-card arms on one card (H54b, H54c). -- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H18w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. +- **Replications.** H59 on the i9-13900T (H59d) is INCONCLUSIVE: llama.cpp's arm was OOMKilled at the fit's pod limit and not deliverable in two holds, so no statistic was scored. H59 on the i7-13700 (H59w) is PENDING, as are the wildct client-host rows after H25w. Until a row is SUPPORTED for its class, those CPU classes keep OVMS. - **EAT 1.1.0 through `eat-up.sh`.** Pinned since 2026-10-29 and pre-registered as F4; not run yet. - **EAS upstream moves.** EAS #61 merged on 2026-10-30 (EAS `main` `0f7bc2b`). The pin stays at `97f6157`; moving past #61 needs patch 0002 regenerated. EAS #50 and #52 (OpenShift) are open and would move 0006. EAS #71 (teardown hang) is open; `eas down` uses its workaround. - **ERAG and Arc under EAS.** EAS's Enterprise RAG layer is not wired into `k3d-gpu eas` yet, and EAS has no Intel GPU path on this branch; Arc and Gaudi stay with `k3d-gpu agent up`. diff --git a/docs/intel/technical-report.md b/docs/intel/technical-report.md index 92b12015..0f56dc15 100644 --- a/docs/intel/technical-report.md +++ b/docs/intel/technical-report.md @@ -1,17 +1,17 @@ # k3d-gpu on Intel hardware: technical report -Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d, H52w and H18w runs (82 rows). +Status: working report, 2026-11-01. Ledger: [EXPERIMENTS.md](../../EXPERIMENTS.md), verdicts as of the H57dp, H54q, H53, H62, H58, H59, H60, H55, H59d, H52w, H18w and H25w runs (82 rows). ## Abstract k3d-gpu turns one Linux host into a single-node k3s cluster (k3d) that detects the host's Intel hardware and configures it. It sets NRI balloons for hybrid P/E and NUMA CPUs, device plugins for iGPU, dGPU, NPU, DSA, IAA, QAT and Gaudi, and placement labels for agent and inference pods. -This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w and H18w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. +This report covers what was measured, and how. Every result but H52, H52b, H7x, H25x, H27x, H28x, H34x, H50, H50o, H51, H53, H54a, H54d, H54q, H54b, H54c, H56, H58, H59, H60 and H62 (a Xeon 6741P host, the B70 rows on its Arc Pro B70s) and H52w, H18w and H25w (wildct, an i7-13700 client host) comes from one host: an i9-13900T (Raptor Lake-S, 8 P-cores and 16 E-cores, UHD 770 iGPU with no XMX, AVX-VNNI, no AMX, no NPU). It is a T-SKU rated at 35 W base power, but its firmware runs PL1 = PL2 = 117 W, so it does not run the T-series 35 W envelope. -- 21 hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w and H18w to wildct, an i7-13700. H18w is the newest: H18 replicated on that host. In 3 of 3 runs `k3d-gpu sandbox up` from the release kept the sandbox on an E-core (CPU 17) behind default-deny egress: the host's tailscale and LAN addresses were refused from the sandbox and reached from a control pod, and `pip install --user` from PyPI worked [ledger:H18w]. Before it, H52w: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. +- 22 hypotheses are SUPPORTED. Sixteen are scoped to that host; H25x (H25 replicated), H59 and H60 are scoped to the Xeon 6741P, and H52w, H18w and H25w to wildct, an i7-13700. H25w is the newest: H25 replicated on that host. With OVMS's shipped prefix caching on, an added agent's first chunk arrived 14.3x sooner than with it off (2.20 s vs 31.44 s, CI 13.38-14.89) under matched `typical` load; one of the six rounds was cut by the level's time cap [ledger:H25w]. Before it, H18w: H18 replicated on that host. In 3 of 3 runs `k3d-gpu sandbox up` from the release kept the sandbox on an E-core (CPU 17) behind default-deny egress: the host's tailscale and LAN addresses were refused from the sandbox and reached from a control pod, and `pip install --user` from PyPI worked [ledger:H18w]. Before it, H52w: on that client Raptor Lake host, with the box owner's opt-out of its two NVIDIA cards and its iGPU, a plain `k3d-gpu up` found the hybrid P/E cores and the six ISA extensions the launcher labels, and nothing the host lacks. It named both GPU vendors as detected and opted out, and no pod mounted or asked for a GPU [ledger:H52w]. Before it, H60: Qwen3.8-Flash-Next, served from the Xeon's DRAM by llama.cpp v0.8.0, decoded 16.0 tok/s at one stream and read 0.82 of a same-hold Gemma 4 QAT control's share of the DRAM bandwidth ceiling (CI 0.819-0.831, band 0.7-1.3), with GSM8K 50/50 [ledger:H60]. Before it, H59: on the Xeon 6741P at four concurrent requests, llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter decoded Gemma 4 26B-A4B 3.50x faster per request than OVMS with OpenVINO's base-model int4 IR (95% CI 3.42-3.61), faster in each of 3 holds, at GSM8K 49 against 50 of 50. So with no backend set, `agent up` now serves llama.cpp on the CPU of that class [ledger:H59]. - Twenty are REFUTED. Six of them changed shipped defaults. The newest, H28x, is H28 replicated on the Xeon 6741P: eight pods that mmap one 1 GiB model through an image volume again shared one page-cache copy (PSS 132.6 MiB per pod, 1.00x the size on the node against 9.00x for per-pod copies), but they were ready in 0.714 of the copy arm's time, not the 0.5 or less the row needed, so only the memory part replicated [ledger:H28x]. Before it, H7x is H7 run to its full protocol on the Xeon 6741P: at matched 16-bit weights (Qwen3-4B, 16 CPUs), OVMS 2026.4.1 decoded at 1.22x the rate of EAT 1.1.0's stock vLLM CPU engine (v0.28.0) in all five block pairs, so OVMS's decode lead is not only weight precision; vLLM prefilled the long prompt faster [ledger:H7x]. Before it, H58 is the zero-touch thesis on the Xeon 6741P, with the two serving pairs holding about half of every B70. With no flag, `k3d-gpu up` advertised three of the four B70s and no DSA work queue. `k3d-gpu agent up --model` then refused all three models. Gemma 4 26B-A4B and Qwen3.8-27B went to OVMS on the GPU, where their int4 IRs missed the best card's free-memory budget by 0.1 and 0.5 GiB. Qwen3.8-Flash-Next was refused because no DRAM bandwidth was cached. So EAT's banking flow never came up [ledger:H58]. Before it, H52b is H52 after `k3d-gpu host-setup --apply` on the Xeon 6741P: DSA was advertised and reached a pod, but IAA got no work queue, the QAT plugin advertised none of the VFs, and `up` advertised two of the four B70s: the serving pairs hold about half the VRAM of every card, and its busy-card filter (half or more) left out the two just over half and kept the two just under it [ledger:H52b]. Before it, H51 ran the graded agents-per-box ramp on the Xeon 6741P: the k3d-gpu CPU stack failed at one agent (turn p95 10.41 s against the 10 s SLO), and so did EAT's stock vLLM CPU arm [ledger:H51]. H50 ran inside EAT's banking flow on the same host: EAT's current stock vLLM CPU arm (release-1.1.0) delivered at least as many tasks within the turn budget as the k3d-gpu CPU stack: 21 of 24 against 19 of 24 [ledger:H50]. - 34 are INCONCLUSIVE, mostly because the shared host throttles or its load was unmatched. H62 (the k3d-gpu CPU stack against Intel's new default, EAS v1.2.0-rc.1 installing EAT 2.0, each in a KVM guest on the Xeon 6741P) is INCONCLUSIVE because Intel's own install did not finish: enterprise-inference's `cpu: auto` sized the Gemma 4 pod to 76 of the guest's 95.4 allocatable CPUs, so EAT's sandbox router could not be scheduled, twice, and no hold ran. H54q (the QAT-matched int4 IR of Gemma 4 26B-A4B against the bf16 QAT weights and OpenVINO's base-model IR, on GSM8K items 0-199 on the Xeon 6741P) is INCONCLUSIVE: the QAT-matched IR stayed inside the pre-registered margin of the bf16 reference, but the base-model IR scored no lower than it, with every build near the item set's ceiling. H50o (the k3d-gpu CPU stack against EAT's own OVMS chart on the Xeon 6741P) is INCONCLUSIVE because that chart's OVMS 2025.4 crashed on loading Gemma 4 26B-A4B, so no pair ran. H54a and H54d (Gemma 4 26B-A4B QAT on the Xeon 6741P) are INCONCLUSIVE because the stock arm, EAT 1.0.3's vLLM CPU v0.24.0, as this harness rendered EAT's entry (no CPU balloon, requests or limits), stalled in both of H54a's holds, so no arm could be scored against it. EAT 1.1.0's vLLM CPU v0.28.0 was not measured in H54a; H50 ran it in EAT's banking flow. H54b and H54c (one Arc Pro B70 against two, and experts in DRAM with the rest on one B70) are INCONCLUSIVE because their "one card" pods saw all four cards, so no arm was delivered as pre-registered. H53 (EAT's banking flow on the B70s: k3d-gpu's one-card OVMS pick against EAT 1.1.0's vLLM XPU path across all four cards) is INCONCLUSIVE because the EAT arm did not start in a k3d-gpu cluster: oneCCL needs `/dev/dri/by-path` for its cross-card collective, and the k3d node has none, so no hold ran. H57da (k3d-gpu's OVMS pick vs llama.cpp for Gemma 4 QAT on the i9-13900T) did not resolve: the strictly pinned llama.cpp arms tracked host load; H57db did not run (memory). H57dp and H57de are INCONCLUSIVE: they re-ran that comparison with an unpinned llama.cpp arm, and the load gate failed on OVMS's low foreign P-core load (0.45 CPU-s/s against the llama.cpp arms' 3.1-3.4); the unpinned arm was 1.108x the pinned one and 1.369x OVMS, not gated, and OVMS ran without its P-core balloon. H55 (SLO-gated admission of an E-core neighbour on the i9-13900T) is INCONCLUSIVE: all 25 runs completed on a busy host, but in one round the controller's run closed three request windows, one short of its gate; not gated, the controller held the neighbour at its lowest quota in four of five rounds, and no arm kept prefill within the row's bar of quiet. H59d (the engine choice on the i9-13900T: llama.cpp with Google's QAT Q4_0 GGUF and the QAT MTP drafter against OVMS with OpenVINO's base-model int4 IR) is INCONCLUSIVE because the llama.cpp arm was not deliverable in two of its four holds: in both holds where it followed OVMS, its server was OOMKilled at the fit's 45Gi pod limit (4 x 262,144 tokens), while the same render served in both holds where it ran first. Not gated, in the one performance hold it served it decoded 1.05x OVMS per request at four concurrent requests, and GSM8K read 50 against 49 of 50. So with no backend set, the i9-13900T keeps OVMS. H56 (two holds at once in cpuset slices of the Xeon 6741P) is INCONCLUSIVE: in every hold with a sibling, the sibling's banking block ended shortly after the measured block began, so all four were VOID and no interference statistic was scored; crescent rows stay at one hold at a time. -- 7 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H25w, H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. +- 6 are PENDING: the engine-choice row H59w (i7-13700), and the second client host's (wildct, i7-13700) rows H28w, H34w, H57wa, H57wm and H57wb, pre-registered and not yet run. H52, the first row run on that host (a Xeon 6741P with four Arc Pro B70s), is REFUTED. H27x, H27 re-run on the Xeon (AMX), is REFUTED as on the i9. H34x, H34 re-run there, is REFUTED: the CPU Fast Draft that sped up the i9 slowed decode on the Xeon. - Every other Intel family is detected and configured in fixture tests only. None has a measurement. No world-first, superlative or priority claim is made; [docs/provenance.md](../provenance.md) supports none. @@ -48,7 +48,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p ## Results -### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w and H18w to wildct's i7-13700) +### SUPPORTED (quotable, scoped to the i9-13900T; H25x, H59 and H60 to the Xeon 6741P; H52w, H18w and H25w to wildct's i7-13700) | Row | Result | Scope limit | |---|---|---| @@ -72,6 +72,7 @@ The host is a shared, live machine: about 53 other containers, about 80 W idle p | H40 | The agent harness reproduces the BFCL board: bf16 Qwen3-4B-Instruct-2507 on vLLM CUDA scored 88.0% on the fixed 100-id subset in BFCL's own mode (board-weighted 84.9%, band 77.9-91.9%), every category inside its binomial interval [ledger:H40] | A check of the harness, not of the hardware; one 100-id subset, one run, on the NVIDIA dGPU | | H52w | On wildct (an i7-13700 with two NVIDIA cards and a UHD 770 iGPU), a plain `k3d-gpu up` with the box owner's GPU opt-out (`K3D_GPU_VENDORS=auto,-nvidia,-intel`) detected every CPU engine present and nothing phantom: the hybrid P/E cores with the balloons policy (`k3d-gpu test`'s P-core pod on CPU 0, its E-core pod on CPU 17) and the launcher's six ISA labels (AVXVNNI, VAES, GFNI, SHA, CETIBT, CETSS); no AVX-512, AMX or NUMA label, and no DSA, IAA, QAT, NPU or Gaudi resource or pod. Both GPU vendors were named as detected and opted out, no GPU resource was allocatable and no pod mounted a `/dev` path [ledger:H52w] | One hold, release 0.1.89. Detection with the GPUs opted out: says nothing about a GPU reaching a pod on this host | | H18w | H18 replicated on wildct (an i7-13700, E-cores 16-23) with the release node image and the box owner's GPU opt-out: in 3 of 3 runs `k3d-gpu sandbox up` exited 0 with the sandbox pod on CPU 17; TCP from the sandbox to the host's tailscale (`100.66.38.121:22`) and LAN (`192.168.8.238:22`) addresses was refused, with 2 packets on kube-router's drop-log rule, while a control pod without the policy connected to both; `pip install --user tomli_w==1.2.0` from PyPI installed into `/tmp/.local`, where the import found it [ledger:H18w] | One hold, release 0.1.89, agent-sandbox v1.0.8, kube-router on k3s v1.37.1, IPv4 targets only | +| H25w | H25 replicated on wildct (an i7-13700, OVMS on its 16 P-core CPUs, the 0.6B CPU Fast Draft in both arms) with the release node image and the box owner's GPU opt-out: with prefix caching on, an added agent's first chunk arrived 14.3x sooner than with it off (2.198 s vs 31.437 s, CI 13.38-14.89) under matched `typical` load on E-cores 20-23 [ledger:H25w] | One hold, release 0.1.89, one prefix shape; memory per agent not measured. OFF has 2 rounds (n = 10): the level's time cap cut the third. Both arms ran 2.2-2.9x slower than on the i9, the package at its 35 W PL1 | ### REFUTED (all but H5, H7x, H27, H27x, H28x, H34x, H43, H45, H48, H50, H51 and H52b changed a default)