A declarative, reproducible home-and-lab NixOS fleet — nine machines, one k3s cluster, full-disk secrets, GitOps-deployed workloads, and a unified HTTPS service mesh over tailscale — all from a single flake.
sudo nixos-rebuild switch --flake .#<host>rebuilds any machine.nix run .#nixidy -- switch .#prodreconciles the entire cluster.
Every host — from the k3s server de-msa2, to the GPU box desg0, to the
laptops razerblade / tensorbook — is defined in hosts/<name>/ and built
from the same flake. Add a machine: drop a directory, add one line to
flake.nix, rebuild.
| Host | Role |
|---|---|
de-msa2 |
k3s server + monitoring/alerting + git forge + self-hosted apps |
desg0 |
GPU node (remote builder, LLM serving) |
meshify / superserver / poweredge / de-n5 |
servers & nodes |
razerblade / tensorbook |
laptops (Hyprland desktop) |
A self-hosted k3s cluster whose workloads are rendered by
nixidy from Nix (env/*.nix) into YAML
manifests (manifests/prod/), then continuously reconciled by ArgoCD with
auto-sync, prune and self-heal. The cluster manages itself: ArgoCD and
cert-manager are declared as nixidy apps and bootstrapped through GitOps.
env/argocd.nix— ArgoCD, exposed athttps://argocd.k3s.lanenv/cert_manager.nix— a self-signed root CA (k3s-lan-ca) generated in-cluster, its public cert trusted fleet-wide, so every*.k3s.lanservice has a browser-trusted TLS cert with zero manual trust stepsenv/host_ingress.nix— fronts host-local NixOS services (ntfy, forgejo, grafana, vikunja) at*.k3s.lanvia traefik + cert-managerenv/homepage.nix— homepage dashboard athttps://home.k3s.lan, auto-discovering every annotated*.k3s.lanIngress
hosts/de-msa2/alerting.nix + prometheus.nix declare the entire monitoring
pipeline, rules included, reproducibly:
victoriametrics → vmalert (rule eval) → alertmanager
→ alertmanager-ntfy bridge → ntfy-sh (push to phone)
Node, ZFS, NVIDIA-GPU and Kubernetes (kube-state-metrics) metrics are all
scraped. Alert rules are set to notify if things go haywire, so the stack fires
on NodeLoadHigh, ContainerCPUNearLimit, PodRestartLooping, OOMKilled
and more — paging the ntfy app at https://ntfy.k3s.lan/cluster-alerts.
In the ntfy web interface at https://ntfy.k3s.lan, manually subscribe to the
cluster-alerts topic. This is the only required subscription; it receives
production, development (prefixed [dev]), host, storage, and cluster alerts.
ntfy topics are created on first publish, so they are not listed automatically
in a new browser profile.
An explorable version with guided views is in
docs/diagrams/monitoring-alerting.html.
Host-specific secrets (k3s token, grafana secret key, …) live encrypted in
secrets/ and are decrypted at activation via
agenix. Plaintext never touches the repo.
modules/mullvad_tailscale.nix wires a Mullvad WireGuard exit node alongside
tailscale, with a deterministic DNS fallback via /etc/hosts so routing works
regardless of who owns resolv.conf. Every *.k3s.lan name resolves to a
node's tailscale IP through networking.hosts in modules/base_system.nix.
modules/ai/ declares a battery of local LLM serving options: vllm_cuda_container, tensorrt_llm_container, llama-cpp, a
hermes-agent runner, qwen_code, and a pi-agent harness. The
remote_builder.nix module turns GPU hosts into distributed nix builders over
SSH.
desg0 serves Qwen3.8-27B-NVFP4 with SGLang
(hosts/desg0/sglang_qwen3_container.nix). The concurrency limit
(--max-running-requests 12) comes from production metrics in
VictoriaMetrics, not from a synthetic benchmark. The data is 8,769 busy
1-minute windows since 2026-09-11, with a median prompt of ~47k tokens:
- Aggregate decode peaks at 8 concurrent requests (~464 tok/s).
- Decode plus prefill compute keeps rising to ~3.3k tok/s at 12.
- From 12 to 13 there is a cliff. Inter-token latency goes from 30 to 73 ms, median TTFT goes from 7 to 19 s, and decode throughput halves. The cause is the ~50k-token prefills, which take the place of decode steps in the batch. The KV pool is only ~60% used at this point, so compute is the limit, not memory.
- Above 13, total throughput is almost all prefill (~4.5k tok/s ceiling). The streams then get only 4-9 tok/s each.
meshify runs a llama.cpp router (modules/ai/llama-cpp.nix) on its 24GB
RTX 3090, which also drives the desktop (~3GB). The router has a 200k-token
q8_0 KV pool (--ctx-size 204800 --kv-unified) shared by 4 slots
(--parallel 4): median agent prompts are ~50k tokens, so 4 fit at once, and
one request can still use the whole pool. hosts/meshify/constants.nix
enables only the models that hold this pool fully in VRAM. The other models
stay in the list as comments, for a bigger GPU. The concurrency sweep below used llama-batched-bench (q8_0 KV cache,
flash-attn, all layers on the GPU) with an 8,192-token prompt and 128
generated tokens per user (2026-10-07). The dashed lines are the five models
added after a Hugging Face search:
Generation tok/s, aggregate over all users (per user in parentheses), and prompt processing tok/s:
| Model | 1 | 2 | 4 | 8 | 16 | Prompt |
|---|---|---|---|---|---|---|
| Ling-3.0-tiny UD-Q8_K_XL (MoE) | 149 | 232 (116) | 460 (115) | 652 (82) | 832 (52) | ~5.5k |
| gemma-4-26B-A4B Q4_K_M (MoE) | 110 | 188 (94) | 261 (65) | 344 (43) | 518 (32) | ~4.3k |
| gemma-4-26B-A4B QAT UD-Q4_K_XL (MoE) [new] | 122 | 219 (109) | 290 (72) | 360 (45) | 514 (32) | ~4.3k |
| Qwen-AgentWorld-35B-A3B UD-IQ4_XS (MoE) [new] | 125 | 182 (91) | 258 (65) | 314 (39) | 337 (21) | ~3.2k |
| North-Mini-Code-1.0 UD-IQ4_XS (MoE) [new] | 118 | 171 (86) | 209 (52) | 224 (28) | 302 (19) | ~3.6k |
| Ornith-1.5-9B Q8_0 [new] | 71 | 125 (62) | 207 (52) | 292 (37) | 415 (26) | ~3.7k |
| gemma-4-12b UD-Q8_K_XL | 48 | 89 (44) | 153 (38) | 231 (29) | 344 (22) | ~2.9k |
| Qwen3.8-27B GSQ-RCO IQ3_S [new] | 39 | 58 (29) | 83 (21) | 104 (13) | 155 (10) | ~1.2k |
| Muse-Glimmer-30B Q4_K_XL | 37 | 67 (33) | 88 (22) | 108 (13) | 233 (15) | ~1.3k |
| Qwen3.6-27B Q4_K_XL | 32 | 56 (28) | 67 (17) | 97 (12) | OOM | ~1.2k |
Why the new models were picked (scores are from the model cards, not checked here):
- Qwen3.8-27B is the model that desg0 serves. Its card reports SWE-bench Pro
61.7 and Terminal-Bench 2.1 73.0, against 53.5 / 63.4 for Qwen3.6-27B and
51.2 / 51.7 for Muse-Glimmer-30B. The ISTA-DASLab GSQ-RCO IQ3_S quant
(3.5 bpw) reports BF16-level scores (GPQA-D 89.4 vs 89.9, LiveCodeBench v6
85.7 vs 85.7). The
IQ3_Stag resolves to the-mtpfile, which also has the MTP head for speculative decoding. - Qwen-AgentWorld-35B-A3B scores 56.4 overall on Qwen's agent suite (MCP, search, terminal, SWE, Android, web, OS), against 56.0 for Claude Sonnet 4.6 and 47.7 for Qwen3.5-35B-A3B.
- Ornith-1.5-9B reports SWE-bench Verified 70.6 and Terminal-Bench 2.1 46.2, close to Qwen3.6-35B-A3B (73.4 / 52.5) at a quarter of the size.
- North-Mini-Code-1.0 is Cohere's 30B-A3B model for code and agentic software engineering (256k context). Its card shows the scores only as an image.
- The gemma-4-26B-A4B QAT build is Google's quantization-aware-trained checkpoint. It is 2.5GiB smaller and 11% faster for one user than the Q4_K_M build of the same model.
Findings:
- gemma-4-26B-A4B (QAT or Q4_K_M) is the best fit for many users: ~515 tok/s aggregate and 32 tok/s per user at 16 users. Ling-3.0-tiny is faster, but it is a much smaller model.
- For one user, the 3B-4B-active MoE models give 118-125 tok/s. Qwen-AgentWorld and North-Mini-Code scale less well: their aggregate speed is almost flat from 8 to 16 users.
- Qwen3.8-27B at 3.5 bpw is 20% faster than Qwen3.6-27B at 4.5 bpw (39 vs 32 tok/s) and, with smaller weights, it fits 16 users. Qwen3.6-27B runs out of memory at 16 users (133k tokens of context).
- Prompt processing speed does not increase with more users. The time to read all prompts thus grows linearly. The dense 27B-30B models read only ~1.2-1.3k tok/s, so with 8 users the last user waits ~55 s for the first token, and with 16 users ~100 s.
- With one user, generation speed for 8k prompts is within 15% of the speed for 512-token prompts.
- The router runs 4 slots, so the 8-16 user columns need more slots than it
has; requests above 4 wait in the queue.
kevis disabled on meshify because it lazy-loads ~5.2GB onto the same GPU. - This measures the llama.cpp engine directly, not the HTTP server. The server also loads the vision projector.
- Not added: Ternary-Bonsai-2-27B needs the PrismML llama.cpp fork for its PQ2_0 / PTQ1_0 types. Xing4.0-29B-A4B and Cloudflare Clef use architectures that the packaged llama.cpp does not support yet. The gemma-4-31B QAT build (16.1GiB + 2.1GiB F32 mmproj) leaves no room for its sliding-window cache.
Long prompts: the same benchmark with the router's 200k unified KV pool
(-kvu -c 204800) and a 50,000-token prompt per user. "Wait" is the time to
read the prompts of all users (no prompt cache, so the cold-start worst
case). "Spare" is the free VRAM left at 4 users after the vision projector:
| Model | Wait, 1 user | tok/s, 1 user | tok/s per user, 2 | Wait, 4 users | tok/s per user, 4 | Spare |
|---|---|---|---|---|---|---|
| Ling-3.0-tiny | 13 s | 141 | 109 | 117 s | 73 | 9.1GB |
| gemma-4-26B-A4B QAT | 15 s | 100 | 59 | 125 s | 36 | 1.4GB |
| Ornith-1.5-9B | 15 s | 69 | 42 | 97 s | 26 | 7.3GB |
| gemma-4-26B-A4B Q4_K_M | 16 s | 85 | 55 | 130 s | 34 | none |
| North-Mini-Code-1.0 | 19 s | 73 | 55 | 148 s | 31 | 2.3GB |
| Qwen-AgentWorld-35B-A3B | 19 s | 96 | 63 | 116 s | 36 | 1.5GB |
| gemma-4-12b | 25 s | 39 | 33 | 201 s | 23 | 4.6GB |
| Muse-Glimmer-30B | 42 s | 32 | 30 | 249 s | 19 | 0.7GB |
| Qwen3.8-27B GSQ-RCO IQ3_S | 53 s | 29 | 18 | 357 s | 11 | 0.5GB |
| Qwen3.6-27B Q4_K_XL | OOM |
- Qwen3.6-27B is disabled: its 200k pool (6.8GB) does not fit next to the weights. gemma-4-26B-A4B Q4_K_M is disabled too: it fits only without its vision projector, and the QAT build of the same model replaces it.
- Ling-3.0-tiny and Muse-Glimmer-30B were trained for 128k context only, so requests beyond that may lose quality.
- Compared to 8k prompts, generation for one user is 15-25% slower at 50k. gemma-4-26B-A4B QAT and Qwen-AgentWorld-35B-A3B are the best fit: ~15-19 s to the first token and ~100 tok/s. The dense 27B-30B models need ~45-55 s for one cold 50k prompt.
- Agent turns mostly reuse the previous prompt, and the server caches it, so a later turn reads only the new tokens.
The cluster runs google/ax as the task API on
top of Agent Substrate, which
runs each task in a gVisor sandbox. A task can be suspended into a snapshot
and resumed later. Both projects are pre-alpha: every image is built with Nix
and pinned by digest. Build notes and open items are in
docs/ax_stack_todo.md. The full guide to writing,
deploying and operating Tasks is docs/ax_usage.md.
How it fits together:
- Substrate (
env/substrate.nix,hosts/de-msa2/substrate.nix): API server, controller,ateletand the gVisor worker pool. Workers run only on the Zen-4 nodes (desg0,de-n5), because golden snapshots must not move between CPU generations. Snapshots go to rustfs (S3). - ax (
env/ax.nix,pkgs/ax/):ax-server(gRPC API),ax-controllerand Redis inax-system. The API is atax.k3s.lan:80(plaintext h2c through traefik). It is also on the homepage under AI. - Default task image: tasks without
spec.imagerun the Nix-builtax-task-runner. It shipsgit,curl, the fleet CA and thepicoding agent, which is preconfigured for Qwen3.8 on desg0's SGLang (OPENAI_BASE_URL=http://192.168.0.13:8000/v1). - ax objects (
manifests/ax/): Gateways, Workspaces and Tasks live in ax's Redis, not in Kubernetes. ArgoCD does not apply them. Useax_apply.
Every Home Manager host has the ax and kubectl-ate CLIs (home/home.nix).
There are two ways to reach the server:
| Where | How ax connects | What works |
|---|---|---|
| tensorbook (no fleet kubeconfig) | AX_SERVER=ax.k3s.lan:80, set in nushell |
apply, get, describe, watch, suspend, resume, delete |
de-msa2, meshify (as m) |
port-forward through the fleet kubeconfig (home/k3s_kubeconfig.nix) |
everything, including ax ssh and kubectl ate |
ax ssh and kubectl ate always need a kubeconfig, because they tunnel to
atenet-router. Do not run ax with sudo: sudo keeps HOME and leaves
root-owned ~/.kube and ~/.ax behind. In a shell other than nushell on
tensorbook, export AX_SERVER=ax.k3s.lan:80 first.
Apply the shared objects (Gateways, Models, Workspaces), then a Task:
nix run .#ax_apply # everything shared in manifests/ax/
nix run .#ax_apply -- manifests/ax/task-smoke-pi.yaml # one TaskA minimal Task that lets pi work on a cloned repo:
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
name: smoke-pi
atespace: default
spec:
command: [bash, -c, 'pi -p --no-session "Summarize this repo into /workspace/summary.md"']
workspaces:
- name: monty-persona # cloned into /workspace, the working directory
gateway:
name: lan-llm # egress to desg0 (SGLang) and Forgejo
debug: true # needed for `ax ssh`Watch and inspect it:
ax get tasks
ax watch task smoke-pi # "Running" is reported as final; check the result with ax ssh
ax ssh smoke-pi -- cat /workspace/summary.md
ax suspend task smoke-pi # snapshot; `ax resume task smoke-pi` continues
ax delete task smoke-pi # also removes the Substrate actorGotchas:
- Delete a Task before you re-apply it under the same name. Otherwise Substrate resumes the old actor, with its old image and snapshot.
- The Gateway egress allowlist is recorded but not enforced yet (no egress gateway is deployed), so sandbox egress is allow-all.
- Pods can't resolve
*.k3s.lannames exceptforgejo.k3s.lan. Use LAN IPs for other services. - To ship a new runner image, run
nix run .#ax-push-images(afterskopeo login --tls-verify=false de-msa2:2999) before you push the re-rendered manifests. Otherwise the controller points at a digest that is not in the registry yet.
The fleet's store paths live in attic
on de-msa2 (hosts/de-msa2/attic.nix), exposed at https://attic.k3s.lan
through the cluster ingress. Every host lists it as its first substituter
(modules/base_system.nix), with cache.nixos.org appended after; the
nixos cache is public, so pulls need no token — only the per-cache
signing key.
desg0 keeps it stocked: the nixos-cache-builder timer
(modules/nixos_cache_builder.nix) runs daily at 04:00 — clone the repo,
nix flake update, build every host's system.build.toplevel, attic push
the results, and only then commit and push the new flake.lock. A lock that
does not build is never committed.
Since it updates the lock on every run, that timer is also the fleet's canary
for upstream nixpkgs changes — it breaks the day nixpkgs marks a package we
install as insecure, renames an option or drops a package. A failing host no
longer aborts the run: the other hosts are still built and pushed, then a pi
agent gets the build log, diagnoses the breakage, patches it, checks that every
host still evaluates and pushes a cache-mechanic/fix-* branch for review
(same pattern as hosts/de-msa2/clanker-bot.nix). It never pushes to main,
so no unreviewed LLM edit can reach a host's next rebuild.
Alongside it, attic-watch-store uploads
every new path that lands in desg0's store, so remote builds from the
laptops (via modules/remote_builder.nix) end up in the cache too.
Retention is enforced by GC: every 12 hours, paths not pulled for 3 months go.
An explorable version with guided views is in
docs/diagrams/nix-cache-attic.html.
A curated set of self-hosted apps, each a NixOS module, fronted over HTTPS through the cluster ingress where it matters:
ntfy (push notifications) · forgejo (git + actions + LFS) · grafana (dashboards) · vikunja (tasks) · attic (nix binary cache) · polaris (music) · calibre-web (ebooks) · mealie (recipes) · immich (photos) · searx (search) · readeck · uptime-kuma · and more.
Laptops run Hyprland managed by Home Manager (home/home_hyprland.nix): a
curated terminal set, helix editor, yazi file manager, waybar,
animated wallpapers via awww, keyboard-driven mouse control via stochos,
and per-host home configs (home/<host>.nix).
yubi_key.nix for GPG/SSH, k3s_nvidia.nix for container GPU passthrough,
virtualization_host.nix for libvirt VMs (vms/tor.nix, vms/waterfox.nix),
backup.nix / backup_home_to_remote.nix for off-host backups.
hosts/ one dir per machine — its configuration.nix + host-local services
modules/ reusable NixOS modules (base system, k3s, AI, networking, desktop)
home/ Home Manager configs (shell, editors, Hyprland, per-host tweaks)
env/ nixidy cluster environment: argocd, cert-manager, host_ingress, homepage, substrate, ax
manifests/ rendered k8s YAML (auto-generated, committed for ArgoCD)
secrets/ agenix-encrypted secrets
scripts/ small nix scripts (wake-on-lan, forgejo starred sync, app list)
vms/ libvirt VM definitions
flake.nix the single entry point for every host and the nixidy env
Rebuild any host:
sudo nixos-rebuild switch --flake .#meshifyOr with nh for a friendlier flow:
nh os switch .Reconcile the whole k3s cluster (renders + pushes manifests; ArgoCD syncs):
nix run .#nixidy -- switch .#prodPull the newest nixpkgs and rebuild:
nix flake update
sudo nixos-rebuild switch --flake .#<host> --upgrade-allUpdate a single input:
nix flake lock --update-input nixpkgs-unstableInspect flake metadata:
nix flake metadata .nix-store --gc # manual GC
nh clean all --keep 5 # keep the last 5 boot generationsCached evaluation errors — if you hit error: cached failure of attribute 'nixosConfigurations.<host>…', bypass the eval cache:
--option eval-cache falseWhy does X depend on Y? (e.g. an insecure package pulled in transitively):
nix why-depends /run/current-system \
$(nix-build '<nixpkgs>' -A electron_35 --no-out-link)Exposing a new host-local service at *.k3s.lan — add one entry to the
services list in env/host_ingress.nix, add the hostname to
modules/base_system.nix, set the service's base URL. Done.
The flake includes nixos-vex as an
input. It evaluates your NixOS configuration and emits a VEX feed — a
machine-readable list of CVE suppression decisions that apply to your system
— so grype can filter out false positives and leave only real, untriaged
findings.
Full scan workflow for a host (e.g. meshify):
# 1. Build the system closure
nix build .#nixosConfigurations.meshify.config.system.build.toplevel -o system
# 2. Generate a CycloneDX SBOM from the closure
nix run github:tiiuae/sbomnix#sbomnix -- ./system --cdx=sbom.cdx.json
# 3. Build the VEX feed for your specific configuration
nix build .#nixosConfigurations.meshify.config.system.build.vexFeed -o vex-feed.json
# 4. Scan, suppressing known non-applicable CVEs
grype sbom:sbom.cdx.json --vex vex-feed.jsonThe EPSS column in grype's output is the daily probability (0–1) that a CVE
is actively exploited in the wild. Prioritise by EPSS, not CVSS: a CVSS 9.8
with EPSS 0.001 is far less urgent than a CVSS 5.0 with EPSS 0.8.
MIT.