From c5e2912bb8f71ee5c872d52c6db444cb37efbe9a Mon Sep 17 00:00:00 2001 From: Joshua Gilman Date: Fri, 16 Oct 2026 09:26:02 -0700 Subject: [PATCH] docs: add the services VLAN, bulk pool, and media stack runbook --- ...ose-nas01-hdds-as-a-clustered-bulk-pool.md | 58 +++++++ ...e-service-vms-a-dedicated-services-vlan.md | 58 +++++++ .../docs/reference/networking/address-plan.md | 35 ++-- docs/docs/runbooks/media-stack.md | 157 ++++++++++++++++++ docs/mkdocs.yml | 3 + 5 files changed, 297 insertions(+), 14 deletions(-) create mode 100644 docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md create mode 100644 docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md create mode 100644 docs/docs/runbooks/media-stack.md diff --git a/docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md b/docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md new file mode 100644 index 0000000..ad8717d --- /dev/null +++ b/docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md @@ -0,0 +1,58 @@ +--- +status: accepted +date: 2026-10-16 +decision-makers: + - Josh Gilman +--- + +# ADR-0007: Expose the nas01 HDDs as a Clustered Bulk Pool + +## Context and Problem Statement + +`nas01` carries the `hdd` zpool (4x 6 TB raidz1), but it was OS-level only: +Incus storage pools in a cluster must exist on every member, and the HDDs +exist only on `nas01`. The media stack is the first workload that needs those +disks. How should instances on `nas01` consume the HDD pool? + +## Decision Drivers + +- Instances get the HDDs through normal Incus custom volumes (quotas, + snapshots, OpenTofu ownership), not ad-hoc host paths. +- Nothing intended for the HDDs may land on lab SSDs by accident. +- Converged by fleet's `cluster/` project like every other pool. + +## Considered Options + +- Clustered pool with per-member sources: `nas01` uses `hdd/bulk`, every + other member a quota-capped placeholder on its `data` pool. +- Host-path or virtiofs passthrough of an `hdd` dataset into one VM. +- Defer until the object-storage design (T16). + +## Decision Outcome + +Chosen option: **clustered pool with per-member sources**. The Incus pool +`bulk` is a ZFS pool whose `source` is `hdd/bulk` on `nas01` and a 1 GiB +quota-capped `data/bulk` dataset on each lab member. Volumes and instances +that use `bulk` are pinned to `nas01`. + +### Consequences + +- Good, because consumers use ordinary Incus volumes, so OpenTofu owns them + and they survive instance rebuilds. +- Good, because a volume accidentally targeted at a lab member fails at 1 GiB + instead of silently consuming SSD. +- Bad, because `incus storage list` shows `bulk` on lab members that have no + real capacity behind it. +- Neutral: this does not settle object storage or backup (T16); a future + Garage deployment can still consume `bulk` volumes. + +### Confirmation + +`incus storage info nas01:bulk --target nas01` reports the raidz1 capacity, +and `--target lab01` reports 1 GiB. Fleet's +`test_bulk_pool_lands_on_hdd_only_on_nas01` pins the member mapping. + +## More Information + +- Implementation: `GilmanLab/fleet` `cluster/src/fleet_cluster/deploys/storage.py` +- First consumer: [Deploy and operate the media stack](../runbooks/media-stack.md) diff --git a/docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md b/docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md new file mode 100644 index 0000000..be72146 --- /dev/null +++ b/docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md @@ -0,0 +1,58 @@ +--- +status: accepted +date: 2026-10-16 +decision-makers: + - Josh Gilman +--- + +# ADR-0008: Give Service VMs a Dedicated Services VLAN + +## Context and Problem Statement + +Long-lived service VMs (first: `media01`) need a routed LAN address reachable +from home and the tailnet. The instance VLAN 40 parent `fast40` is owned by +OVN as its physical uplink, and a raw macvlan on that parent stops OVS from +attaching it, breaking north-south traffic whenever that member is the OVN +gateway ([lxc/incus#3986](https://github.com/lxc/incus/issues/3986)). Where +should service VMs attach? + +## Decision Drivers + +- Must not interfere with the OVN sandbox fabric (ADR-0006). +- Keep internet-adjacent workloads off the management VLAN. +- Plain L2 attachment with no dependency on OVN central. + +## Considered Options + +- New routed services VLAN 50 carried only to the members that host service + VMs. +- A dedicated OVN network with a network forward. +- Bridged NIC on the management VLAN 10, like `ovncentral01`. + +## Decision Outcome + +Chosen option: **services VLAN 50** (`10.10.50.0/24`, gateway `gw01`). It is +tagged on `sw-core01` port 8 (gw01 trunk) and the ports of members that host +service VMs (today `nas01`, port 7). IncusOS declares an address-less +`fast50` interface on those members, and service VMs attach with +`nictype=macvlan parent=fast50`. gw01 lets the VLAN reach the internet and +blocks new connections from it to management, OOB, sandbox, and home networks. +Management, home, and Tailscale clients can open connections into it. + +### Consequences + +- Good, because OVN keeps exclusive use of `fast40` on every member. +- Good, because a compromised service VM cannot open connections to the + management plane or the home network. +- Bad, because a new VLAN touches gw01, sw-core01, and fleet, and every new + service host member needs its switch port tagged. + +### Confirmation + +`incus network show nas01:fast40 --target ` lists only +`fast40-uplink` in `used_by`. The media root rejects `network_parent=fast40`. + +## More Information + +- [Network address and VLAN plan](../reference/networking/address-plan.md) +- [Deploy and operate the media stack](../runbooks/media-stack.md) diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md index f49c1b3..a0c9f68 100644 --- a/docs/docs/reference/networking/address-plan.md +++ b/docs/docs/reference/networking/address-plan.md @@ -40,6 +40,7 @@ traffic retains its source addresses. | `10` | Management | `10.10.10.0/24` | `10.10.10.1` | IncusOS management, network-device management, and `nas01` management | | `30` | Storage | `10.10.30.0/24` | None (not routed) | Incus node storage network on the compute-facing `sw-core01` links | | `40` | Sandbox/workload | `10.10.40.0/24` | `10.10.40.1` | `sandbox01`, Incus instances attached over the cluster fast links, and other explicitly attached workload endpoints | +| `50` | Services | `10.10.50.0/24` | `10.10.50.1` | Long-lived service VMs on members that carry `fast50` (today `nas01`); see [ADR-0008](../../decisions/0008-give-service-vms-a-dedicated-services-vlan.md) | | `70` | OOB | `10.10.70.0/24` | `10.10.70.1` | MS-02 AMT, `pikvm01`, `kvm01`, and management-switch administration | VLAN 20 and `10.10.20.0/24` are retired. The lab does not retain a PXE or @@ -68,6 +69,7 @@ by the `GilmanLab/fleet` `cluster/` project and mirrored in each node's seed. | `gw01` `glab.lol` mirror | `10.10.10.54` | Local service address | | `ovncentral01` OVN central | `10.10.10.15` | Static VM interface on `nas01`'s unmanaged `mgmt` bridge | | `agentcompute01` MCP service | `10.10.10.16` | Static `/32` on a routed Incus NIC through `lab01`'s `_vmgmt` interface; host gateway `169.254.0.1` | +| `media01` media stack | `10.10.50.20` | Static VM interface; macvlan on `nas01`'s IncusOS `fast50` interface | ### Hosts @@ -89,8 +91,9 @@ configuration. Infrastructure endpoints do not depend on DHCP: gateway and managed-switch interface addresses and the local DNS mirror address are static interface -configuration; the OVN central address is static VM network configuration; -IncusOS node management addresses are static in each node's seed (bound to the +configuration; the OVN central and `media01` addresses are static VM network +configuration; IncusOS node management addresses are static in each node's +seed (bound to the management NIC's hardware MAC in `GilmanLab/fleet`); storage-network addresses are static IncusOS runtime configuration (converged by the fleet `cluster/` project and mirrored in the seeds); and lab-node AMT addresses are static in @@ -170,7 +173,7 @@ for deployment and cutover checks. | `gw01` chassis port | VyOS interface | Mode | Network | | --- | --- | --- | --- | | `SFP+ 1` | `eth0` | Routed | Transit `10.0.0.2/30` to `rtr01` | -| `SFP+ 2` | `eth1` | 802.1Q trunk | VLANs 10 and 40 to `sw-core01` | +| `SFP+ 2` | `eth1` | 802.1Q trunk | VLANs 10, 40, and 50 to `sw-core01`; VLAN 50 is routed on `eth1.50` without a bridge | | `Port 1` | `eth3` | 802.1Q trunk | VLANs 10 and 70 to `sw-mgmt01` | | `Port 2` | `eth2` | Untagged access | VLAN 40 to `sandbox01` | | `Port 3` | `eth4` | Untagged access | VLAN 70 to `pikvm01` | @@ -199,22 +202,26 @@ the `nas01` 5GbE NIC therefore negotiate no faster than 2.5Gbps on this switch. ### `sw-core01` -Port 8 is the 802.1Q trunk to `gw01` and carries VLANs 10 and 40. VLAN 10 +Port 8 is the 802.1Q trunk to `gw01` and carries VLANs 10, 40, and 50. VLAN 10 provides the switch management path. Ports 1 through 6 form three 802.3ad LAGs, one per MS-02 SFP+ pair: `bond-lab01` (ports 1–2), `bond-lab02` (ports 3–4), and `bond-lab03` (ports 5–6). Port 7 connects the `nas01` 10GbE interface. VLANs 30 and 40 are tagged on the three LAGs and port 7: VLAN 30 is the node storage network, and VLAN 40 carries Incus instance traffic so -workloads stay off the management plane. These links are not required for -IncusOS management boot. The host-side counterpart, converged by the -`GilmanLab/fleet` `cluster/` project, is the `vlan_tags` allow-list plus an -IncusOS-declared VLAN interface per carried VLAN (`fast30`, `fast40`). -Instances must attach to the IncusOS-owned interface (for example macvlan -with `parent=fast40`), never with an Incus-created `vlan=` sub-interface on -`fast`: the visible `fast` device is an IncusOS-internal VLAN-filtering -bridge, and only IncusOS-declared VLANs receive bridge self-port membership, -so other tagged sub-interfaces pass no traffic. Additional instance VLANs -join these links when their first consumer arrives. +workloads stay off the management plane. VLAN 50 (services) is tagged only on +port 7. These links are not required for IncusOS management boot. The +host-side counterpart, converged by the `GilmanLab/fleet` `cluster/` project, +is the `vlan_tags` allow-list plus an IncusOS-declared VLAN interface per +carried VLAN (`fast30`, `fast40`, and `fast50` on `nas01`). Instances must +attach to an IncusOS-owned interface, never with an Incus-created `vlan=` +sub-interface on `fast`: the visible `fast` device is an IncusOS-internal +VLAN-filtering bridge, and only IncusOS-declared VLANs receive bridge +self-port membership, so other tagged sub-interfaces pass no traffic. OVN +owns `fast40` as the `fast40-uplink` parent; never attach a macvlan to it, +because OVS then cannot add the parent and that member's OVN gateway fails +([lxc/incus#3986](https://github.com/lxc/incus/issues/3986)). Service VMs use +macvlan with `parent=fast50`. Additional instance VLANs join these links when +their first consumer arrives. ## DHCP and DNS ownership diff --git a/docs/docs/runbooks/media-stack.md b/docs/docs/runbooks/media-stack.md new file mode 100644 index 0000000..ebbf625 --- /dev/null +++ b/docs/docs/runbooks/media-stack.md @@ -0,0 +1,157 @@ +--- +title: Deploy and operate the media stack +description: Deploy media01 on nas01, deliver the ProtonVPN key, wire the apps, and recover the stack. +--- + +# Deploy and Operate the Media Stack + +`media01` runs Jellyfin, Sonarr, Radarr, Prowlarr, and qBittorrent under +Docker Compose on `nas01`. qBittorrent shares gluetun's network namespace, so +its only egress is the ProtonVPN WireGuard tunnel. The OpenTofu root is +`GilmanLab/fleet` `incus/media/`; its README lists the resources and volumes. + +| App | URL | +| --- | --- | +| Jellyfin | `http://10.10.50.20:8096` | +| Sonarr | `http://10.10.50.20:8989` | +| Radarr | `http://10.10.50.20:7878` | +| Prowlarr | `http://10.10.50.20:9696` | +| qBittorrent | `http://10.10.50.20:8080` | + +The address and VLAN are canonical in the +[address plan](../reference/networking/address-plan.md). Home clients reach +VLAN 50 through `gw01`; tailnet clients reach it through the gw01 subnet route. + +## Preconditions + +- `nas01:` Incus remote with the operator client certificate. The OpenTofu + provider on macOS needs + `INCUS_CONF="$HOME/Library/Application Support/incus"`. +- `AWS_PROFILE=lab-admin` with a live SSO session (state backend and SOPS). +- The `bulk` pool and `nas01` `fast50` interface converged by + `fleet-cluster storage` and `fleet-cluster network` + ([ADR-0007](../decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md), + [ADR-0008](../decisions/0008-give-service-vms-a-dedicated-services-vlan.md)). +- VLAN 50 on `sw-core01` port 7 and `gw01` `eth1.50` (`GilmanLab/networking`). +- A ProtonVPN Plus (or higher) subscription. + +## Safety impact + +Creating the VM touches only `nas01`. The `media` and `media01-appdata` +volumes have `prevent_destroy`; a `tofu destroy` of the root fails rather than +deleting the library or app settings. Never attach the VM to `fast40`: a +macvlan there breaks `nas01`'s OVN gateway. + +## Create the ProtonVPN secret + +1. In the Proton dashboard, open **Downloads → WireGuard configuration**. + Choose platform **Linux**, turn on **NAT-PMP (Port Forwarding)**, pick a + **P2P** server, and download the file. +2. Copy the `PrivateKey` value from the file, then create the secret in the + `GilmanLab/secrets` checkout: + + ```bash + mkdir -p services/media + AWS_PROFILE=lab-admin sops edit services/media/vpn.sops.yaml + ``` + + ```yaml + protonvpn: + wireguard_private_key: + server_countries: Switzerland # optional, comma-separated + ``` + +3. Commit it through a pull request like every other secret. + +## Deploy + +From `GilmanLab/fleet` `incus/media/`: + +```bash +export AWS_PROFILE=lab-admin GLAB_AWS_STATE_BUCKET=glab-lab-tfstate-186067932323 +export INCUS_CONF="$HOME/Library/Application Support/incus" +just init +just plan # expect the VM and, on first deploy, both volumes +just apply +incus exec nas01:media01 -- cloud-init status --wait +just deliver-vpn ~/code/glab/secrets/services/media/vpn.sops.yaml +``` + +The stack stays down until `deliver-vpn` installs `/etc/media/vpn.env`. +Docker waits for `/data` and `/srv/appdata` to be mounted before it starts +any container. + +## Wire the apps + +The first login on each app creates its admin account. qBittorrent prints a +temporary admin password once, in its log: + +```bash +incus exec nas01:media01 -- docker logs qbittorrent 2>&1 | grep -i password +``` + +Change it in **Tools → Options → Web UI**. Leave **Bypass authentication for +clients on localhost** on; gluetun uses it to push the forwarded port. + +All containers see the library volume at `/data`: + +| Path | Purpose | +| --- | --- | +| `/data/torrents/{tv,movies}` | qBittorrent category save paths | +| `/data/torrents/incomplete` | qBittorrent incomplete downloads | +| `/data/media/{tv,movies}` | Sonarr and Radarr root folders; Jellyfin libraries (read-only) | + +1. Sonarr and Radarr: **Settings → Media Management**, add root folder + `/data/media/tv` or `/data/media/movies`. Leave **Use Hardlinks instead of + Copy** on. +2. Sonarr and Radarr: **Settings → Download Clients → qBittorrent**, host + `gluetun`, port `8080`, category `tv` or `movies`. +3. qBittorrent: create categories `tv` → `/data/torrents/tv` and `movies` → + `/data/torrents/movies`. +4. Prowlarr: **Settings → Apps**, add Sonarr (`http://sonarr:8989`) and Radarr + (`http://radarr:7878`) with their API keys and Prowlarr server + `http://prowlarr:9696`. Add indexers in Prowlarr only; it syncs them. +5. Jellyfin: add libraries on `/data/media/tv` and `/data/media/movies`. + +## Verify + +```bash +# VPN egress: must be a Proton address, never the home WAN address. +incus exec nas01:media01 -- docker exec gluetun wget -qO- https://ipinfo.io/ip +# Forwarded port gluetun negotiated and pushed into qBittorrent. +incus exec nas01:media01 -- docker logs gluetun 2>&1 | grep -i 'port forwarded' +# Kill switch: with gluetun stopped, qBittorrent has no network at all. +incus exec nas01:media01 -- docker stop gluetun +incus exec nas01:media01 -- docker exec qbittorrent wget -qO- -T5 https://ipinfo.io/ip # must fail +incus exec nas01:media01 -- docker compose --project-directory /opt/media up -d +``` + +## Recovery + +- **Rebuild the VM:** `tofu apply -replace=incus_instance.media`, wait for + cloud-init, then `just deliver-vpn`. Both volumes and every app setting + survive. If the pinned image has expired from `images:`, pin a current + `images:ubuntu/26.04/cloud` VM fingerprint in `variables.tf` first. +- **Rotate the VPN key:** edit the secret, then rerun `just deliver-vpn`; it + recreates gluetun and qBittorrent. +- **Stack down after a reboot:** check `systemctl status docker` in the VM. + A failed `ExecStartPre` means a volume did not mount; inspect + `incus config device show nas01:media01`. + +## Hardware transcoding + +`gpu_pci_address` can pass the Radeon 890M (`0000:c7:00.0`) through for +VA-API. On 2026-10-16 a test VM bound `amdgpu` to it, but PSP firmware init +failed (`PSP tmr init failed`) with the factory BIOS (2 GiB UMA, VBIOS +`113-STRIXEMU-001` from the ROM BAR). The likely fix is setting +`UMA_SPECIFIED` to 8 GiB and enabling ReBAR in the `nas01` BIOS, which needs a +`nas01` reboot. Until that is verified on a throwaway VM, Jellyfin transcodes in +software. + +## Escalate + +- gluetun reports no forwarded port: the Proton key was generated without + NAT-PMP, or the server is not P2P. Regenerate the key. +- `incus storage info nas01:bulk --target nas01` shows the pool degraded: + follow the `hdd` pool recovery notes in + [Rebuild nas01](rebuild-nas01.md). diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index dc9f7fc..b4fbae2 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -38,6 +38,8 @@ nav: - "ADR-0004: Manage RouterOS Devices with OpenTofu": decisions/0004-manage-routeros-devices-with-opentofu.md - "ADR-0005: Anchor Internal PKI on the KMS Root with Sibling Intermediates": decisions/0005-anchor-internal-pki-on-kms-root-with-sibling-intermediates.md - "ADR-0006: OVN as the Sandbox Network Fabric": decisions/0006-ovn-as-the-sandbox-network-fabric.md + - "ADR-0007: Expose the nas01 HDDs as a Clustered Bulk Pool": decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md + - "ADR-0008: Give Service VMs a Dedicated Services VLAN": decisions/0008-give-service-vms-a-dedicated-services-vlan.md - Designs: - Lab v2 core network: designs/lab-v2-core-network.md - "Draft: agentcompute": designs/drafts/agentcompute.md @@ -61,6 +63,7 @@ nav: - Manage the sw-mgmt01 configuration: runbooks/sw-mgmt01-configuration.md - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md - Deploy private image runners: runbooks/private-image-runners.md + - Deploy and operate the media stack: runbooks/media-stack.md plugins: - search