jmgilman · Oct 16, 2026 · Oct 16, 2026
diff --git a/docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md b/docs/docs/decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md
@@ -0,0 +1,58 @@
+---
+status: accepted
+date: 2026-10-16
+decision-makers:
+  - Josh Gilman
+---
+
+# ADR-0007: Expose the nas01 HDDs as a Clustered Bulk Pool
+
+## Context and Problem Statement
+
+`nas01` carries the `hdd` zpool (4x 6 TB raidz1), but it was OS-level only:
+Incus storage pools in a cluster must exist on every member, and the HDDs
+exist only on `nas01`. The media stack is the first workload that needs those
+disks. How should instances on `nas01` consume the HDD pool?
+
+## Decision Drivers
+
+- Instances get the HDDs through normal Incus custom volumes (quotas,
+  snapshots, OpenTofu ownership), not ad-hoc host paths.
+- Nothing intended for the HDDs may land on lab SSDs by accident.
+- Converged by fleet's `cluster/` project like every other pool.
+
+## Considered Options
+
+- Clustered pool with per-member sources: `nas01` uses `hdd/bulk`, every
+  other member a quota-capped placeholder on its `data` pool.
+- Host-path or virtiofs passthrough of an `hdd` dataset into one VM.
+- Defer until the object-storage design (T16).
+
+## Decision Outcome
+
+Chosen option: **clustered pool with per-member sources**. The Incus pool
+`bulk` is a ZFS pool whose `source` is `hdd/bulk` on `nas01` and a 1 GiB
+quota-capped `data/bulk` dataset on each lab member. Volumes and instances
+that use `bulk` are pinned to `nas01`.
+
+### Consequences
+
+- Good, because consumers use ordinary Incus volumes, so OpenTofu owns them
+  and they survive instance rebuilds.
+- Good, because a volume accidentally targeted at a lab member fails at 1 GiB
+  instead of silently consuming SSD.
+- Bad, because `incus storage list` shows `bulk` on lab members that have no
+  real capacity behind it.
+- Neutral: this does not settle object storage or backup (T16); a future
+  Garage deployment can still consume `bulk` volumes.
+
+### Confirmation
+
+`incus storage info nas01:bulk --target nas01` reports the raidz1 capacity,
+and `--target lab01` reports 1 GiB. Fleet's
+`test_bulk_pool_lands_on_hdd_only_on_nas01` pins the member mapping.
+
+## More Information
+
+- Implementation: `GilmanLab/fleet` `cluster/src/fleet_cluster/deploys/storage.py`
+- First consumer: [Deploy and operate the media stack](../runbooks/media-stack.md)
diff --git a/docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md b/docs/docs/decisions/0008-give-service-vms-a-dedicated-services-vlan.md
@@ -0,0 +1,58 @@
+---
+status: accepted
+date: 2026-10-16
+decision-makers:
+  - Josh Gilman
+---
+
+# ADR-0008: Give Service VMs a Dedicated Services VLAN
+
+## Context and Problem Statement
+
+Long-lived service VMs (first: `media01`) need a routed LAN address reachable
+from home and the tailnet. The instance VLAN 40 parent `fast40` is owned by
+OVN as its physical uplink, and a raw macvlan on that parent stops OVS from
+attaching it, breaking north-south traffic whenever that member is the OVN
+gateway ([lxc/incus#3986](https://github.com/lxc/incus/issues/3986)). Where
+should service VMs attach?
+
+## Decision Drivers
+
+- Must not interfere with the OVN sandbox fabric (ADR-0006).
+- Keep internet-adjacent workloads off the management VLAN.
+- Plain L2 attachment with no dependency on OVN central.
+
+## Considered Options
+
+- New routed services VLAN 50 carried only to the members that host service
+  VMs.
+- A dedicated OVN network with a network forward.
+- Bridged NIC on the management VLAN 10, like `ovncentral01`.
+
+## Decision Outcome
+
+Chosen option: **services VLAN 50** (`10.10.50.0/24`, gateway `gw01`). It is
+tagged on `sw-core01` port 8 (gw01 trunk) and the ports of members that host
+service VMs (today `nas01`, port 7). IncusOS declares an address-less
+`fast50` interface on those members, and service VMs attach with
+`nictype=macvlan parent=fast50`. gw01 lets the VLAN reach the internet and
+blocks new connections from it to management, OOB, sandbox, and home networks.
+Management, home, and Tailscale clients can open connections into it.
+
+### Consequences
+
+- Good, because OVN keeps exclusive use of `fast40` on every member.
+- Good, because a compromised service VM cannot open connections to the
+  management plane or the home network.
+- Bad, because a new VLAN touches gw01, sw-core01, and fleet, and every new
+  service host member needs its switch port tagged.
+
+### Confirmation
+
+`incus network show nas01:fast40 --target <member>` lists only
+`fast40-uplink` in `used_by`. The media root rejects `network_parent=fast40`.
+
+## More Information
+
+- [Network address and VLAN plan](../reference/networking/address-plan.md)
+- [Deploy and operate the media stack](../runbooks/media-stack.md)
diff --git a/docs/docs/reference/networking/address-plan.md b/docs/docs/reference/networking/address-plan.md
@@ -40,6 +40,7 @@ traffic retains its source addresses.
 | `10` | Management | `10.10.10.0/24` | `10.10.10.1` | IncusOS management, network-device management, and `nas01` management |
 | `30` | Storage | `10.10.30.0/24` | None (not routed) | Incus node storage network on the compute-facing `sw-core01` links |
 | `40` | Sandbox/workload | `10.10.40.0/24` | `10.10.40.1` | `sandbox01`, Incus instances attached over the cluster fast links, and other explicitly attached workload endpoints |
+| `50` | Services | `10.10.50.0/24` | `10.10.50.1` | Long-lived service VMs on members that carry `fast50` (today `nas01`); see [ADR-0008](../../decisions/0008-give-service-vms-a-dedicated-services-vlan.md) |
 | `70` | OOB | `10.10.70.0/24` | `10.10.70.1` | MS-02 AMT, `pikvm01`, `kvm01`, and management-switch administration |
 
 VLAN 20 and `10.10.20.0/24` are retired. The lab does not retain a PXE or
@@ -68,6 +69,7 @@ by the `GilmanLab/fleet` `cluster/` project and mirrored in each node's seed.
 | `gw01` `glab.lol` mirror | `10.10.10.54` | Local service address |
 | `ovncentral01` OVN central | `10.10.10.15` | Static VM interface on `nas01`'s unmanaged `mgmt` bridge |
 | `agentcompute01` MCP service | `10.10.10.16` | Static `/32` on a routed Incus NIC through `lab01`'s `_vmgmt` interface; host gateway `169.254.0.1` |
+| `media01` media stack | `10.10.50.20` | Static VM interface; macvlan on `nas01`'s IncusOS `fast50` interface |
 
 ### Hosts
 
@@ -89,8 +91,9 @@ configuration.
 
 Infrastructure endpoints do not depend on DHCP: gateway and managed-switch
 interface addresses and the local DNS mirror address are static interface
-configuration; the OVN central address is static VM network configuration;
-IncusOS node management addresses are static in each node's seed (bound to the
+configuration; the OVN central and `media01` addresses are static VM network
+configuration; IncusOS node management addresses are static in each node's
+seed (bound to the
 management NIC's hardware MAC in `GilmanLab/fleet`); storage-network addresses
 are static IncusOS runtime configuration (converged by the fleet `cluster/`
 project and mirrored in the seeds); and lab-node AMT addresses are static in
@@ -170,7 +173,7 @@ for deployment and cutover checks.
 | `gw01` chassis port | VyOS interface | Mode | Network |
 | --- | --- | --- | --- |
 | `SFP+ 1` | `eth0` | Routed | Transit `10.0.0.2/30` to `rtr01` |
-| `SFP+ 2` | `eth1` | 802.1Q trunk | VLANs 10 and 40 to `sw-core01` |
+| `SFP+ 2` | `eth1` | 802.1Q trunk | VLANs 10, 40, and 50 to `sw-core01`; VLAN 50 is routed on `eth1.50` without a bridge |
 | `Port 1` | `eth3` | 802.1Q trunk | VLANs 10 and 70 to `sw-mgmt01` |
 | `Port 2` | `eth2` | Untagged access | VLAN 40 to `sandbox01` |
 | `Port 3` | `eth4` | Untagged access | VLAN 70 to `pikvm01` |
@@ -199,22 +202,26 @@ the `nas01` 5GbE NIC therefore negotiate no faster than 2.5Gbps on this switch.
 
 ### `sw-core01`
 
-Port 8 is the 802.1Q trunk to `gw01` and carries VLANs 10 and 40. VLAN 10
+Port 8 is the 802.1Q trunk to `gw01` and carries VLANs 10, 40, and 50. VLAN 10
 provides the switch management path. Ports 1 through 6 form three 802.3ad
 LAGs, one per MS-02 SFP+ pair: `bond-lab01` (ports 1–2), `bond-lab02` (ports
 3–4), and `bond-lab03` (ports 5–6). Port 7 connects the `nas01` 10GbE
 interface. VLANs 30 and 40 are tagged on the three LAGs and port 7: VLAN 30
 is the node storage network, and VLAN 40 carries Incus instance traffic so
-workloads stay off the management plane. These links are not required for
-IncusOS management boot. The host-side counterpart, converged by the
-`GilmanLab/fleet` `cluster/` project, is the `vlan_tags` allow-list plus an
-IncusOS-declared VLAN interface per carried VLAN (`fast30`, `fast40`).
-Instances must attach to the IncusOS-owned interface (for example macvlan
-with `parent=fast40`), never with an Incus-created `vlan=` sub-interface on
-`fast`: the visible `fast` device is an IncusOS-internal VLAN-filtering
-bridge, and only IncusOS-declared VLANs receive bridge self-port membership,
-so other tagged sub-interfaces pass no traffic. Additional instance VLANs
-join these links when their first consumer arrives.
+workloads stay off the management plane. VLAN 50 (services) is tagged only on
+port 7. These links are not required for IncusOS management boot. The
+host-side counterpart, converged by the `GilmanLab/fleet` `cluster/` project,
+is the `vlan_tags` allow-list plus an IncusOS-declared VLAN interface per
+carried VLAN (`fast30`, `fast40`, and `fast50` on `nas01`). Instances must
+attach to an IncusOS-owned interface, never with an Incus-created `vlan=`
+sub-interface on `fast`: the visible `fast` device is an IncusOS-internal
+VLAN-filtering bridge, and only IncusOS-declared VLANs receive bridge
+self-port membership, so other tagged sub-interfaces pass no traffic. OVN
+owns `fast40` as the `fast40-uplink` parent; never attach a macvlan to it,
+because OVS then cannot add the parent and that member's OVN gateway fails
+([lxc/incus#3986](https://github.com/lxc/incus/issues/3986)). Service VMs use
+macvlan with `parent=fast50`. Additional instance VLANs join these links when
+their first consumer arrives.
 
 ## DHCP and DNS ownership
 

diff --git a/docs/docs/runbooks/media-stack.md b/docs/docs/runbooks/media-stack.md
@@ -0,0 +1,157 @@
+---
+title: Deploy and operate the media stack
+description: Deploy media01 on nas01, deliver the ProtonVPN key, wire the apps, and recover the stack.
+---
+
+# Deploy and Operate the Media Stack
+
+`media01` runs Jellyfin, Sonarr, Radarr, Prowlarr, and qBittorrent under
+Docker Compose on `nas01`. qBittorrent shares gluetun's network namespace, so
+its only egress is the ProtonVPN WireGuard tunnel. The OpenTofu root is
+`GilmanLab/fleet` `incus/media/`; its README lists the resources and volumes.
+
+| App | URL |
+| --- | --- |
+| Jellyfin | `http://10.10.50.20:8096` |
+| Sonarr | `http://10.10.50.20:8989` |
+| Radarr | `http://10.10.50.20:7878` |
+| Prowlarr | `http://10.10.50.20:9696` |
+| qBittorrent | `http://10.10.50.20:8080` |
+
+The address and VLAN are canonical in the
+[address plan](../reference/networking/address-plan.md). Home clients reach
+VLAN 50 through `gw01`; tailnet clients reach it through the gw01 subnet route.
+
+## Preconditions
+
+- `nas01:` Incus remote with the operator client certificate. The OpenTofu
+  provider on macOS needs
+  `INCUS_CONF="$HOME/Library/Application Support/incus"`.
+- `AWS_PROFILE=lab-admin` with a live SSO session (state backend and SOPS).
+- The `bulk` pool and `nas01` `fast50` interface converged by
+  `fleet-cluster storage` and `fleet-cluster network`
+  ([ADR-0007](../decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md),
+  [ADR-0008](../decisions/0008-give-service-vms-a-dedicated-services-vlan.md)).
+- VLAN 50 on `sw-core01` port 7 and `gw01` `eth1.50` (`GilmanLab/networking`).
+- A ProtonVPN Plus (or higher) subscription.
+
+## Safety impact
+
+Creating the VM touches only `nas01`. The `media` and `media01-appdata`
+volumes have `prevent_destroy`; a `tofu destroy` of the root fails rather than
+deleting the library or app settings. Never attach the VM to `fast40`: a
+macvlan there breaks `nas01`'s OVN gateway.
+
+## Create the ProtonVPN secret
+
+1. In the Proton dashboard, open **Downloads → WireGuard configuration**.
+   Choose platform **Linux**, turn on **NAT-PMP (Port Forwarding)**, pick a
+   **P2P** server, and download the file.
+2. Copy the `PrivateKey` value from the file, then create the secret in the
+   `GilmanLab/secrets` checkout:
+
+   ```bash
+   mkdir -p services/media
+   AWS_PROFILE=lab-admin sops edit services/media/vpn.sops.yaml
+   ```
+
+   ```yaml
+   protonvpn:
+     wireguard_private_key: <PrivateKey>
+     server_countries: Switzerland   # optional, comma-separated
+   ```
+
+3. Commit it through a pull request like every other secret.
+
+## Deploy
+
+From `GilmanLab/fleet` `incus/media/`:
+
+```bash
+export AWS_PROFILE=lab-admin GLAB_AWS_STATE_BUCKET=glab-lab-tfstate-186067932323
+export INCUS_CONF="$HOME/Library/Application Support/incus"
+just init
+just plan     # expect the VM and, on first deploy, both volumes
+just apply
+incus exec nas01:media01 -- cloud-init status --wait
+just deliver-vpn ~/code/glab/secrets/services/media/vpn.sops.yaml
+```
+
+The stack stays down until `deliver-vpn` installs `/etc/media/vpn.env`.
+Docker waits for `/data` and `/srv/appdata` to be mounted before it starts
+any container.
+
+## Wire the apps
+
+The first login on each app creates its admin account. qBittorrent prints a
+temporary admin password once, in its log:
+
+```bash
+incus exec nas01:media01 -- docker logs qbittorrent 2>&1 | grep -i password
+```
+
+Change it in **Tools → Options → Web UI**. Leave **Bypass authentication for
+clients on localhost** on; gluetun uses it to push the forwarded port.
+
+All containers see the library volume at `/data`:
+
+| Path | Purpose |
+| --- | --- |
+| `/data/torrents/{tv,movies}` | qBittorrent category save paths |
+| `/data/torrents/incomplete` | qBittorrent incomplete downloads |
+| `/data/media/{tv,movies}` | Sonarr and Radarr root folders; Jellyfin libraries (read-only) |
+
+1. Sonarr and Radarr: **Settings → Media Management**, add root folder
+   `/data/media/tv` or `/data/media/movies`. Leave **Use Hardlinks instead of
+   Copy** on.
+2. Sonarr and Radarr: **Settings → Download Clients → qBittorrent**, host
+   `gluetun`, port `8080`, category `tv` or `movies`.
+3. qBittorrent: create categories `tv` → `/data/torrents/tv` and `movies` →
+   `/data/torrents/movies`.
+4. Prowlarr: **Settings → Apps**, add Sonarr (`http://sonarr:8989`) and Radarr
+   (`http://radarr:7878`) with their API keys and Prowlarr server
+   `http://prowlarr:9696`. Add indexers in Prowlarr only; it syncs them.
+5. Jellyfin: add libraries on `/data/media/tv` and `/data/media/movies`.
+
+## Verify
+
+```bash
+# VPN egress: must be a Proton address, never the home WAN address.
+incus exec nas01:media01 -- docker exec gluetun wget -qO- https://ipinfo.io/ip
+# Forwarded port gluetun negotiated and pushed into qBittorrent.
+incus exec nas01:media01 -- docker logs gluetun 2>&1 | grep -i 'port forwarded'
+# Kill switch: with gluetun stopped, qBittorrent has no network at all.
+incus exec nas01:media01 -- docker stop gluetun
+incus exec nas01:media01 -- docker exec qbittorrent wget -qO- -T5 https://ipinfo.io/ip  # must fail
+incus exec nas01:media01 -- docker compose --project-directory /opt/media up -d
+```
+
+## Recovery
+
+- **Rebuild the VM:** `tofu apply -replace=incus_instance.media`, wait for
+  cloud-init, then `just deliver-vpn`. Both volumes and every app setting
+  survive. If the pinned image has expired from `images:`, pin a current
+  `images:ubuntu/26.04/cloud` VM fingerprint in `variables.tf` first.
+- **Rotate the VPN key:** edit the secret, then rerun `just deliver-vpn`; it
+  recreates gluetun and qBittorrent.
+- **Stack down after a reboot:** check `systemctl status docker` in the VM.
+  A failed `ExecStartPre` means a volume did not mount; inspect
+  `incus config device show nas01:media01`.
+
+## Hardware transcoding
+
+`gpu_pci_address` can pass the Radeon 890M (`0000:c7:00.0`) through for
+VA-API. On 2026-10-16 a test VM bound `amdgpu` to it, but PSP firmware init
+failed (`PSP tmr init failed`) with the factory BIOS (2 GiB UMA, VBIOS
+`113-STRIXEMU-001` from the ROM BAR). The likely fix is setting
+`UMA_SPECIFIED` to 8 GiB and enabling ReBAR in the `nas01` BIOS, which needs a
+`nas01` reboot. Until that is verified on a throwaway VM, Jellyfin transcodes in
+software.
+
+## Escalate
+
+- gluetun reports no forwarded port: the Proton key was generated without
+  NAT-PMP, or the server is not P2P. Regenerate the key.
+- `incus storage info nas01:bulk --target nas01` shows the pool degraded:
+  follow the `hdd` pool recovery notes in
+  [Rebuild nas01](rebuild-nas01.md).
diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -38,6 +38,8 @@ nav:
       - "ADR-0004: Manage RouterOS Devices with OpenTofu": decisions/0004-manage-routeros-devices-with-opentofu.md
       - "ADR-0005: Anchor Internal PKI on the KMS Root with Sibling Intermediates": decisions/0005-anchor-internal-pki-on-kms-root-with-sibling-intermediates.md
       - "ADR-0006: OVN as the Sandbox Network Fabric": decisions/0006-ovn-as-the-sandbox-network-fabric.md
+      - "ADR-0007: Expose the nas01 HDDs as a Clustered Bulk Pool": decisions/0007-expose-nas01-hdds-as-a-clustered-bulk-pool.md
+      - "ADR-0008: Give Service VMs a Dedicated Services VLAN": decisions/0008-give-service-vms-a-dedicated-services-vlan.md
   - Designs:
       - Lab v2 core network: designs/lab-v2-core-network.md
       - "Draft: agentcompute": designs/drafts/agentcompute.md
@@ -61,6 +63,7 @@ nav:
       - Manage the sw-mgmt01 configuration: runbooks/sw-mgmt01-configuration.md
       - Deploy the VyOS gateway configuration: runbooks/vyos-gateway-deployment.md
       - Deploy private image runners: runbooks/private-image-runners.md
+      - Deploy and operate the media stack: runbooks/media-stack.md
 
 plugins:
   - search