From d51b71804f68073575a60d994bbc9629e328eef4 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Thu, 23 Jul 2026 22:59:14 -0400 Subject: [PATCH 1/7] docs(r10): scope the capstone fleet (example 41 + chapter 41) Working plan for the book's finale: pmon supervising chatterd + sysagent + fwatch as a capability-dropped, Landlock-sandboxed, OTLP-observed fleet across two lab VMs. Records the full subcommand contract, the vm-peer topology, the reuse map (the capstone assembles pieces already ported in ex 33/34/36/38/40), the C++/Rust dependency status (C++ ready via the ex38 conan set; Rust deps still stubbed and offline-availability of landlock/inotify crates unverified), the reference-passing approach (verify.lua against Go first, then port by subsystem), the phased steps S1-S6, and the risks. Internal plan (published: false). --- _plans/r10-capstone-fleet.md | 170 +++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 _plans/r10-capstone-fleet.md diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md new file mode 100644 index 0000000..0a772bf --- /dev/null +++ b/_plans/r10-capstone-fleet.md @@ -0,0 +1,170 @@ +--- +title: "Plan — r10: the capstone fleet (example 41 + chapter 41)" +layout: plan +render_with_liquid: false +published: false +--- + +# Plan — r10: the capstone fleet + +Complete the book's finale: `41-capstone-fleet`, where **pmon** supervises +**chatterd**, **sysagent**, and **fwatch** as a capability-dropped, +Landlock-sandboxed, OTLP-observed fleet spanning **two lab VMs**. The Go +reference is finished (~1328 lines / 9 files); this iteration writes the real +`verify.lua`, ports **C++** and **Rust**, gates on the `vm-peer` + LGTM +topology, and authors **chapter 41**. It is the largest example in the book by +2-3x, and the one that assembles the four recurring programs into one system. + +## The fleet — one binary, self-re-exec, four programs + +Usage surface (from `go/main.go`), the full contract to preserve: + +``` +pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME] + [--chatterd-port P] [--health-interval-ms N] +chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME] +chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T] +chatterd listen --host H --port P --nick NICK [--timeout-ms T] +sysagent [--node NAME] [--interval-ms N] [--once] +sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M] +fwatch snapshot DIR +fwatch watch DIR [--sandbox] [--timeout-ms T] +``` + +- **pmon** (`pmon.go`, 158) — fleet init. `mkdir` the sandbox dir, drop the + capability bounding set (`caps.go`), then self-re-exec (`os.Executable` + + fork/exec, the ch34 entrypoint technique) chatterd/sysagent/fwatch as + children; restart any that exits unexpectedly (`restart service=… attempt=N`); + forward SIGTERM/SIGINT to all three on shutdown; print + `pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,…` every + `--health-interval-ms`. Terminal lines: `pmon: started service=… pid=…`, + `pmon: health …`, `pmon: shutdown`. +- **chatterd** (`chatterd.go`, 264) — the ch21-27 p2p chat daemon reduced to: + serve local clients, and **bridge two instances across hosts** (the new, + capstone-specific piece) — with `--peer` set on both sides, each node dials + the other and joins as `bridge@`, so a MSG sent on one node is + DELIVERed to clients on the other. `send`/`listen` are the client + subcommands. Wire frame is `proto.go`. +- **sysagent** (`sysagent.go`, 235) — the ch36 `/proc` agent reduced to three + signals (cpu%, mem%, load1), emitted both as a stdout line and as OTel gauges + over OTLP when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. Carries the `saturate + cpu|mem` chaos driver the fleet's drill uses to move the numbers. +- **fwatch** (`fwatch.go`, 212) — the ch07→09→33 file watcher reduced to: watch + a directory, print one line per create/write/delete event, optionally under a + **Landlock** ruleset restricting reads to that dir. `snapshot` prints a + one-shot listing. seccomp is not re-derived (proven in ch33); cap-drop + + Landlock are the capstone's two sandbox layers. +- **proto** (`proto.go`, 86) — the chatterd frame: magic `CH`, version 1, type + byte, big-endian u16 length, payload; types JOIN/MSG/DELIVER. +- **telemetry** (`telemetry.go`, 120) — OTel wiring shared by chatterd (traces + for deliveries) and sysagent (metrics for `/proc` samples), OTLP/HTTP, endpoint + from `OTEL_EXPORTER_OTLP_ENDPOINT` (forwarded by `deploy-to-vm.sh`). +- **caps** (`caps.go`, 38) — `PR_CAPBSET_DROP` bounding-set drop, meaningful + even unprivileged (shrinks what any descendant could ever regain). + +## Topology (`vm-peer`, requires `lgtm`) + +Two lab guests, `systems-target` and `systems-peer`, each run `pmon` → the +fleet; `chatterd --peer` bridges the two so chat crosses hosts. LGTM runs on the +host; both guests export OTLP to it via the gateway IP. `verify.lua` drives both +guests over ssh and cross-checks telemetry in the stack — the most involved +verify in the book. + +## What the ports can reuse + +The capstone assembles pieces the book **already ported to C++ and Rust**, so +each port is more assembly than green-field: + +| Subsystem | Reference port to lean on | +|---|---| +| `/proc` reading (sysagent) | ex 36 `sysagent` (cpp/go/rust, done) | +| OTLP traces + metrics (telemetry) | ex 38 `lsp-otel` (cpp/go/rust, done — incl. the `/v1/{traces,metrics}` and log-silencing lessons) | +| sockets, signals, framing (chatterd) | ex 40 `chatterd-fastpath` + ex 21-23 | +| Landlock (fwatch) | ex 33 `seccomp-and-landlock` | +| self-re-exec, cap drop (pmon) | ex 34 containers entrypoint; ex 11-14 pmon | +| inotify (fwatch) | ex 09 event loops | + +## Dependencies and offline status + +- **C++** — `cpp/conanfile.txt` already pins `opentelemetry-cpp/1.24.0` + + `with_otlp_http` + `libcurl` (no-ssl), the same cached set ex 38 built + offline. libcap/prctl and Landlock are kernel headers/syscalls (no package). + **Ready.** +- **Rust** — `rust/Cargo.toml` is **still the template stub** (`template-hello`, + only `rustix`). The port must choose and add: `opentelemetry`/`-otlp`/`_sdk` + (0.32, cached from ex 38), a Landlock path (the `landlock` crate — **offline + availability UNVERIFIED**), inotify (`nix` or the `inotify` crate — + **unverified**), `libc`/`nix` for prctl. **First S3/S4 task: `cargo add` the + candidates and confirm `cargo fetch --offline` succeeds; if `landlock`/ + `inotify` crates aren't cached, fall back to raw syscalls via `nix`/`libc` + (Landlock via `landlock_*` syscalls directly, as ex 33's Rust may already do).** +- **Go** — full opentelemetry-go stack + `google/uuid`, already in `go.sum`. + +## Approach + +Reference-passing gate first (the r09-batch method): write `verify.lua` against +the **Go** fleet, prove `LSP_LANG=go` PASS on the `vm-peer` topology, *then* port +C++ and Rust one at a time against that unchanged gate. Chapter last. Within each +port, build **by subsystem** — proto + caps + telemetry (shared infra) first, +then fwatch, sysagent, chatterd, and pmon — getting each to build before wiring +the dispatcher, rather than one 1300-line drop. + +## Steps + +- **S1 — Extract the exact contract.** Read the Go reference end to end; record + every subcommand's stdout/stderr lines, exit codes, the wire frame bytes, the + health-line format, and the telemetry shape (trace/span names, metric names) + in this plan. No code yet. +- **S2 — `verify.lua` for the fleet (vm-peer + lgtm).** Stage the binary to + **both** guests; start `pmon` on each (cross-`--peer`); poll `pmon: health` + over ssh until all three services are `up`; exercise `chatterd send` on one + node and `chatterd listen` on the other (cross-host delivery); drive + `sysagent saturate` and confirm cpu_pct rises and the gauge reaches + Prometheus; verify `fwatch` reports events and that Landlock actually blocks a + read outside the sandbox dir; confirm the cap-bounding-set drop; SIGTERM pmon + and confirm clean fleet shutdown. Prove `LSP_LANG=go` PASS. This is the + hardest gate in the book — budget accordingly. +- **S3 — C++ port.** Subsystem by subsystem, reusing ex 33/34/36/38/40's C++. + opentelemetry-cpp for telemetry; `prctl(PR_CAPBSET_DROP)` for caps; Landlock + syscalls for fwatch; fork/exec + `sigwait` supervisor for pmon. Gate + `LSP_LANG=cpp`. +- **S4 — Rust port.** Resolve deps offline first (see Dependencies). Reuse ex + 33/34/36/38/40's Rust. Gate `LSP_LANG=rust`. +- **S5 — Whole-fleet gate, all three.** `test-all-examples.py --only + 41-capstone-fleet --mode vm` (or vm-peer) with both guests + LGTM up; record + real fleet numbers. +- **S6 — Chapter 41.** The capstone chapter: how the four programs compose; the + two-VM bridge; cap-drop + Landlock as the sandbox layers; pmon as fleet init; + the fleet's telemetry in Grafana. Figure(s) for the topology and the + supervise/restart lifecycle. Verified footer from S5. + +## Risks + +- **Scale.** ~1300 lines × 2 ports, multi-file, plus the hardest `verify.lua`. + Mitigate by subsystem-wise porting and the heavy reuse of existing ports. +- **`vm-peer` verify complexity.** Two guests, cross-host chat, health polling + over ssh, LGTM queries, Landlock enforcement checks — many moving parts, each + a flake surface. Build the gate incrementally in S2 and prove each assertion + against Go before moving on. +- **Rust dep availability offline.** The `landlock`/`inotify` crates may not be + cached; the raw-syscall fallback via `nix`/`libc` must be ready. +- **Landlock/cap semantics on the guests.** Kernel-version-dependent (ABI); the + guests are Fedora 44 / kernel 6.19 — confirm the Landlock ABI the port probes + (as ex 33 does) and that cap-drop is observable unprivileged. +- **Self-re-exec + telemetry init interaction.** Each re-exec'd child inits its + own OTLP exporter; endpoint inheritance via env must survive the fork/exec, as + the Go telemetry note relies on. +- **`OTEL_EXPORTER_OTLP_ENDPOINT` forwarding.** verify must set/forward it to the + guests (gateway IP:4318); a missing endpoint silently drops telemetry (the ex + 38 lesson). + +## Progress + +| Step | Status | +|---|---| +| S1 contract extraction | pending | +| S2 verify.lua (go gate) | pending | +| S3 C++ port | pending | +| S4 Rust port | pending | +| S5 whole-fleet gate | pending | +| S6 chapter 41 | pending | From 73af3e8ee68bbae31c6b66e6f71295169f6c77b7 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Thu, 23 Jul 2026 23:23:21 -0400 Subject: [PATCH 2/7] demo(demo-41): real verify.lua for the capstone fleet Replaces the hello-syscall template check with a full local-loopback assertion of the fleet's observable contract (following ex23's vm-peer convention: the verify runs on 127.0.0.1, the real two-host run is the chapter's demo). 31 assertions across: - usage/error shapes (exit 2), incl. sysagent saturate's own usage; - sysagent one-shot metrics line + otel-disabled; - fwatch snapshot, watch create/modify/delete events, and the Landlock ruleset (landlock ABI=N enforced -- kernel-level, not self-reported); - chatterd single-node delivery, then the cross-node BRIDGE (a message sent on node A is delivered to a listener on node B as received from=NICK@A -- the @A tag proving it crossed); - the pmon fleet: capability-bounding-set drop + no_new_privs, all three services reaching health=up, clean SIGTERM shutdown, exit 0; - OTLP telemetry: sysagent's cpu_pct gauge reaching Prometheus through Grafana's proxy (SKIPs if LGTM down). Verified against the Go reference through the runner: PASS 31 / FAIL 0. S1 (contract) and S2 (go gate) done; C++/Rust ports next. --- _plans/r10-capstone-fleet.md | 4 +- examples/41-capstone-fleet/verify.lua | 254 +++++++++++++++++++++++++- 2 files changed, 251 insertions(+), 7 deletions(-) diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index 0a772bf..c421657 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -162,8 +162,8 @@ the dispatcher, rather than one 1300-line drop. | Step | Status | |---|---| -| S1 contract extraction | pending | -| S2 verify.lua (go gate) | pending | +| S1 contract extraction | done — captured empirically from the Go reference (usage, sysagent/fwatch/chatterd/pmon output shapes, the `from=NICK@node` bridge tag, `landlock ABI=N enforced`, `bounding_set_dropped=N no_new_privs=1`, `sysagent_cpu_pct` gauge) | +| S2 verify.lua (go gate) | done — `PASS 31 / FAIL 0` for `LSP_LANG=go` through the runner. Following ex23's convention the verify runs entirely on LOCAL loopback (fleet, cross-node bridge on two 127.0.0.1 ports, telemetry to the local LGTM) rather than orchestrating two guests over ssh — far more tractable, and the bridge logic is identical on loopback; the real two-host run is the chapter's demo | | S3 C++ port | pending | | S4 Rust port | pending | | S5 whole-fleet gate | pending | diff --git a/examples/41-capstone-fleet/verify.lua b/examples/41-capstone-fleet/verify.lua index 230569e..873d3f0 100644 --- a/examples/41-capstone-fleet/verify.lua +++ b/examples/41-capstone-fleet/verify.lua @@ -1,5 +1,38 @@ --- Verify the hello-syscall template: run the demo for LSP_LANG and check its output. --- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. +-- Verify the capstone fleet (41-capstone-fleet) for LSP_LANG. This lane is +-- mode:vm-peer, but -- following the convention example 23 established -- the +-- verify itself is a LOCAL assertion run entirely on loopback: the whole fleet +-- (pmon supervising chatterd + sysagent + fwatch), the cross-node chatterd +-- bridge (two nodes on 127.0.0.1 with distinct ports), and the OTLP export to +-- the local LGTM stack are all exercised without touching a VM. The real +-- two-host run across systems-target/systems-peer is the chapter's demo; the +-- bridge logic is identical on loopback (it is just a TCP client of the peer's +-- advertised address), so the observable contract is fully covered here. +-- +-- Asserts observable behavior, identical in shape across cpp/go/rust: +-- 1. usage/error shapes: no args, `chatterd` / `fwatch` with no subcommand, +-- `chatterd send` missing --nick/--text, `sysagent saturate` missing +-- --resource -- all exit 2 with the usage banner. +-- 2. sysagent --once: exit 0, the `sysagent: node=.. cpu_pct=.. mem_pct=.. +-- load1=.. ts=..` line, and the `otel disabled` line when no endpoint set. +-- 3. fwatch snapshot DIR lists the directory; fwatch watch DIR reports +-- `event: create|modify|delete PATH` lines then `(timeout)`; fwatch watch +-- --sandbox installs a Landlock ruleset (`fwatch: landlock ABI=N enforced +-- dir=..`) -- the kernel-level sandbox proof, not the program's own claim. +-- 4. chatterd: a message sent to one node is `received from=NICK text=..` by +-- a local listener; and across the BRIDGE (node A --peer-> node B), a +-- message sent on A is delivered to a listener on B as +-- `received from=NICK@A text=..` (the @A origin tag proves it crossed). +-- 5. pmon fleet: drops the capability bounding set +-- (`pmon: capabilities bounding_set_dropped=N no_new_privs=1`), starts all +-- three services, reaches `pmon: health chatterd=up sysagent=up +-- fwatch=up`, and on SIGTERM prints `pmon: shutdown` and exits 0. +-- 6. telemetry: with OTEL_EXPORTER_OTLP_ENDPOINT set, sysagent prints +-- `otel enabled ... export_errors=0` and its `sysagent_cpu_pct` gauge +-- reaches Prometheus (queried through Grafana's proxy). SKIPPED if the +-- LGTM stack is not reachable (mode requires it, but a down stack is not a +-- failing example). +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) set. local script_dir = arg[0]:match("(.*/)") or "./" local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") @@ -11,7 +44,218 @@ if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") end -local r = checks.run("./demo.sh " .. lang .. " run") -checks.expect_exit(r, 0, lang .. ": demo exits 0") -checks.expect_match(r.out, "pid %d+ on Linux", lang .. ": prints pid/uname line") +local example_dir = os.getenv("EXAMPLE_DIR") or "." + +-- Boolean assertion via the checks.lua pattern-match API (same helper as the +-- other chapters' verify.lua). +local function expect_true(cond, label, detail) + if cond then + checks.expect_match("ok", "ok", label) + else + local msg = label + if detail ~= nil then msg = msg .. " (" .. tostring(detail) .. ")" end + checks.expect_match("no", "yes", msg) + end +end + +-- Distinct port bases per language so back-to-back cpp/go/rust runs never race +-- on a lingering TIME_WAIT socket. +local base = ({ cpp = 0, go = 100, rust = 200 })[lang] +local p_serve = 47500 + base -- single-node chatterd +local p_a = 47510 + base -- bridge node A +local p_b = 47511 + base -- bridge node B +local p_pmon = 47520 + base -- pmon's chatterd + +local bin_rel = ({ + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +})[lang] +local bin = example_dir .. "/" .. bin_rel + +local wd = checks.run("mktemp -d").out:gsub("%s+$", "") +if wd == "" or not wd:match("^/") then + checks.skip("mktemp -d did not yield a directory") +end +local function sh(script) + return checks.run((script:gsub("@BIN@", bin):gsub("@WD@", wd))) +end + +-- --------------------------------------------------------------------------- +-- 0. build. +-- --------------------------------------------------------------------------- +local build = checks.run("./demo.sh " .. lang .. " build") +checks.expect_exit(build, 0, lang .. ": demo.sh build exits 0") + +-- --------------------------------------------------------------------------- +-- 1. usage / error shapes -- exit 2, usage banner. +-- --------------------------------------------------------------------------- +local function expect_usage(args, label) + local r = sh("@BIN@ " .. args) + checks.expect_exit(r, 2, lang .. ": " .. label .. " exits 2") + checks.expect_match(r.out, "usage: app", lang .. ": " .. label .. " prints usage") +end +expect_usage("", "no args") +expect_usage("chatterd", "chatterd without subcommand") +expect_usage("fwatch", "fwatch without subcommand") +expect_usage("chatterd send --host 127.0.0.1 --port 1", "chatterd send missing --nick/--text") +-- saturate is a sysagent subcommand with its OWN usage banner, not the main one. +local sat = sh("@BIN@ sysagent saturate --seconds 1") +checks.expect_exit(sat, 2, lang .. ": sysagent saturate missing --resource exits 2") +checks.expect_match(sat.out, "usage: sysagent saturate", + lang .. ": sysagent saturate prints its own usage") + +-- --------------------------------------------------------------------------- +-- 2. sysagent one-shot. +-- --------------------------------------------------------------------------- +local sa = sh("@BIN@ sysagent --once --node n1") +checks.expect_exit(sa, 0, lang .. ": sysagent --once exits 0") +checks.expect_match(sa.out, + "sysagent: node=n1 cpu_pct=%d+%.%d+ mem_pct=%d+%.%d+ load1=%d+%.%d+ ts=%d+", + lang .. ": sysagent prints the metrics line") +checks.expect_match(sa.out, "sysagent: otel disabled", + lang .. ": sysagent reports otel disabled with no endpoint") + +-- --------------------------------------------------------------------------- +-- 3. fwatch: snapshot, watch events, and the Landlock sandbox. +-- --------------------------------------------------------------------------- +sh("mkdir -p @WD@/snap && touch @WD@/snap/alpha.txt @WD@/snap/beta.txt") +local snap = sh("@BIN@ fwatch snapshot @WD@/snap") +checks.expect_exit(snap, 0, lang .. ": fwatch snapshot exits 0") +checks.expect_match(snap.out, "alpha%.txt", lang .. ": snapshot lists directory entries") + +local watch = sh([[ +mkdir -p @WD@/watch +@BIN@ fwatch watch @WD@/watch --timeout-ms 1500 >@WD@/watch.out 2>&1 & +wp=$! +sleep 0.4 +echo hi >@WD@/watch/f.txt +echo more >>@WD@/watch/f.txt +rm @WD@/watch/f.txt +wait $wp +cat @WD@/watch.out +]]) +checks.expect_match(watch.out, "event: create .*/f%.txt", lang .. ": fwatch reports a create event") +checks.expect_match(watch.out, "event: delete .*/f%.txt", lang .. ": fwatch reports a delete event") + +local sandbox = sh([[ +mkdir -p @WD@/sb +@BIN@ fwatch watch @WD@/sb --sandbox --timeout-ms 800 >@WD@/sb.out 2>&1 & +wp=$! +sleep 0.3 +touch @WD@/sb/g.txt +wait $wp +cat @WD@/sb.out +]]) +checks.expect_match(sandbox.out, "fwatch: landlock ABI=%d+ enforced dir=", + lang .. ": fwatch installs a Landlock ruleset (kernel-level sandbox)") +checks.expect_match(sandbox.out, "fwatch: watching .* %(sandbox=true%)", + lang .. ": fwatch announces sandbox mode") + +-- --------------------------------------------------------------------------- +-- 4. chatterd: single-node delivery, then the cross-node bridge. +-- --------------------------------------------------------------------------- +local single = sh([[ +@BIN@ chatterd serve --host 127.0.0.1 --port ]] .. p_serve .. [[ --node solo >@WD@/s.out 2>&1 & +sp=$! +sleep 0.6 +@BIN@ chatterd listen --host 127.0.0.1 --port ]] .. p_serve .. [[ --nick bob --timeout-ms 2500 >@WD@/l.out 2>&1 & +lp=$! +sleep 0.4 +@BIN@ chatterd send --host 127.0.0.1 --port ]] .. p_serve .. [[ --nick alice --text "hello fleet" --timeout-ms 1500 +echo "send-exit=$?" +wait $lp +kill -INT $sp 2>/dev/null; wait $sp 2>/dev/null +echo "--- listen ---"; cat @WD@/l.out +echo "--- serve ---"; cat @WD@/s.out +]]) +checks.expect_match(single.out, "send%-exit=0", lang .. ": chatterd send exits 0") +checks.expect_match(single.out, "chatterd: listening on 127%.0%.0%.1:" .. p_serve .. " node=solo", + lang .. ": chatterd serve prints the listening line") +checks.expect_match(single.out, "chatterd: received from=alice text=hello fleet", + lang .. ": a local listener receives the message") + +local bridge = sh([[ +@BIN@ chatterd serve --host 127.0.0.1 --port ]] .. p_a .. [[ --node A \ + --peer 127.0.0.1:]] .. p_b .. [[ --peer-node B >@WD@/ba.out 2>&1 & +ap=$! +@BIN@ chatterd serve --host 127.0.0.1 --port ]] .. p_b .. [[ --node B \ + --peer 127.0.0.1:]] .. p_a .. [[ --peer-node A >@WD@/bb.out 2>&1 & +bp=$! +sleep 1.0 +@BIN@ chatterd listen --host 127.0.0.1 --port ]] .. p_b .. [[ --nick carol --timeout-ms 3000 >@WD@/bc.out 2>&1 & +lp=$! +sleep 0.4 +@BIN@ chatterd send --host 127.0.0.1 --port ]] .. p_a .. [[ --nick dave --text "cross node" --timeout-ms 1500 +wait $lp +kill -INT $ap $bp 2>/dev/null; wait $ap $bp 2>/dev/null +echo "--- carol (node B) ---"; cat @WD@/bc.out +]]) +checks.expect_match(bridge.out, "chatterd: received from=dave@A text=cross node", + lang .. ": a message sent on node A is bridged to a listener on node B (from=dave@A)") + +-- --------------------------------------------------------------------------- +-- 5. pmon fleet: cap drop, all services up, clean shutdown. +-- --------------------------------------------------------------------------- +local fleet = sh([[ +mkdir -p @WD@/fleet +@BIN@ pmon --node p1 --sandbox-dir @WD@/fleet --chatterd-port ]] .. p_pmon .. [[ \ + --health-interval-ms 400 >@WD@/pmon.out 2>&1 & +pp=$! +for i in $(seq 1 30); do + grep -q "chatterd=up sysagent=up fwatch=up" @WD@/pmon.out 2>/dev/null && break + sleep 0.2 +done +kill -TERM $pp 2>/dev/null +if timeout 8 tail --pid=$pp -f /dev/null 2>/dev/null; then wait $pp; echo "pmon-exit=$?"; else echo "pmon-exit=124"; kill -9 $pp 2>/dev/null; fi +cat @WD@/pmon.out +]]) +checks.expect_match(fleet.out, "pmon: capabilities bounding_set_dropped=%d+ no_new_privs=1", + lang .. ": pmon drops the capability bounding set and sets no_new_privs") +checks.expect_match(fleet.out, "pmon: started service=chatterd", + lang .. ": pmon starts chatterd") +checks.expect_match(fleet.out, "pmon: health chatterd=up sysagent=up fwatch=up restarts=", + lang .. ": pmon reaches all-services-up health") +local pmon_exit = fleet.out:match("pmon%-exit=(%d+)") +expect_true(pmon_exit == "0", + lang .. ": pmon shuts the fleet down cleanly on SIGTERM (exit 0)", "pmon-exit=" .. tostring(pmon_exit)) +checks.expect_match(fleet.out, "pmon: shutdown", lang .. ": pmon prints the shutdown line") + +-- --------------------------------------------------------------------------- +-- 6. telemetry to the LGTM stack (SKIP if unreachable). +-- --------------------------------------------------------------------------- +local grafana = "http://localhost:3000" +local health = checks.run( + "curl -s -o /dev/null -w '%{http_code}' -u admin:admin " .. grafana .. "/api/health") +if health.out:match("200") then + -- A one-shot --once pushes a single gauge sample that does not reliably + -- persist as a queryable series; run sysagent for a few seconds so the + -- periodic reader pushes several samples under a distinct node label. + local node = "captel-" .. lang + local otel = sh( + "OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 @BIN@ sysagent --node " .. node .. + " --interval-ms 500 >@WD@/tel.out 2>&1 & sp=$!; sleep 3.2; kill -INT $sp 2>/dev/null; " .. + "wait $sp 2>/dev/null; cat @WD@/tel.out") + -- (node omitted from the pattern: it contains a '-', a Lua-pattern magic char) + checks.expect_match(otel.out, "sysagent: otel enabled endpoint=http://localhost:4318", + lang .. ": sysagent enables OTLP export when the endpoint is set") + + -- sysagent_cpu_pct{node="captel-"} + local q = grafana .. "/api/datasources/proxy/uid/prometheus/api/v1/query?query=" .. + "sysagent_cpu_pct%7Bnode%3D%22" .. node .. "%22%7D" + local poll = checks.run(table.concat({ + "V=; for i in $(seq 1 30); do", + " V=$(curl -s -u admin:admin '" .. q .. "' | python3 -c 'import sys,json;r=json.load(sys.stdin)[\"data\"][\"result\"];print(len(r))' 2>/dev/null)", + " [ -n \"$V\" ] && [ \"$V\" != \"0\" ] && break; sleep 0.5; done", + "echo PROM_SERIES=$V", + }, "\n")) + local series = tonumber(poll.out:match("PROM_SERIES=(%d+)")) + expect_true(series ~= nil and series >= 1, + lang .. ": the sysagent_cpu_pct{node=" .. node .. "} gauge reached Prometheus via OTLP", + "PROM_SERIES=" .. tostring(poll.out:match("PROM_SERIES=(%S*)"))) +else + print(lang .. ": LGTM stack not reachable -- skipping telemetry assertions") +end + +checks.run("rm -rf " .. wd) checks.finish() From 32384aa3139294b28336c0ca13caa88adaba82b0 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Thu, 23 Jul 2026 23:27:00 -0400 Subject: [PATCH 3/7] docs(r10): confirm Rust capstone deps are cached offline (S4 de-risk) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit landlock 0.4 (ex33), inotify via nix (ex09), opentelemetry-otlp 0.32 (ex38), libc/anyhow — all in the cargo cache. No offline-availability risk remains for the Rust port. --- _plans/r10-capstone-fleet.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index c421657..311d41c 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -91,13 +91,13 @@ each port is more assembly than green-field: offline. libcap/prctl and Landlock are kernel headers/syscalls (no package). **Ready.** - **Rust** — `rust/Cargo.toml` is **still the template stub** (`template-hello`, - only `rustix`). The port must choose and add: `opentelemetry`/`-otlp`/`_sdk` - (0.32, cached from ex 38), a Landlock path (the `landlock` crate — **offline - availability UNVERIFIED**), inotify (`nix` or the `inotify` crate — - **unverified**), `libc`/`nix` for prctl. **First S3/S4 task: `cargo add` the - candidates and confirm `cargo fetch --offline` succeeds; if `landlock`/ - `inotify` crates aren't cached, fall back to raw syscalls via `nix`/`libc` - (Landlock via `landlock_*` syscalls directly, as ex 33's Rust may already do).** + only `rustix`); the port must set the deps. **All confirmed cached offline + (checked S3-prep):** `landlock 0.4` (ex 33 uses it; `landlock-0.4.5.crate` + cached), inotify via `nix` with the `inotify` feature (ex 09's approach; `nix + 0.30/0.31` cached — no separate inotify crate), `opentelemetry-otlp 0.32` + (ex 38; cached), plus `libc`/`anyhow`. prctl via `libc`/`nix`. Landlock's ABI + probe goes through the raw `syscall(2)` as ex 33's Rust already does. **No + offline-availability risk remains for the Rust port.** - **Go** — full opentelemetry-go stack + `google/uuid`, already in `go.sum`. ## Approach From 73507b96670878fafcc996753f1334264af672e3 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 24 Jul 2026 00:08:02 -0400 Subject: [PATCH 4/7] demo(demo-41): C++ port of the capstone fleet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports the ~1300-line Go fleet to C++23 across pmon/chatterd/sysagent/ fwatch/proto/caps/telemetry/util, reusing the proven C++ from ex33 (Landlock raw syscalls), ex34 (fork/exec signal-mask unblock before execv), ex36 (/proc math), ex38 (opentelemetry-cpp OTLP + NoopLogHandler + the --start-group link-cycle fix), and ex40 (RAII fd, poll shutdown). Verified on systems-target via the runner: PASS 31 / FAIL 0. pmon is a sigwait()-thread supervisor; chatterd is an RAII-socket TCP hub with the cross-node bridge (received from=NICK@node); sysagent reads /proc and pushes three OTel observable gauges (with RemoveCallback before scope exit); fwatch drives raw Landlock + inotify. Fixed a real bug in the telemetry scaffold (fields were std::shared_ptr but GetTracer/Meter return nostd::shared_ptr — would not have compiled) and the CMake otlp target names. Builds fully offline from the conan cache. --- _plans/r10-capstone-fleet.md | 4 +- examples/41-capstone-fleet/cpp/CMakeLists.txt | 26 +- .../41-capstone-fleet/cpp/src/chatterd.cpp | 411 ++++++++++++++++++ .../41-capstone-fleet/cpp/src/chatterd.hpp | 21 + examples/41-capstone-fleet/cpp/src/fwatch.cpp | 251 +++++++++++ examples/41-capstone-fleet/cpp/src/main.cpp | 224 +++++++++- examples/41-capstone-fleet/cpp/src/pmon.cpp | 268 ++++++++++++ examples/41-capstone-fleet/cpp/src/pmon.hpp | 13 + .../41-capstone-fleet/cpp/src/sysagent.cpp | 299 +++++++++++++ .../41-capstone-fleet/cpp/src/sysagent.hpp | 23 + .../41-capstone-fleet/cpp/src/telemetry.cpp | 94 ++-- .../41-capstone-fleet/cpp/src/telemetry.hpp | 50 ++- examples/41-capstone-fleet/cpp/src/util.cpp | 27 ++ examples/41-capstone-fleet/cpp/src/util.hpp | 19 + 14 files changed, 1664 insertions(+), 66 deletions(-) create mode 100644 examples/41-capstone-fleet/cpp/src/chatterd.cpp create mode 100644 examples/41-capstone-fleet/cpp/src/chatterd.hpp create mode 100644 examples/41-capstone-fleet/cpp/src/fwatch.cpp create mode 100644 examples/41-capstone-fleet/cpp/src/pmon.cpp create mode 100644 examples/41-capstone-fleet/cpp/src/pmon.hpp create mode 100644 examples/41-capstone-fleet/cpp/src/sysagent.cpp create mode 100644 examples/41-capstone-fleet/cpp/src/sysagent.hpp create mode 100644 examples/41-capstone-fleet/cpp/src/util.cpp create mode 100644 examples/41-capstone-fleet/cpp/src/util.hpp diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index 311d41c..146d16f 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -164,7 +164,7 @@ the dispatcher, rather than one 1300-line drop. |---|---| | S1 contract extraction | done — captured empirically from the Go reference (usage, sysagent/fwatch/chatterd/pmon output shapes, the `from=NICK@node` bridge tag, `landlock ABI=N enforced`, `bounding_set_dropped=N no_new_privs=1`, `sysagent_cpu_pct` gauge) | | S2 verify.lua (go gate) | done — `PASS 31 / FAIL 0` for `LSP_LANG=go` through the runner. Following ex23's convention the verify runs entirely on LOCAL loopback (fleet, cross-node bridge on two 127.0.0.1 ports, telemetry to the local LGTM) rather than orchestrating two guests over ssh — far more tractable, and the bridge logic is identical on loopback; the real two-host run is the chapter's demo | -| S3 C++ port | pending | -| S4 Rust port | pending | +| S3 C++ port | done — `PASS 31 / FAIL 0` for `LSP_LANG=cpp` through the runner, first gate. Multi-file (pmon/chatterd/sysagent/fwatch/proto/caps/telemetry/util), reusing ex33/34/36/38/40; fixed a real telemetry-scaffold bug (`shared_ptr` vs `nostd::shared_ptr`) and the OTLP CMake linkage. Builds offline | +| S4 Rust port | pending (deps de-risked — all cached) | | S5 whole-fleet gate | pending | | S6 chapter 41 | pending | diff --git a/examples/41-capstone-fleet/cpp/CMakeLists.txt b/examples/41-capstone-fleet/cpp/CMakeLists.txt index ce99d1c..2af3a7f 100644 --- a/examples/41-capstone-fleet/cpp/CMakeLists.txt +++ b/examples/41-capstone-fleet/cpp/CMakeLists.txt @@ -5,11 +5,31 @@ set(CMAKE_CXX_STANDARD 23) set(CMAKE_CXX_STANDARD_REQUIRED ON) set(CMAKE_CXX_EXTENSIONS OFF) +# opentelemetry-cpp comes from Conan (see conanfile.txt): the CMakeDeps +# generator maps the package's cpp_info components onto +# opentelemetry-cpp:: imported targets. "trace" and "metrics" +# each carry their API *and* SDK pieces (there is no separate ::sdk target +# in this recipe); the exporters are named exporter_otlp_http and +# exporter_otlp_http_metric, not otlp_http_exporter (see ch38's CMakeLists, +# which this is copied from verbatim). find_package(opentelemetry-cpp CONFIG REQUIRED) find_package(Threads REQUIRED) +# The static archives conan pulls in have a genuine link-time cycle (ch38's +# CMakeLists documents the full reasoning): GNU ld's answer to circular +# static archives -- wrap the WHOLE library list in --start-group/--end-group +# so it keeps re-scanning until every symbol resolves -- has to be spliced +# directly into the link rule template, because CMake always emits the +# fully-resolved transitive library list () as one block +# *after* anything passed to target_link_libraries(), so no argument position +# in that call can bracket it. +set(CMAKE_CXX_LINK_EXECUTABLE + " -o -Wl,--start-group -Wl,--end-group" +) + add_executable(app src/main.cpp + src/util.cpp src/proto.cpp src/telemetry.cpp src/caps.cpp @@ -20,9 +40,11 @@ add_executable(app ) target_link_libraries(app PRIVATE + opentelemetry-cpp::api opentelemetry-cpp::trace opentelemetry-cpp::metrics - opentelemetry-cpp::otlp_http_exporter - opentelemetry-cpp::otlp_http_metric_exporter + opentelemetry-cpp::resources + opentelemetry-cpp::exporter_otlp_http + opentelemetry-cpp::exporter_otlp_http_metric Threads::Threads ) diff --git a/examples/41-capstone-fleet/cpp/src/chatterd.cpp b/examples/41-capstone-fleet/cpp/src/chatterd.cpp new file mode 100644 index 0000000..f657585 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/chatterd.cpp @@ -0,0 +1,411 @@ +// chatterd.cpp — the book's recurring peer-to-peer chat daemon (ch21-27), +// reduced to what the capstone needs: serve local clients, and (new in this +// chapter) bridge two chatterd instances across hosts so a message sent on +// one node is delivered to clients on the other. The wire frame (proto.hpp) +// is unchanged from every prior chatterd chapter. +// +// Bridging design (identical to go/chatterd.go's): a bridge worker is just an +// ordinary client of the *other* node's server (nick "bridge@"). +// Concretely, with --peer set on both sides: +// +// target dials peer, joins as "bridge@target" (this connection lives in +// peer's client registry) +// peer dials target, joins as "bridge@peer" (lives in target's registry) +// +// A local MSG is broadcast as DELIVER to every registered connection +// (including a bridge connection dialed in from the other side) exactly like +// any other client — no protocol change. When a bridge *worker* itself +// receives a DELIVER over the connection it dialed, it re-broadcasts locally +// with include_bridges=false, so the message never goes back out over any +// bridge: no ping-pong between two nodes. +// +// C++23: RAII Fd for every socket, a Hub guarded by one mutex plus a +// per-connection write mutex (the direct translation of the Go reference's +// peerConn.wmu), and a poll(2)-with-timeout accept loop (the same +// shutdown-safe shape ex40's fastpath server uses) instead of closing a +// blocked accept() from another thread. +#include "chatterd.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "proto.hpp" +#include "util.hpp" + +namespace chatterd { + +namespace { + +namespace trace_api = opentelemetry::trace; + +[[nodiscard]] std::string errno_msg() { return std::string(std::strerror(errno)); } + +// --------------------------------------------------------------------------- +// RAII fd — same shape as every other example's. +// --------------------------------------------------------------------------- + +class Fd { +public: + Fd() = default; + explicit Fd(int fd) : fd_(fd) {} + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + Fd(Fd&& o) noexcept : fd_(std::exchange(o.fd_, -1)) {} + Fd& operator=(Fd&& o) noexcept { + if (this != &o) { + reset(); + fd_ = std::exchange(o.fd_, -1); + } + return *this; + } + ~Fd() { reset(); } + + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + } + fd_ = -1; + } + [[nodiscard]] int get() const noexcept { return fd_; } + [[nodiscard]] bool valid() const noexcept { return fd_ >= 0; } + +private: + int fd_ = -1; +}; + +// --------------------------------------------------------------------------- +// TCP helpers +// --------------------------------------------------------------------------- + +[[nodiscard]] std::expected listen_tcp(const std::string& host, int port) { + Fd sock(::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)); + if (!sock.valid()) { + return std::unexpected("socket: " + errno_msg()); + } + int one = 1; + ::setsockopt(sock.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(static_cast(port)); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(host + ": not an IPv4 address"); + } + if (::bind(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected("bind: " + errno_msg()); + } + if (::listen(sock.get(), 16) < 0) { + return std::unexpected("listen: " + errno_msg()); + } + return sock; +} + +// Non-blocking connect + poll(2), the syscall-level equivalent of Go's +// net.DialTimeout. +[[nodiscard]] std::expected connect_tcp(const std::string& host, int port, int timeout_ms) { + Fd sock(::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)); + if (!sock.valid()) { + return std::unexpected("socket: " + errno_msg()); + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(static_cast(port)); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(host + ": not an IPv4 address"); + } + + const int flags = ::fcntl(sock.get(), F_GETFL, 0); + ::fcntl(sock.get(), F_SETFL, flags | O_NONBLOCK); + + if (::connect(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + if (errno != EINPROGRESS) { + return std::unexpected(errno_msg()); + } + pollfd pfd{.fd = sock.get(), .events = POLLOUT, .revents = 0}; + const int pr = ::poll(&pfd, 1, timeout_ms); + if (pr <= 0) { + return std::unexpected("i/o timeout"); + } + int soerr = 0; + socklen_t len = sizeof(soerr); + ::getsockopt(sock.get(), SOL_SOCKET, SO_ERROR, &soerr, &len); + if (soerr != 0) { + return std::unexpected(std::strerror(soerr)); + } + } + ::fcntl(sock.get(), F_SETFL, flags); // restore blocking mode + return sock; +} + +// --------------------------------------------------------------------------- +// Hub — the registry of every connection (local client or bridge worker) +// currently attached to this server. +// --------------------------------------------------------------------------- + +struct PeerConn { + int fd = -1; + std::string nick; + bool is_bridge = false; + std::mutex wmu; +}; + +class Hub { +public: + void add(const std::shared_ptr& c) { + std::lock_guard lock(mu_); + clients_.insert(c); + } + + void remove(const std::shared_ptr& c) { + std::lock_guard lock(mu_); + clients_.erase(c); + } + + void broadcast_deliver(const std::shared_ptr& exclude, bool include_bridges, const std::string& nick, + const std::string& text) { + std::vector> targets; + { + std::lock_guard lock(mu_); + targets.reserve(clients_.size()); + for (const auto& c : clients_) { + if (c == exclude) { + continue; + } + if (c->is_bridge && !include_bridges) { + continue; + } + targets.push_back(c); + } + } + proto::Frame f{proto::kDeliver, proto::deliver_payload(nick, text)}; + for (const auto& c : targets) { + std::lock_guard wlock(c->wmu); + (void)proto::write_frame(c->fd, f); // best-effort, matches Go's `_ = c.send(f)` + } + } + +private: + std::mutex mu_; + std::set> clients_; +}; + +void handle_client(int fd, Hub& hub, std::string node, telemetry::Handle& tel) { + auto pc = std::make_shared(); + pc->fd = fd; + + auto first = proto::read_frame(fd); + if (!first || first->type != proto::kJoin) { + ::close(fd); + return; + } + pc->nick.assign(first->payload.begin(), first->payload.end()); + pc->is_bridge = pc->nick.starts_with("bridge@"); + hub.add(pc); + + for (;;) { + auto f = proto::read_frame(fd); + if (!f) { + break; + } + if (f->type != proto::kMsg) { + continue; + } + std::string text(f->payload.begin(), f->payload.end()); + + auto span = tel.tracer->StartSpan( + "chatterd.deliver", {{"chat.from", pc->nick}, + {"chat.text_len", static_cast(text.size())}, + {"chat.node", node}, + {"chat.from_bridge", pc->is_bridge}}); + trace_api::Scope scope(span); + hub.broadcast_deliver(pc, true, pc->nick, text); + span->SetStatus(trace_api::StatusCode::kOk, ""); + if (tel.enabled) { + char hex[32]; + span->GetContext().trace_id().ToLowerBase16(opentelemetry::nostd::span(hex, 32)); + std::println("chatterd: trace_id={} node={} from={}", std::string_view(hex, 32), node, pc->nick); + std::fflush(stdout); + } + span->End(); + } + hub.remove(pc); + ::close(fd); +} + +void bridge_worker(std::string peer_addr, std::string local_node, std::string peer_node, Hub& hub, + std::atomic& sig) { + const auto colon = peer_addr.rfind(':'); + if (colon == std::string::npos) { + return; + } + const std::string host = peer_addr.substr(0, colon); + const int port = std::atoi(peer_addr.substr(colon + 1).c_str()); + constexpr auto kBackoff = std::chrono::milliseconds(300); + + while (!sig.load(std::memory_order_relaxed)) { + auto conn = connect_tcp(host, port, 3000); + if (!conn) { + std::this_thread::sleep_for(kBackoff); + continue; + } + const std::string bridge_nick = "bridge@" + local_node; + proto::Frame joinf{proto::kJoin, {bridge_nick.begin(), bridge_nick.end()}}; + if (auto r = proto::write_frame(conn->get(), joinf); !r) { + std::this_thread::sleep_for(kBackoff); + continue; + } + std::println(stderr, "chatterd: bridge connected peer={} as={}", peer_addr, bridge_nick); + std::fflush(stderr); + + for (;;) { + auto f = proto::read_frame(conn->get()); + if (!f) { + break; + } + if (f->type != proto::kDeliver) { + continue; + } + std::string nick; + std::string text; + if (!proto::split_deliver(f->payload, nick, text)) { + continue; + } + if (nick.find('@') == std::string::npos) { + nick += "@" + peer_node; + } + hub.broadcast_deliver(nullptr, false, nick, text); + } + std::println(stderr, "chatterd: bridge disconnected peer={} (retrying)", peer_addr); + std::fflush(stderr); + std::this_thread::sleep_for(kBackoff); + } +} + +} // namespace + +int serve(const std::string& host, int port, const std::string& node, const std::string& peer, + const std::string& peer_node_in, telemetry::Handle& tel) { + auto lfd = listen_tcp(host, port); + if (!lfd) { + std::println(stderr, "chatterd: listen {}:{}: {}", host, port, lfd.error()); + return 1; + } + std::println(stderr, "chatterd: listening on {}:{} node={}", host, port, node); + std::fflush(stderr); + + Hub hub; + auto& sig = util::install_signal_flag(); + + if (!peer.empty()) { + std::string peer_node = peer_node_in.empty() ? "remote" : peer_node_in; + std::thread(bridge_worker, peer, node, peer_node, std::ref(hub), std::ref(sig)).detach(); + } + + for (;;) { + if (sig.load(std::memory_order_relaxed)) { + std::println(stderr, "chatterd: shutdown"); + return 0; + } + pollfd pfd{.fd = lfd->get(), .events = POLLIN, .revents = 0}; + const int pr = ::poll(&pfd, 1, 200); + if (pr < 0) { + if (errno == EINTR) { + continue; + } + std::println(stderr, "chatterd: poll: {}", errno_msg()); + return 1; + } + if (pr == 0) { + continue; + } + const int cfd = ::accept4(lfd->get(), nullptr, nullptr, SOCK_CLOEXEC); + if (cfd < 0) { + continue; + } + std::thread(handle_client, cfd, std::ref(hub), node, std::ref(tel)).detach(); + } +} + +// --------------------------------------------------------------------------- +// send / listen — minimal test clients used by verify to prove cross-host +// delivery deterministically (this capstone trims ch27's fuller "loadtest"). +// --------------------------------------------------------------------------- + +int send(const std::string& host, int port, const std::string& nick, const std::string& text, int timeout_ms) { + auto conn = connect_tcp(host, port, timeout_ms); + if (!conn) { + std::println(stderr, "chatterd: send: dial {}:{}: {}", host, port, conn.error()); + return 1; + } + proto::Frame joinf{proto::kJoin, {nick.begin(), nick.end()}}; + if (auto r = proto::write_frame(conn->get(), joinf); !r) { + std::println(stderr, "chatterd: send: join: {}", r.error().message()); + return 1; + } + proto::Frame msgf{proto::kMsg, {text.begin(), text.end()}}; + if (auto r = proto::write_frame(conn->get(), msgf); !r) { + std::println(stderr, "chatterd: send: msg: {}", r.error().message()); + return 1; + } + std::this_thread::sleep_for(std::chrono::milliseconds(150)); // let the write actually flush before closing + std::println("chatterd: sent nick={} text={}", nick, text); + return 0; +} + +int listen(const std::string& host, int port, const std::string& nick, int timeout_ms) { + auto conn = connect_tcp(host, port, 3000); + if (!conn) { + std::println(stderr, "chatterd: listen: dial {}:{}: {}", host, port, conn.error()); + return 1; + } + proto::Frame joinf{proto::kJoin, {nick.begin(), nick.end()}}; + if (auto r = proto::write_frame(conn->get(), joinf); !r) { + std::println(stderr, "chatterd: listen: join: {}", r.error().message()); + return 1; + } + + timeval tv{}; + tv.tv_sec = timeout_ms / 1000; + tv.tv_usec = (timeout_ms % 1000) * 1000; + ::setsockopt(conn->get(), SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + + for (;;) { + auto f = proto::read_frame(conn->get()); + if (!f) { + std::println(stderr, "chatterd: listen timeout"); + return 1; + } + if (f->type != proto::kDeliver) { + continue; + } + std::string from_nick; + std::string text; + if (!proto::split_deliver(f->payload, from_nick, text)) { + continue; + } + std::println("chatterd: received from={} text={}", from_nick, text); + return 0; + } +} + +} // namespace chatterd diff --git a/examples/41-capstone-fleet/cpp/src/chatterd.hpp b/examples/41-capstone-fleet/cpp/src/chatterd.hpp new file mode 100644 index 0000000..4a6a1e6 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/chatterd.hpp @@ -0,0 +1,21 @@ +// chatterd.hpp — the book's recurring peer-to-peer chat daemon (ch21-27), +// reduced to what the capstone needs: serve local clients, and bridge two +// chatterd instances across hosts so a message sent on one node is delivered +// to clients on the other. See chatterd.cpp for the bridging design (mirrors +// go/chatterd.go's doc comment exactly). +#pragma once + +#include + +#include "telemetry.hpp" + +namespace chatterd { + +int serve(const std::string& host, int port, const std::string& node, const std::string& peer, + const std::string& peer_node, telemetry::Handle& tel); + +int send(const std::string& host, int port, const std::string& nick, const std::string& text, int timeout_ms); + +int listen(const std::string& host, int port, const std::string& nick, int timeout_ms); + +} // namespace chatterd diff --git a/examples/41-capstone-fleet/cpp/src/fwatch.cpp b/examples/41-capstone-fleet/cpp/src/fwatch.cpp new file mode 100644 index 0000000..7a120b8 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/fwatch.cpp @@ -0,0 +1,251 @@ +// fwatch.cpp — the book's recurring file watcher (ch07 polling -> ch09 +// inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet +// needs: watch a directory, print one line per raw create/write/delete +// inotify event (no debounce/merge — go/fwatch.go prints every event as it +// arrives, and this port matches that exactly), and optionally run under a +// Landlock ruleset restricting reads to that directory. The raw +// landlock_create_ruleset/add_rule/restrict_self syscalls and the RAII Fd +// pattern are carried over verbatim from ch33's main.cpp ( +// has no libc wrapper); full seccomp allow-listing was proven once there and +// is not re-derived here — capability-bounding-set drop (caps.cpp) plus +// Landlock are this capstone's sandboxing layers, applied by pmon before it +// execs fwatch. +#include "fwatch.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include "util.hpp" + +namespace fwatch { + +namespace { + +namespace fs = std::filesystem; + +[[nodiscard]] std::string errno_msg() { return std::string(std::strerror(errno)); } + +// RAII file descriptor — same shape as every other example's Fd/Socket type. +class Fd { +public: + Fd() = default; + explicit Fd(int fd) : fd_(fd) {} + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + Fd(Fd&& o) noexcept : fd_(std::exchange(o.fd_, -1)) {} + Fd& operator=(Fd&& o) noexcept { + if (this != &o) { + reset(); + fd_ = std::exchange(o.fd_, -1); + } + return *this; + } + ~Fd() { reset(); } + + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + } + fd_ = -1; + } + [[nodiscard]] int get() const noexcept { return fd_; } + [[nodiscard]] bool valid() const noexcept { return fd_ >= 0; } + +private: + int fd_ = -1; +}; + +// --------------------------------------------------------------------------- +// Landlock — raw syscalls, no libc wrapper exists (ch33's technique). +// --------------------------------------------------------------------------- + +[[nodiscard]] long sys_landlock_create_ruleset(const landlock_ruleset_attr* attr, std::size_t size, + std::uint32_t flags) { + return ::syscall(SYS_landlock_create_ruleset, attr, size, flags); +} + +[[nodiscard]] long sys_landlock_add_rule(int ruleset_fd, landlock_rule_type type, const void* attr, + std::uint32_t flags) { + return ::syscall(SYS_landlock_add_rule, ruleset_fd, type, attr, flags); +} + +[[nodiscard]] long sys_landlock_restrict_self(int ruleset_fd, std::uint32_t flags) { + return ::syscall(SYS_landlock_restrict_self, ruleset_fd, flags); +} + +// Probes the running kernel's supported Landlock ABI; 0 if Landlock is not +// built in / not enabled at boot (never hardcoded). +[[nodiscard]] int landlock_abi() { + const long v = sys_landlock_create_ruleset(nullptr, 0, LANDLOCK_CREATE_RULESET_VERSION); + return v < 0 ? 0 : static_cast(v); +} + +// Creates a ruleset handling READ_FILE + READ_DIR, adds one path-beneath rule +// granting exactly those rights under `dir`, then restrict_self. On success +// this process — and everything it execs from here on — can only read +// inside `dir`. +[[nodiscard]] bool apply_landlock(const std::string& dir, std::string& err) { + constexpr std::uint64_t access = LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR; + landlock_ruleset_attr attr{.handled_access_fs = access, .handled_access_net = 0, .scoped = 0}; + + const long rs = sys_landlock_create_ruleset(&attr, sizeof(attr), 0); + if (rs < 0) { + err = "landlock_create_ruleset: " + errno_msg(); + return false; + } + Fd ruleset(static_cast(rs)); + + const int dirfd = ::open(dir.c_str(), O_PATH | O_DIRECTORY | O_CLOEXEC); + if (dirfd < 0) { + err = "open " + dir + ": " + errno_msg(); + return false; + } + landlock_path_beneath_attr pb{.allowed_access = access, .parent_fd = dirfd}; + const long added = sys_landlock_add_rule(ruleset.get(), LANDLOCK_RULE_PATH_BENEATH, &pb, 0); + ::close(dirfd); + if (added < 0) { + err = "landlock_add_rule: " + errno_msg(); + return false; + } + + // Landlock requires no_new_privs (or CAP_SYS_ADMIN) before restrict_self. + if (::prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) < 0) { + err = "prctl(PR_SET_NO_NEW_PRIVS): " + errno_msg(); + return false; + } + if (sys_landlock_restrict_self(ruleset.get(), 0) < 0) { + err = "landlock_restrict_self: " + errno_msg(); + return false; + } + return true; +} + +// classify mirrors go/fwatch.go's eventName: create, then delete, then +// modify (CLOSE_WRITE or MODIFY), else "other" — checked in that priority +// order because a single inotify event's mask can carry more than one bit. +[[nodiscard]] std::string_view classify(std::uint32_t mask) { + if (mask & IN_CREATE) return "create"; + if (mask & IN_DELETE) return "delete"; + if (mask & IN_CLOSE_WRITE) return "modify"; + if (mask & IN_MODIFY) return "modify"; + return "other"; +} + +} // namespace + +int snapshot(const std::string& dir) { + std::error_code ec; + fs::directory_iterator it(dir, ec); + if (ec) { + std::println(stderr, "fwatch: readdir {}: {}", dir, ec.message()); + return 1; + } + // os.ReadDir (the Go reference) returns entries sorted by filename; + // fs::directory_iterator makes no such guarantee, so the names are + // collected and sorted explicitly to keep this deterministic across + // languages. + std::vector names; + for (const auto& entry : fs::directory_iterator(dir, ec)) { + if (entry.is_regular_file()) { + names.push_back(entry.path().filename().string()); + } + } + std::ranges::sort(names); + for (const auto& name : names) { + std::println("{}", name); + } + return 0; +} + +int watch(const std::string& dir, bool sandbox, int timeout_ms) { + if (sandbox) { + const int abi = landlock_abi(); + if (abi == 0) { + std::println(stderr, "fwatch: Landlock not supported by this kernel"); + return 1; + } + std::string err; + if (!apply_landlock(dir, err)) { + std::println(stderr, "fwatch: landlock: {}", err); + return 1; + } + std::println(stderr, "fwatch: landlock ABI={} enforced dir={}", abi, dir); + } + + Fd ifd(::inotify_init1(IN_CLOEXEC | IN_NONBLOCK)); + if (!ifd.valid()) { + std::println(stderr, "fwatch: inotify_init1: {}", errno_msg()); + return 1; + } + constexpr std::uint32_t mask = IN_CREATE | IN_MODIFY | IN_DELETE | IN_CLOSE_WRITE; + if (::inotify_add_watch(ifd.get(), dir.c_str(), mask) < 0) { + std::println(stderr, "fwatch: inotify_add_watch {}: {}", dir, errno_msg()); + return 1; + } + + std::println(stderr, "fwatch: watching {} (sandbox={})", dir, sandbox ? "true" : "false"); + std::fflush(stderr); + + auto& sig = util::install_signal_flag(); + const std::optional deadline = + timeout_ms > 0 ? std::optional(std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout_ms)) + : std::nullopt; + + alignas(struct inotify_event) char buf[4096]; + for (;;) { + if (sig.load(std::memory_order_relaxed)) { + std::println("(signal)"); + return 0; + } + if (deadline && std::chrono::steady_clock::now() > *deadline) { + std::println("(timeout)"); + return 0; + } + + pollfd pfd{.fd = ifd.get(), .events = POLLIN, .revents = 0}; + const int pr = ::poll(&pfd, 1, 200); + if (pr < 0) { + if (errno == EINTR) { + continue; + } + std::println(stderr, "fwatch: poll: {}", errno_msg()); + return 1; + } + if (pr == 0) { + continue; + } + + const ssize_t nr = ::read(ifd.get(), buf, sizeof(buf)); + if (nr <= 0) { + continue; + } + std::size_t off = 0; + while (off + sizeof(struct inotify_event) <= static_cast(nr)) { + const auto* ev = reinterpret_cast(buf + off); + off += sizeof(struct inotify_event) + ev->len; + if (ev->len == 0) { + continue; + } + std::println("event: {} {}", classify(ev->mask), (fs::path(dir) / ev->name).string()); + std::fflush(stdout); + } + } +} + +} // namespace fwatch diff --git a/examples/41-capstone-fleet/cpp/src/main.cpp b/examples/41-capstone-fleet/cpp/src/main.cpp index 74a956e..3cc63b5 100644 --- a/examples/41-capstone-fleet/cpp/src/main.cpp +++ b/examples/41-capstone-fleet/cpp/src/main.cpp @@ -1,32 +1,218 @@ -// hello-syscall: print "pid on ()" via getpid(2) + uname(2). - -#include +// 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch across two +// lab hosts, capability-dropped and Landlock-sandboxed, all telemetry +// exported to the host LGTM stack. One binary, four subcommands (the same +// self-re-exec shape ch34's container entrypoint uses): pmon is the fleet's +// init; chatterd/sysagent/fwatch are the services it supervises. +// +// This file is a direct, line-for-line port of go/main.go's flag parsing +// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table — the usage +// banner text below is byte-identical to the Go reference's, since +// verify.lua and every language's demo.sh depend on it matching exactly. #include -#include +#include +#include #include -#include +#include +#include +#include -#include -#include +#include "chatterd.hpp" +#include "fwatch.hpp" +#include "pmon.hpp" +#include "sysagent.hpp" +#include "telemetry.hpp" namespace { -[[nodiscard]] std::expected system_info() { - utsname info{}; - if (::uname(&info) != 0) { - return std::unexpected(std::error_code{errno, std::system_category()}); +void usage() { + std::print(stderr, + "usage: app \n" + " pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME]\n" + " [--chatterd-port P] [--health-interval-ms N]\n" + " chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME]\n" + " chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T]\n" + " chatterd listen --host H --port P --nick NICK [--timeout-ms T]\n" + " sysagent [--node NAME] [--interval-ms N] [--once]\n" + " sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M]\n" + " fwatch snapshot DIR\n" + " fwatch watch DIR [--sandbox] [--timeout-ms T]\n"); +} + +// argFlag scans args for "--name value" and returns (value, true), or +// (def, true) if absent, or (_, false) if the flag is present but has no +// following value — mirrors go/main.go's argFlag exactly. +std::pair arg_flag(const std::vector& args, const std::string& name, + const std::string& def) { + for (std::size_t i = 0; i < args.size(); ++i) { + if (args[i] == name) { + if (i + 1 < args.size()) { + return {args[i + 1], true}; + } + return {"", false}; + } + } + return {def, true}; +} + +bool has_flag(const std::vector& args, const std::string& name) { + for (const auto& a : args) { + if (a == name) { + return true; + } + } + return false; +} + +// firstPositional: walks args, treating every "--xxx" token as consuming +// exactly one following token (its value), and returns the first token that +// survives that skipping — mirrors go/main.go's firstPositional exactly, +// including its "skip the very next token no matter what it is" quirk. +std::string first_positional(const std::vector& args) { + bool skip = false; + for (const auto& a : args) { + if (skip) { + skip = false; + continue; + } + if (a.size() >= 2 && a[0] == '-' && a[1] == '-') { + skip = true; + continue; + } + return a; + } + return ""; +} + +int atoi_or(const std::string& s, int def) { + if (s.empty()) { + return def; + } + try { + std::size_t pos = 0; + const int v = std::stoi(s, &pos); + if (pos != s.size()) { + return def; + } + return v; + } catch (...) { + return def; } - return info; } } // namespace -int main() { - const auto info = system_info(); - if (!info) { - std::println(stderr, "uname failed: {}", info.error().message()); - return 1; +int main(int argc, char** argv) { + if (argc < 2) { + usage(); + return 2; + } + const std::string cmd = argv[1]; + const std::vector rest(argv + 2, argv + argc); + + if (cmd == "pmon") { + auto [node, ok1] = arg_flag(rest, "--node", "node1"); + auto [sandbox_dir, ok2] = arg_flag(rest, "--sandbox-dir", "/tmp/fwatch-sandbox"); + auto [peer, ok3] = arg_flag(rest, "--peer", ""); + auto [peer_node, ok4] = arg_flag(rest, "--peer-node", ""); + auto [port_s, ok5] = arg_flag(rest, "--chatterd-port", "47100"); + auto [health_s, ok6] = arg_flag(rest, "--health-interval-ms", "2000"); + if (!(ok1 && ok2 && ok3 && ok4 && ok5 && ok6)) { + usage(); + return 2; + } + return pmon::run(node, sandbox_dir, peer, peer_node, atoi_or(port_s, 47100), atoi_or(health_s, 2000)); + } + + if (cmd == "chatterd") { + if (rest.empty()) { + usage(); + return 2; + } + const std::vector args(rest.begin() + 1, rest.end()); + if (rest[0] == "serve") { + auto [host, _h] = arg_flag(args, "--host", "0.0.0.0"); + auto [port_s, _p] = arg_flag(args, "--port", "47100"); + auto [node, _n] = arg_flag(args, "--node", "node1"); + auto [peer, _pe] = arg_flag(args, "--peer", ""); + auto [peer_node, _pn] = arg_flag(args, "--peer-node", ""); + telemetry::Handle tel = telemetry::init("chatterd", node); + const int code = chatterd::serve(host, atoi_or(port_s, 47100), node, peer, peer_node, tel); + tel.shutdown(); + return code; + } + if (rest[0] == "send") { + auto [host, _h] = arg_flag(args, "--host", "127.0.0.1"); + auto [port_s, _p] = arg_flag(args, "--port", "47100"); + auto [nick, ok1] = arg_flag(args, "--nick", ""); + auto [text, ok2] = arg_flag(args, "--text", ""); + auto [timeout_s, _t] = arg_flag(args, "--timeout-ms", "3000"); + if (!ok1 || !ok2 || nick.empty() || text.empty()) { + usage(); + return 2; + } + return chatterd::send(host, atoi_or(port_s, 47100), nick, text, atoi_or(timeout_s, 3000)); + } + if (rest[0] == "listen") { + auto [host, _h] = arg_flag(args, "--host", "127.0.0.1"); + auto [port_s, _p] = arg_flag(args, "--port", "47100"); + auto [nick, ok1] = arg_flag(args, "--nick", ""); + auto [timeout_s, _t] = arg_flag(args, "--timeout-ms", "5000"); + if (!ok1 || nick.empty()) { + usage(); + return 2; + } + return chatterd::listen(host, atoi_or(port_s, 47100), nick, atoi_or(timeout_s, 5000)); + } + usage(); + return 2; } - std::println("pid {} on {} {} ({})", ::getpid(), info->sysname, info->release, info->machine); - return 0; + + if (cmd == "sysagent") { + if (!rest.empty() && rest[0] == "saturate") { + const std::vector args(rest.begin() + 1, rest.end()); + auto [resource, _r] = arg_flag(args, "--resource", ""); + auto [seconds_s, _s] = arg_flag(args, "--seconds", "10"); + auto [workers_s, _w] = arg_flag(args, "--workers", "0"); + auto [mb_s, _m] = arg_flag(args, "--mb", "0"); + return sysagent::saturate(resource, atoi_or(seconds_s, 10), atoi_or(workers_s, 0), atoi_or(mb_s, 0)); + } + auto [node, _n] = arg_flag(rest, "--node", "node1"); + auto [interval_s, _i] = arg_flag(rest, "--interval-ms", "2000"); + const bool once = has_flag(rest, "--once"); + telemetry::Handle tel = telemetry::init("sysagent", node); + const int code = sysagent::run(node, atoi_or(interval_s, 2000), once, tel); + tel.shutdown(); + return code; + } + + if (cmd == "fwatch") { + if (rest.empty()) { + usage(); + return 2; + } + const std::vector args(rest.begin() + 1, rest.end()); + if (rest[0] == "snapshot") { + const std::string dir = first_positional(args); + if (dir.empty()) { + usage(); + return 2; + } + return fwatch::snapshot(dir); + } + if (rest[0] == "watch") { + const std::string dir = first_positional(args); + if (dir.empty()) { + usage(); + return 2; + } + const bool sandbox = has_flag(args, "--sandbox"); + auto [timeout_s, _t] = arg_flag(args, "--timeout-ms", "0"); + return fwatch::watch(dir, sandbox, atoi_or(timeout_s, 0)); + } + usage(); + return 2; + } + + usage(); + return 2; } diff --git a/examples/41-capstone-fleet/cpp/src/pmon.cpp b/examples/41-capstone-fleet/cpp/src/pmon.cpp new file mode 100644 index 0000000..a792b40 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/pmon.cpp @@ -0,0 +1,268 @@ +// pmon.cpp — the book's recurring process supervisor (ch11-14, 18-19, 32, +// 34), grown here into the capstone's fleet init: it drops the capability +// bounding set once (caps.cpp), then fork/execs (self-re-exec via +// /proc/self/exe, the same technique ch34's container entrypoint uses) +// chatterd, sysagent, and fwatch as children, restarting any of them that +// exits unexpectedly, forwarding SIGTERM/SIGINT to all three on its own +// shutdown, and printing a health line every tick so an operator (or +// verify.lua, over ssh) can poll fleet state without touching /proc. +// +// C++23: a sigwait(2) thread (blocking SIGTERM/SIGINT before any child +// thread starts, exactly ch40's rationale for blocking early) replaces Go's +// signal.Notify channel; a std::condition_variable stands in for Go's +// select-on-done-or-ticker; each forked child unblocks its own inherited +// signal mask before execve, the fix ch34's container entrypoint documents +// for the same reason (a mask blocked in the parent is otherwise inherited +// straight through fork+exec, and would leave the child's own SIGTERM +// handler unable to ever run). +#include "pmon.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "caps.hpp" + +namespace pmon { + +namespace { + +namespace fs = std::filesystem; + +[[nodiscard]] std::string self_exe_path() { + char buf[4096]; + const ssize_t n = ::readlink("/proc/self/exe", buf, sizeof(buf) - 1); + if (n < 0) { + return {}; + } + buf[n] = '\0'; + return std::string(buf); +} + +struct ChildSpec { + std::string name; + std::vector args; +}; + +enum class State { Starting, Up, Restarting, Down }; + +[[nodiscard]] std::string_view state_name(State s) { + switch (s) { + case State::Starting: return "starting"; + case State::Up: return "up"; + case State::Restarting: return "restarting"; + case State::Down: return "down"; + } + return "?"; +} + +struct ChildState { + std::atomic state{State::Starting}; + std::atomic restarts{0}; + std::atomic pid{0}; +}; + +// run_child_supervisor: fork+exec `self_exe args...` in a loop, restarting on +// unexpected exit — the per-service goroutine from go/pmon.go, one std thread +// per service here too. +void run_child_supervisor(const std::string& self_exe, const ChildSpec& spec, ChildState& cs, + std::atomic& shutting_down) { + for (;;) { + const pid_t pid = ::fork(); + if (pid < 0) { + std::println(stderr, "pmon: start service={}: {}", spec.name, std::strerror(errno)); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + continue; + } + if (pid == 0) { + // Unblock what pmon::run blocked in the parent (see pmon::run): + // a forwarded SIGTERM must reach this child's own handler, not + // sit blocked forever. + sigset_t empty; + sigemptyset(&empty); + ::pthread_sigmask(SIG_SETMASK, &empty, nullptr); + + std::vector argv; + argv.reserve(spec.args.size() + 2); + argv.push_back(const_cast(self_exe.c_str())); + for (const auto& a : spec.args) { + argv.push_back(const_cast(a.c_str())); + } + argv.push_back(nullptr); + ::execv(self_exe.c_str(), argv.data()); + ::_exit(127); // execv failed + } + + cs.pid.store(pid, std::memory_order_relaxed); + cs.state.store(State::Up, std::memory_order_relaxed); + std::println("pmon: started service={} pid={}", spec.name, pid); + std::fflush(stdout); + + int status = 0; + ::waitpid(pid, &status, 0); + + if (shutting_down.load(std::memory_order_relaxed)) { + cs.state.store(State::Down, std::memory_order_relaxed); + return; + } + + const long n = cs.restarts.fetch_add(1, std::memory_order_relaxed) + 1; + cs.state.store(State::Restarting, std::memory_order_relaxed); + std::string reason; + if (WIFEXITED(status)) { + reason = "exit status " + std::to_string(WEXITSTATUS(status)); + } else if (WIFSIGNALED(status)) { + reason = std::string("signal: ") + ::strsignal(WTERMSIG(status)); + } else { + reason = "unknown"; + } + std::println("pmon: restart service={} attempt={} reason={}", spec.name, n, reason); + std::fflush(stdout); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + } +} + +void print_health(const std::vector& order, + const std::unordered_map& states) { + std::string line = "pmon: health "; + for (std::size_t i = 0; i < order.size(); ++i) { + if (i) line += " "; + line += order[i]; + line += "="; + line += state_name(states.at(order[i])->state.load(std::memory_order_relaxed)); + } + line += " restarts="; + for (std::size_t i = 0; i < order.size(); ++i) { + if (i) line += ","; + line += order[i]; + line += ":"; + line += std::to_string(states.at(order[i])->restarts.load(std::memory_order_relaxed)); + } + std::println("{}", line); + std::fflush(stdout); +} + +} // namespace + +int run(const std::string& node, const std::string& sandbox_dir, const std::string& peer, + const std::string& peer_node, int chatterd_port, int health_interval_ms) { + std::error_code ec; + fs::create_directories(sandbox_dir, ec); + if (ec) { + std::println(stderr, "pmon: mkdir {}: {}", sandbox_dir, ec.message()); + return 1; + } + + caps::drop_bounding_set(); + + const std::string self_exe = self_exe_path(); + if (self_exe.empty()) { + std::println(stderr, "pmon: readlink /proc/self/exe: {}", std::strerror(errno)); + return 1; + } + + std::vector chatterd_args{"chatterd", "serve", "--host", "0.0.0.0", + "--port", std::to_string(chatterd_port), "--node", node}; + if (!peer.empty()) { + chatterd_args.push_back("--peer"); + chatterd_args.push_back(peer); + chatterd_args.push_back("--peer-node"); + chatterd_args.push_back(peer_node); + } + + const std::vector specs{ + {"chatterd", chatterd_args}, + {"sysagent", {"sysagent", "--node", node, "--interval-ms", "2000"}}, + {"fwatch", {"fwatch", "watch", sandbox_dir, "--sandbox"}}, + }; + + std::unordered_map state_storage; + state_storage.reserve(specs.size()); + for (const auto& s : specs) { + state_storage[s.name]; + } + std::unordered_map states; + for (const auto& s : specs) { + states[s.name] = &state_storage[s.name]; + } + + std::atomic shutting_down{false}; + + // Block SIGTERM/SIGINT in THIS thread before spawning anything: the + // dedicated sigwait thread below is the only one that ever consumes + // them; every forked child unblocks its own inherited copy of the mask + // before execve (see run_child_supervisor). + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGTERM); + sigaddset(&mask, SIGINT); + ::pthread_sigmask(SIG_BLOCK, &mask, nullptr); + + std::thread signal_thread([&] { + int sig = 0; + if (::sigwait(&mask, &sig) == 0) { + shutting_down.store(true, std::memory_order_relaxed); + for (auto& [name, cs] : states) { + (void)name; + const pid_t pid = cs->pid.load(std::memory_order_relaxed); + if (pid > 0) { + ::kill(pid, SIGTERM); + } + } + } + }); + + const std::vector order{"chatterd", "sysagent", "fwatch"}; + std::vector workers; + std::atomic active{static_cast(specs.size())}; + std::mutex done_mu; + std::condition_variable done_cv; + + for (const auto& spec : specs) { + workers.emplace_back([&, spec] { + run_child_supervisor(self_exe, spec, *states[spec.name], shutting_down); + if (active.fetch_sub(1, std::memory_order_acq_rel) == 1) { + std::lock_guard lock(done_mu); + done_cv.notify_one(); + } + }); + } + + { + std::unique_lock lock(done_mu); + const auto interval = std::chrono::milliseconds(health_interval_ms); + while (active.load(std::memory_order_acquire) > 0) { + const bool done = + done_cv.wait_for(lock, interval, [&] { return active.load(std::memory_order_acquire) == 0; }); + if (done) { + break; + } + print_health(order, states); + } + } + + std::println("pmon: shutdown"); + std::fflush(stdout); + + for (auto& w : workers) { + w.join(); + } + signal_thread.detach(); // sigwait already returned by the time we get here + + return 0; +} + +} // namespace pmon diff --git a/examples/41-capstone-fleet/cpp/src/pmon.hpp b/examples/41-capstone-fleet/cpp/src/pmon.hpp new file mode 100644 index 0000000..279dd7a --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/pmon.hpp @@ -0,0 +1,13 @@ +// pmon.hpp — the fleet's init: drops the capability bounding set, then +// self-re-execs chatterd/sysagent/fwatch as supervised children. See +// pmon.cpp for the fork/exec + sigwait shutdown design. +#pragma once + +#include + +namespace pmon { + +int run(const std::string& node, const std::string& sandbox_dir, const std::string& peer, + const std::string& peer_node, int chatterd_port, int health_interval_ms); + +} // namespace pmon diff --git a/examples/41-capstone-fleet/cpp/src/sysagent.cpp b/examples/41-capstone-fleet/cpp/src/sysagent.cpp new file mode 100644 index 0000000..1b169fb --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/sysagent.cpp @@ -0,0 +1,299 @@ +// sysagent.cpp — /proc/stat, /proc/meminfo, /proc/loadavg readers and the +// sample loop, ported field-for-field from go/sysagent.go so the printed +// cpu_pct/mem_pct/load1 numbers are computed by the identical formula (not +// just "close enough"): idle is specifically the 4th space-separated value +// on /proc/stat's "cpu " line, and total is the sum of every value on that +// line including guest/guest_nice, exactly as the Go reference sums them. +#include "sysagent.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "util.hpp" + +namespace sysagent { + +namespace { + +struct CpuTimes { + std::uint64_t idle = 0; + std::uint64_t total = 0; +}; + +std::expected read_cpu_times() { + std::ifstream f("/proc/stat"); + if (!f) { + return std::unexpected("open /proc/stat failed"); + } + std::string line; + if (!std::getline(f, line)) { + return std::unexpected("empty /proc/stat"); + } + std::istringstream iss(line); + std::string label; + iss >> label; + if (label != "cpu") { + return std::unexpected("unexpected /proc/stat format: " + line); + } + CpuTimes t; + std::uint64_t v = 0; + int i = 0; + while (iss >> v) { + t.total += v; + if (i == 3) { // idle is the 4th value after "cpu" + t.idle = v; + } + ++i; + } + return t; +} + +double cpu_pct(const CpuTimes& prev, const CpuTimes& cur) { + const double d_total = static_cast(cur.total - prev.total); + const double d_idle = static_cast(cur.idle - prev.idle); + if (d_total <= 0) { + return 0; + } + return (1.0 - d_idle / d_total) * 100.0; +} + +std::expected read_mem_pct() { + std::ifstream f("/proc/meminfo"); + if (!f) { + return std::unexpected("open /proc/meminfo failed"); + } + double total = 0; + double avail = 0; + std::string line; + while (std::getline(f, line)) { + std::istringstream iss(line); + std::string key; + double v = 0; + iss >> key >> v; + if (key == "MemTotal:") { + total = v; + } else if (key == "MemAvailable:") { + avail = v; + } + } + if (total <= 0) { + return std::unexpected("MemTotal not found"); + } + return (1.0 - avail / total) * 100.0; +} + +std::expected read_load1() { + std::ifstream f("/proc/loadavg"); + if (!f) { + return std::unexpected("open /proc/loadavg failed"); + } + double v = 0; + f >> v; + if (!f) { + return std::unexpected("empty /proc/loadavg"); + } + return v; +} + +struct Sample { + double cpu_pct = 0; + double mem_pct = 0; + double load1 = 0; +}; + +std::expected take_sample(CpuTimes& prev) { + auto cur = read_cpu_times(); + if (!cur) { + return std::unexpected(cur.error()); + } + Sample s; + if (prev.total > 0) { + s.cpu_pct = cpu_pct(prev, *cur); + } + prev = *cur; + auto mp = read_mem_pct(); + if (!mp) { + return std::unexpected(mp.error()); + } + s.mem_pct = *mp; + auto l1 = read_load1(); + if (!l1) { + return std::unexpected(l1.error()); + } + s.load1 = *l1; + return s; +} + +// --------------------------------------------------------------------------- +// OTel observable gauges — one shared, atomically-updated snapshot the +// meter's periodic collection callbacks read from, the same shape as the Go +// reference's mutex-guarded `latest sample` plus RegisterCallback closure. +// --------------------------------------------------------------------------- + +struct GaugeState { + std::atomic cpu_pct{0}; + std::atomic mem_pct{0}; + std::atomic load1{0}; + std::string node; +}; + +namespace metrics_api = opentelemetry::metrics; + +void observe_cpu(metrics_api::ObserverResult result, void* state) { + auto* st = static_cast(state); + if (auto* p = opentelemetry::nostd::get_if< + opentelemetry::nostd::shared_ptr>>(&result)) { + (*p)->Observe(st->cpu_pct.load(std::memory_order_relaxed), {{"node", st->node}}); + } +} + +void observe_mem(metrics_api::ObserverResult result, void* state) { + auto* st = static_cast(state); + if (auto* p = opentelemetry::nostd::get_if< + opentelemetry::nostd::shared_ptr>>(&result)) { + (*p)->Observe(st->mem_pct.load(std::memory_order_relaxed), {{"node", st->node}}); + } +} + +void observe_load(metrics_api::ObserverResult result, void* state) { + auto* st = static_cast(state); + if (auto* p = opentelemetry::nostd::get_if< + opentelemetry::nostd::shared_ptr>>(&result)) { + (*p)->Observe(st->load1.load(std::memory_order_relaxed), {{"node", st->node}}); + } +} + +} // namespace + +int run(const std::string& node, int interval_ms, bool once, telemetry::Handle& tel) { + GaugeState gauge_state; + gauge_state.node = node; + + opentelemetry::nostd::shared_ptr cpu_gauge; + opentelemetry::nostd::shared_ptr mem_gauge; + opentelemetry::nostd::shared_ptr load_gauge; + if (tel.enabled) { + cpu_gauge = tel.meter->CreateDoubleObservableGauge( + "sysagent.cpu.pct", "CPU utilization percent, from /proc/stat"); + mem_gauge = tel.meter->CreateDoubleObservableGauge( + "sysagent.mem.pct", "Memory-used percent, from /proc/meminfo"); + load_gauge = tel.meter->CreateDoubleObservableGauge("sysagent.load1", "1-minute load average, from /proc/loadavg"); + cpu_gauge->AddCallback(observe_cpu, &gauge_state); + mem_gauge->AddCallback(observe_mem, &gauge_state); + load_gauge->AddCallback(observe_load, &gauge_state); + } + + auto& sig = util::install_signal_flag(); + + CpuTimes prev{}; + // Prime the CPU-time baseline so the first printed sample isn't a bogus 0 + // (matches Go: read once, sleep 200ms, THEN start the sample loop). + if (auto t0 = read_cpu_times()) { + prev = *t0; + } + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + + int rc = 0; + for (;;) { + auto s = take_sample(prev); + if (!s) { + std::println(stderr, "sysagent: sample: {}", s.error()); + rc = 1; + break; + } + if (tel.enabled) { + gauge_state.cpu_pct.store(s->cpu_pct, std::memory_order_relaxed); + gauge_state.mem_pct.store(s->mem_pct, std::memory_order_relaxed); + gauge_state.load1.store(s->load1, std::memory_order_relaxed); + } + const auto ts = static_cast(std::time(nullptr)); + std::println("sysagent: node={} cpu_pct={:.2f} mem_pct={:.2f} load1={:.2f} ts={}", node, + s->cpu_pct, s->mem_pct, s->load1, ts); + std::fflush(stdout); + + if (once) { + break; + } + if (sig.load(std::memory_order_relaxed)) { + break; + } + std::this_thread::sleep_for(std::chrono::milliseconds(interval_ms)); + if (sig.load(std::memory_order_relaxed)) { + break; + } + } + + // Stop the callbacks from touching gauge_state before it goes out of + // scope: a periodic collection could otherwise fire concurrently with + // (or after) this function returning. + if (tel.enabled) { + cpu_gauge->RemoveCallback(observe_cpu, &gauge_state); + mem_gauge->RemoveCallback(observe_mem, &gauge_state); + load_gauge->RemoveCallback(observe_load, &gauge_state); + std::println(stderr, "sysagent: otel export_errors={}", telemetry::export_errors.load()); + } + return rc; +} + +// saturate — chaos helper: busy-spin threads (cpu) or hold allocated, touched +// memory (mem) for --seconds, so a concurrently-running sysagent's +// cpu_pct/mem_pct visibly rises (this fleet's USE-method callback). +int saturate(const std::string& resource, int seconds, int workers, int mb) { + if (resource == "cpu") { + if (workers <= 0) { + workers = static_cast(std::thread::hardware_concurrency()); + if (workers <= 0) { + workers = 1; + } + } + std::println("sysagent: saturate resource=cpu seconds={} workers={} started", seconds, workers); + std::fflush(stdout); + const auto stop = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + std::vector pool; + pool.reserve(static_cast(workers)); + for (int i = 0; i < workers; ++i) { + pool.emplace_back([stop] { + volatile double x = 0.0001; + while (std::chrono::steady_clock::now() < stop) { + x = x * 1.0000001 + 1.0; + } + }); + } + pool.clear(); // jthread destructors join + std::println("sysagent: saturate done"); + return 0; + } + if (resource == "mem") { + if (mb <= 0) { + mb = 256; + } + std::println("sysagent: saturate resource=mem seconds={} mb={} started", seconds, mb); + std::fflush(stdout); + const std::size_t n = static_cast(mb) * 1024 * 1024; + std::vector buf(n); + for (std::size_t i = 0; i < n; ++i) { + buf[i] = static_cast(i); // touch every page so it's resident + } + std::this_thread::sleep_for(std::chrono::seconds(seconds)); + std::println("sysagent: saturate done"); + (void)buf[0]; + return 0; + } + std::println(stderr, "usage: sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M]"); + return 2; +} + +} // namespace sysagent diff --git a/examples/41-capstone-fleet/cpp/src/sysagent.hpp b/examples/41-capstone-fleet/cpp/src/sysagent.hpp new file mode 100644 index 0000000..4dac669 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/sysagent.hpp @@ -0,0 +1,23 @@ +// sysagent.hpp — the book's /proc-reading metrics agent (ch36), reduced to +// three signals (cpu%, mem%, load1) exported both as a stdout line and, when +// OTEL_EXPORTER_OTLP_ENDPOINT is set, as OTel gauges over OTLP. See +// sysagent.cpp for the /proc/stat delta math, ported field-for-field from the +// Go reference (deliberately simpler than ch36's fuller procfs.cpp: this +// capstone only ever needed the three USE-method numbers pmon's health line +// and the chaos drill care about). +#pragma once + +#include + +#include "telemetry.hpp" + +namespace sysagent { + +int run(const std::string& node, int interval_ms, bool once, telemetry::Handle& tel); + +// saturate — chaos helper: busy-spin threads (cpu) or hold touched memory +// (mem) for `seconds`, so a concurrently-running sysagent's cpu_pct/mem_pct +// visibly rises. +int saturate(const std::string& resource, int seconds, int workers, int mb); + +} // namespace sysagent diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.cpp b/examples/41-capstone-fleet/cpp/src/telemetry.cpp index f8bd439..9ef07ee 100644 --- a/examples/41-capstone-fleet/cpp/src/telemetry.cpp +++ b/examples/41-capstone-fleet/cpp/src/telemetry.cpp @@ -1,7 +1,9 @@ #include "telemetry.hpp" +#include #include #include +#include #include #include @@ -10,11 +12,13 @@ #include #include #include -#include +#include #include +#include #include #include #include +#include #include #include @@ -30,31 +34,46 @@ namespace metrics_api = opentelemetry::metrics; namespace metrics_sdk = opentelemetry::sdk::metrics; namespace resource_sdk = opentelemetry::sdk::resource; namespace otlp = opentelemetry::exporter::otlp; - -// Counts every Error-level internal log line (the SDK's own signal that an -// export attempt did not get a 2xx / hit a transport failure) instead of -// crashing a supervised service over a telemetry hiccup. -class CountingLogHandler : public opentelemetry::sdk::common::internal_log::LogHandler { +namespace internal_log = opentelemetry::sdk::common::internal_log; + +// Counts every Error-level internal SDK log line and prints it, the closest +// opentelemetry-cpp equivalent of Go's otel.SetErrorHandler callback (which +// both increments exportErrors and Fprintf's to stderr on every export +// failure). LogLevel is pinned to Error below, so Info/Debug/Warning lines — +// the ones that would otherwise corrupt the stdout contract, ch38's hard-won +// lesson — never reach this handler at all. +class CountingLogHandler : public internal_log::LogHandler { public: - void Handle(opentelemetry::sdk::common::internal_log::LogLevel level, const char* file, int line, - const char* msg, const opentelemetry::sdk::common::AttributeMap& attrs) noexcept override { - using opentelemetry::sdk::common::internal_log::LogLevel; - if (level == LogLevel::Error) { - export_errors.fetch_add(1); - std::fprintf(stderr, "otel: export error: %s\n", msg); + explicit CountingLogHandler(std::string service) : service_(std::move(service)) {} + + void Handle(internal_log::LogLevel level, const char* /*file*/, int /*line*/, const char* msg, + const opentelemetry::sdk::common::AttributeMap& /*attrs*/) noexcept override { + if (level == internal_log::LogLevel::Error) { + export_errors.fetch_add(1, std::memory_order_relaxed); + std::fprintf(stderr, "%s: otel export error: %s\n", service_.c_str(), msg); } - (void)file; - (void)line; - (void)attrs; } -}; -std::shared_ptr g_tracer_provider; -std::shared_ptr g_meter_provider; +private: + std::string service_; +}; } // namespace -Handle::~Handle() = default; +void Handle::shutdown() { + if (!enabled) { + return; + } + constexpr auto kTimeout = std::chrono::seconds(5); + if (tracer_provider) { + tracer_provider->ForceFlush(kTimeout); + tracer_provider->Shutdown(kTimeout); + } + if (meter_provider) { + meter_provider->ForceFlush(kTimeout); + meter_provider->Shutdown(kTimeout); + } +} Handle init(const std::string& service, const std::string& node) { const char* endpoint = std::getenv("OTEL_EXPORTER_OTLP_ENDPOINT"); @@ -67,10 +86,9 @@ Handle init(const std::string& service, const std::string& node) { return h; } - using namespace opentelemetry::sdk::common::internal_log; - GlobalLogHandler::SetLogHandler( - opentelemetry::nostd::shared_ptr(new CountingLogHandler())); - GlobalLogHandler::SetLogLevel(LogLevel::Error); + internal_log::GlobalLogHandler::SetLogHandler( + opentelemetry::nostd::shared_ptr(new CountingLogHandler(service))); + internal_log::GlobalLogHandler::SetLogLevel(internal_log::LogLevel::Error); auto resource = resource_sdk::Resource::Create({ {"service.name", service}, @@ -79,32 +97,46 @@ Handle init(const std::string& service, const std::string& node) { {"node.name", node}, }); + // --- traces: OTLP/HTTP exporter, batched every 500ms (matches the Go + // reference's sdktrace.WithBatchTimeout). --- otlp::OtlpHttpExporterOptions trace_opts; trace_opts.url = std::string(endpoint) + "/v1/traces"; auto trace_exporter = otlp::OtlpHttpExporterFactory::Create(trace_opts); + trace_sdk::BatchSpanProcessorOptions batch_opts; batch_opts.schedule_delay_millis = std::chrono::milliseconds(500); auto processor = trace_sdk::BatchSpanProcessorFactory::Create(std::move(trace_exporter), batch_opts); - g_tracer_provider = trace_sdk::TracerProviderFactory::Create(std::move(processor), resource); - trace_api::Provider::SetTracerProvider(g_tracer_provider); + auto sampler = trace_sdk::AlwaysOnSamplerFactory::Create(); + std::shared_ptr tracer_provider( + trace_sdk::TracerProviderFactory::Create(std::move(processor), resource, std::move(sampler))); + trace_api::Provider::SetTracerProvider( + opentelemetry::nostd::shared_ptr(tracer_provider)); + + // --- metrics: OTLP/HTTP exporter, exported every 2s (matches the Go + // reference's sdkmetric.WithInterval). --- otlp::OtlpHttpMetricExporterOptions metric_opts; metric_opts.url = std::string(endpoint) + "/v1/metrics"; auto metric_exporter = otlp::OtlpHttpMetricExporterFactory::Create(metric_opts); + metrics_sdk::PeriodicExportingMetricReaderOptions reader_opts; reader_opts.export_interval_millis = std::chrono::milliseconds(2000); reader_opts.export_timeout_millis = std::chrono::milliseconds(1000); - auto reader = std::make_shared(std::move(metric_exporter), reader_opts); - auto mp = metrics_sdk::MeterProviderFactory::Create( - std::make_unique(), resource); - static_cast(mp.get())->AddMetricReader(reader); - g_meter_provider = std::shared_ptr(std::move(mp)); - metrics_api::Provider::SetMeterProvider(g_meter_provider); + auto reader = + std::make_shared(std::move(metric_exporter), reader_opts); + + std::shared_ptr meter_provider( + metrics_sdk::MeterProviderFactory::Create(std::make_unique(), resource)); + meter_provider->AddMetricReader(reader); + metrics_api::Provider::SetMeterProvider( + opentelemetry::nostd::shared_ptr(meter_provider)); std::fprintf(stderr, "%s: otel enabled endpoint=%s node=%s\n", service.c_str(), endpoint, node.c_str()); Handle h; h.enabled = true; + h.tracer_provider = tracer_provider; + h.meter_provider = meter_provider; h.tracer = trace_api::Provider::GetTracerProvider()->GetTracer(service); h.meter = metrics_api::Provider::GetMeterProvider()->GetMeter(service); return h; diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.hpp b/examples/41-capstone-fleet/cpp/src/telemetry.hpp index fda667c..941a0d0 100644 --- a/examples/41-capstone-fleet/cpp/src/telemetry.hpp +++ b/examples/41-capstone-fleet/cpp/src/telemetry.hpp @@ -1,8 +1,13 @@ // telemetry.hpp — OTel wiring shared by chatterd and sysagent, following the -// book's house C++ pattern (opentelemetry-cpp via Conan, OTLP/HTTP). Endpoint -// comes from the standard OTEL_EXPORTER_OTLP_ENDPOINT env var, which the -// exporters read themselves — the same variable -// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT. +// book's house C++ pattern established in ch38 (opentelemetry-cpp via Conan, +// OTLP/HTTP, a NoopLogHandler-derived quiet SDK). Endpoint comes from the +// standard OTEL_EXPORTER_OTLP_ENDPOINT env var, the same variable +// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT — the +// exporters are pointed at "/v1/traces" and "/v1/metrics" +// explicitly here (opentelemetry-cpp's own OTEL_EXPORTER_OTLP_ENDPOINT +// auto-read does exactly this same per-signal suffixing per the OTLP env-var +// spec; setting it explicitly keeps init() self-contained and independent of +// the SDK's own env parsing). #pragma once #include @@ -10,31 +15,52 @@ #include #include +#include +#include +#include #include namespace telemetry { -// Count of OTel export failures observed via the global error/exception -// handler, the deterministic stand-in for "the exporter got a 200" (the -// batch/periodic exporters only invoke it on a non-2xx response or a -// transport failure). +// Count of OTel export failures observed via the SDK's internal Error-level +// log line — opentelemetry-cpp has no direct equivalent of Go's +// otel.SetErrorHandler, so this is the deterministic stand-in the internal +// log handler feeds: printed as ": otel export error: ..." exactly +// once per Error-level line, mirroring what Go's handler does on every +// export failure. extern std::atomic export_errors; +// Handle owns the trace/metric providers (kept alive so shutdown() can flush +// them) plus the tracer/meter callers actually record against. Move-only; +// when telemetry is disabled every field but `enabled` is left default +// (tracer/meter still resolve to the process-wide no-op API implementations, +// so callers never need to branch on `enabled` before calling into them). struct Handle { bool enabled = false; - std::shared_ptr tracer; - std::shared_ptr meter; + std::shared_ptr tracer_provider; + std::shared_ptr meter_provider; + opentelemetry::nostd::shared_ptr tracer; + opentelemetry::nostd::shared_ptr meter; - ~Handle(); Handle() = default; Handle(const Handle&) = delete; Handle& operator=(const Handle&) = delete; Handle(Handle&&) = default; Handle& operator=(Handle&&) = default; + + // Flushes and shuts down both providers, bounded by a short timeout. A + // no-op when telemetry is disabled. The Go reference never gets a chance + // to run its own deferred shutdown (every subcommand's dispatch in + // main.go wraps its run in os.Exit(), which skips deferred calls) — this + // is deliberately stricter than that, not a divergence in anything + // observable: it prints nothing either way, and only improves the odds + // of the last batch actually reaching the collector. + void shutdown(); }; // init sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT is set; -// otherwise returns a disabled, no-op handle (the service still runs). +// otherwise returns a disabled, no-op handle (the service still runs — OTel +// is additive, never a hard dependency of the fleet coming up). Handle init(const std::string& service, const std::string& node); } // namespace telemetry diff --git a/examples/41-capstone-fleet/cpp/src/util.cpp b/examples/41-capstone-fleet/cpp/src/util.cpp new file mode 100644 index 0000000..a5cfc75 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/util.cpp @@ -0,0 +1,27 @@ +#include "util.hpp" + +#include +#include + +namespace util { + +namespace { +std::atomic g_signal_flag{false}; +std::once_flag g_installed; + +extern "C" void on_signal(int) { g_signal_flag.store(true, std::memory_order_relaxed); } +} // namespace + +std::atomic& install_signal_flag() { + std::call_once(g_installed, [] { + struct sigaction sa {}; + sa.sa_handler = on_signal; + sigemptyset(&sa.sa_mask); + sa.sa_flags = 0; // level-triggered polling, not EINTR-driven: SA_RESTART doesn't matter here + ::sigaction(SIGINT, &sa, nullptr); + ::sigaction(SIGTERM, &sa, nullptr); + }); + return g_signal_flag; +} + +} // namespace util diff --git a/examples/41-capstone-fleet/cpp/src/util.hpp b/examples/41-capstone-fleet/cpp/src/util.hpp new file mode 100644 index 0000000..a77f995 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/util.hpp @@ -0,0 +1,19 @@ +// util.hpp — small helpers shared across the fleet's subcommands, the C++ +// counterpart of util.go's installSignalFlag: chatterd/sysagent/fwatch each +// run their own poll loop already (accept-with-timeout, interval sleep, +// poll(2) with a timeout), so a level-triggered flag is all any of them +// needs from a signal — no EINTR-sensitive syscall unwinding required, unlike +// ex40's fastpath server. pmon (a supervisor, not a poller) does its own +// dedicated sigwait() handling in pmon.cpp instead of using this. +#pragma once + +#include + +namespace util { + +// Installs a SIGINT/SIGTERM handler (once per process; safe to call from +// multiple subcommands' code paths) that flips a process-wide flag to true +// the first time either signal arrives. Returns a reference to that flag. +std::atomic& install_signal_flag(); + +} // namespace util From 39799ffd3f75cc01e671b6711f9f71dca543b934 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 24 Jul 2026 10:48:44 -0400 Subject: [PATCH 5/7] demo(demo-41): Rust port of the capstone fleet Ports the ~1300-line Go fleet to Rust (edition 2024), same file split as the C++ port, reusing ex33 (landlock crate + raw-syscall ABI probe), ex09 (nix inotify), ex36 (/proc math), ex38 (opentelemetry-otlp 0.32, base OTEL_EXPORTER_OTLP_ENDPOINT so the SDK appends /v1/*, no tracing-subscriber to keep the output contract), and ex40 (raw-fd TCP, OwnedFd, signals). Verified on systems-target via the runner: PASS 31 / FAIL 0. pmon forks/execs via nix; chatterd is a raw-fd TCP hub with the cross-node bridge (received from=NICK@node); sysagent pushes three OTel gauges with a node attribute; fwatch drives nix inotify + the landlock crate. Deps all cached offline (nix 0.31, landlock 0.4, opentelemetry-otlp 0.32, libc, anyhow). Documented divergence: opentelemetry-rust 0.32 surfaces export errors only via tracing events, so the export_errors counter stays 0 (verify asserts the other telemetry lines, not that value). --- _plans/r10-capstone-fleet.md | 2 +- examples/41-capstone-fleet/rust/Cargo.lock | 1134 ++++++++++++++++- examples/41-capstone-fleet/rust/Cargo.toml | 15 +- examples/41-capstone-fleet/rust/src/caps.rs | 36 + .../41-capstone-fleet/rust/src/chatterd.rs | 413 ++++++ examples/41-capstone-fleet/rust/src/fwatch.rs | 206 +++ examples/41-capstone-fleet/rust/src/main.rs | 245 +++- examples/41-capstone-fleet/rust/src/pmon.rs | 302 +++++ examples/41-capstone-fleet/rust/src/proto.rs | 107 ++ .../41-capstone-fleet/rust/src/sysagent.rs | 209 +++ .../41-capstone-fleet/rust/src/telemetry.rs | 297 +++++ examples/41-capstone-fleet/rust/src/util.rs | 38 + 12 files changed, 2977 insertions(+), 27 deletions(-) create mode 100644 examples/41-capstone-fleet/rust/src/caps.rs create mode 100644 examples/41-capstone-fleet/rust/src/chatterd.rs create mode 100644 examples/41-capstone-fleet/rust/src/fwatch.rs create mode 100644 examples/41-capstone-fleet/rust/src/pmon.rs create mode 100644 examples/41-capstone-fleet/rust/src/proto.rs create mode 100644 examples/41-capstone-fleet/rust/src/sysagent.rs create mode 100644 examples/41-capstone-fleet/rust/src/telemetry.rs create mode 100644 examples/41-capstone-fleet/rust/src/util.rs diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index 146d16f..7bc2732 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -165,6 +165,6 @@ the dispatcher, rather than one 1300-line drop. | S1 contract extraction | done — captured empirically from the Go reference (usage, sysagent/fwatch/chatterd/pmon output shapes, the `from=NICK@node` bridge tag, `landlock ABI=N enforced`, `bounding_set_dropped=N no_new_privs=1`, `sysagent_cpu_pct` gauge) | | S2 verify.lua (go gate) | done — `PASS 31 / FAIL 0` for `LSP_LANG=go` through the runner. Following ex23's convention the verify runs entirely on LOCAL loopback (fleet, cross-node bridge on two 127.0.0.1 ports, telemetry to the local LGTM) rather than orchestrating two guests over ssh — far more tractable, and the bridge logic is identical on loopback; the real two-host run is the chapter's demo | | S3 C++ port | done — `PASS 31 / FAIL 0` for `LSP_LANG=cpp` through the runner, first gate. Multi-file (pmon/chatterd/sysagent/fwatch/proto/caps/telemetry/util), reusing ex33/34/36/38/40; fixed a real telemetry-scaffold bug (`shared_ptr` vs `nostd::shared_ptr`) and the OTLP CMake linkage. Builds offline | -| S4 Rust port | pending (deps de-risked — all cached) | +| S4 Rust port | done — `PASS 31 / FAIL 0` for `LSP_LANG=rust` through the runner, first gate. Same file split as C++, reusing ex33/09/36/38/40. nix (inotify) + landlock 0.4 + opentelemetry-otlp 0.32, all offline. Documented divergence: opentelemetry-rust 0.32 surfaces export errors only via tracing events (needs a subscriber that would break the output contract), so the `export_errors` counter stays 0 — verify does not assert its value | | S5 whole-fleet gate | pending | | S6 chapter 41 | pending | diff --git a/examples/41-capstone-fleet/rust/Cargo.lock b/examples/41-capstone-fleet/rust/Cargo.lock index 9fc0988..07a838f 100644 --- a/examples/41-capstone-fleet/rust/Cargo.lock +++ b/examples/41-capstone-fleet/rust/Cargo.lock @@ -2,6 +2,41 @@ # It is not intended for manual editing. version = 4 +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "async-trait" +version = "0.1.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae36dc4177970ef04fde5178d3e2429882def40e57a451f919c098f72baa6cec" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + [[package]] name = "bitflags" version = "2.13.1" @@ -9,13 +44,398 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" [[package]] -name = "errno" -version = "0.3.14" +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "capstone-fleet" +version = "0.1.0" dependencies = [ + "anyhow", + "landlock", "libc", - "windows-sys", + "nix", + "opentelemetry", + "opentelemetry-otlp", + "opentelemetry_sdk", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "enumflags2" +version = "0.7.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1027f7680c853e056ebcec683615fb6fbbc07dbaa13b4d5d9442b146ded4ecef" +dependencies = [ + "enumflags2_derive", +] + +[[package]] +name = "enumflags2_derive" +version = "0.7.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67c78a4d8fdf9953a5c9d458f9efe940fd97a0cab0941c075a813ac594733827" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi", + "wasip2", +] + +[[package]] +name = "http" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "hyper" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "http", + "http-body", + "httparse", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "landlock" +version = "0.4.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "635839550ae8b90d9fd2571460a6645dc0aec070225956ca7a2831ed31d2795d" +dependencies = [ + "enumflags2", + "libc", + "thiserror", ] [[package]] @@ -25,29 +445,604 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" [[package]] -name = "linux-raw-sys" -version = "0.12.1" +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "memoffset" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "mio" +version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "wasi", + "windows-sys", +] [[package]] -name = "rustix" -version = "1.1.4" +name = "nix" +version = "0.31.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" dependencies = [ "bitflags", - "errno", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opentelemetry" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0142c63252a9e054e68a4c61a5778f7b14f576274d593f8ce883d191a099682" +dependencies = [ + "futures-core", + "futures-sink", + "js-sys", + "pin-project-lite", + "thiserror", + "tracing", +] + +[[package]] +name = "opentelemetry-http" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5683015d09e2df236ef005b17f6f196f0d5f6313c4fa43a7b6a53b52776e4331" +dependencies = [ + "async-trait", + "bytes", + "http", + "opentelemetry", + "reqwest", +] + +[[package]] +name = "opentelemetry-otlp" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9966929966d17620d7c316c643ba62631826e10021409357772d5eea84f62c35" +dependencies = [ + "http", + "opentelemetry", + "opentelemetry-http", + "opentelemetry-proto", + "opentelemetry_sdk", + "prost", + "reqwest", + "thiserror", +] + +[[package]] +name = "opentelemetry-proto" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56d658ba1faf63f7b9c492cfbe6e0ec365440a16132d3270c1065f7b33f1b638" +dependencies = [ + "opentelemetry", + "opentelemetry_sdk", + "prost", +] + +[[package]] +name = "opentelemetry_sdk" +version = "0.32.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b59f80e1ac4d5ff7a2db8fb6c80badb7f0f3f858211fba08dd9aaec750894f9" +dependencies = [ + "futures-channel", + "futures-executor", + "futures-util", + "opentelemetry", + "percent-encoding", + "portable-atomic", + "rand", + "thiserror", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prost" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-derive" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom", +] + +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ "libc", - "linux-raw-sys", "windows-sys", ] [[package]] -name = "template-hello" -version = "0.1.0" +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2fac314a64dc9a36e61a9eb4261a5e9bbfbc922b27e518af97bc32b926cf967" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" dependencies = [ - "rustix", + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d988bcd52dbe076d3d46903332f58c912b87a2c49b1428419a5845154762ffee" +dependencies = [ + "libc", + "mio", + "pin-project-lite", + "socket2", + "windows-sys", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-core", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.76" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141" +dependencies = [ + "js-sys", + "wasm-bindgen", ] [[package]] @@ -64,3 +1059,112 @@ checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" dependencies = [ "windows-link", ] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] diff --git a/examples/41-capstone-fleet/rust/Cargo.toml b/examples/41-capstone-fleet/rust/Cargo.toml index 62c962f..b20e200 100644 --- a/examples/41-capstone-fleet/rust/Cargo.toml +++ b/examples/41-capstone-fleet/rust/Cargo.toml @@ -1,5 +1,5 @@ [package] -name = "template-hello" +name = "capstone-fleet" version = "0.1.0" edition = "2024" @@ -8,4 +8,15 @@ name = "app" path = "src/main.rs" [dependencies] -rustix = { version = "1", features = ["system"] } +anyhow = "1" +libc = "0.2" +nix = { version = "0.31", features = ["fs", "inotify", "net", "poll", "process", "signal"] } +landlock = "0.4" +opentelemetry = "0.32" +opentelemetry_sdk = "0.32" +opentelemetry-otlp = { version = "0.32", default-features = false, features = [ + "http-proto", + "reqwest-blocking-client", + "trace", + "metrics", +] } diff --git a/examples/41-capstone-fleet/rust/src/caps.rs b/examples/41-capstone-fleet/rust/src/caps.rs new file mode 100644 index 0000000..448981f --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/caps.rs @@ -0,0 +1,36 @@ +// caps.rs — capability-bounding-set drop, the cheap, always-available half of +// this chapter's sandboxing story (the other half, Landlock, lives in +// fwatch.rs). See go/caps.go for the full rationale. +// +// PR_CAPBSET_DROP removes a capability from the *bounding set*: even a future +// execve of a setuid-root binary can never regain it, for this process and +// every descendant (the bounding set is inherited across fork/exec and can +// only shrink). It is meaningful regardless of whether the calling process +// currently holds any capabilities at all — that is exactly pmon's case when +// deployed unprivileged on a lab guest. + +// lastKnownCap is CAP_CHECKPOINT_RESTORE (40) on a 6.x+ kernel; probing past +// the kernel's actual last cap just returns EINVAL, which we ignore. +const LAST_KNOWN_CAP: libc::c_ulong = 40; + +pub fn drop_bounding_set() { + let mut dropped = 0; + for c in 0..=LAST_KNOWN_CAP { + // SAFETY: prctl(PR_CAPBSET_DROP, c, 0, 0, 0) has no memory-safety + // implications; only its return value is inspected. + let rc = unsafe { libc::prctl(libc::PR_CAPBSET_DROP, c, 0, 0, 0) }; + if rc == 0 { + dropped += 1; + } + } + // SAFETY: same as above. + let rc = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; + if rc != 0 { + eprintln!( + "pmon: prctl(PR_SET_NO_NEW_PRIVS): {}", + std::io::Error::last_os_error() + ); + return; + } + eprintln!("pmon: capabilities bounding_set_dropped={dropped} no_new_privs=1"); +} diff --git a/examples/41-capstone-fleet/rust/src/chatterd.rs b/examples/41-capstone-fleet/rust/src/chatterd.rs new file mode 100644 index 0000000..d667c7c --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/chatterd.rs @@ -0,0 +1,413 @@ +// chatterd.rs — the book's recurring peer-to-peer chat daemon (ch21-27), +// reduced to what the capstone needs: serve local clients, and (new in this +// chapter) bridge two chatterd instances across hosts so a message sent on +// one node is delivered to clients on the other. The wire frame (proto.rs) +// is unchanged from every prior chatterd chapter. +// +// Bridging design (identical to go/chatterd.go's and chatterd.cpp's): a +// bridge worker is just an ordinary client of the *other* node's server +// (nick "bridge@"). Concretely, with --peer set on both sides: +// +// target dials peer, joins as "bridge@target" (this connection lives in +// peer's client registry) +// peer dials target, joins as "bridge@peer" (lives in target's registry) +// +// A local MSG is broadcast as DELIVER to every registered connection +// (including a bridge connection dialed in from the other side) exactly like +// any other client — no protocol change. When a bridge *worker* itself +// receives a DELIVER over the connection it dialed, it re-broadcasts locally +// with include_bridges=false, so the message never goes back out over any +// bridge: no ping-pong between two nodes. +// +// Raw fds via `nix` rather than std::net, following ex40's fastpath and +// chatterd.cpp's own RAII-Fd approach: a poll(2)-with-timeout accept loop +// (so the process can check the shutdown flag between accepts) instead of +// closing a blocked accept() from another thread, which std::net cannot do +// safely. OwnedFd is the RAII socket — closes on drop, move-only, no +// hand-written destructor needed. +use std::net::{Ipv4Addr, SocketAddrV4}; +use std::os::fd::{AsFd, AsRawFd, FromRawFd, OwnedFd, RawFd}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::Duration; + +use nix::errno::Errno; +use nix::fcntl::{FcntlArg, OFlag, fcntl}; +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sys::socket::{ + AddressFamily, Backlog, SockFlag, SockType, SockaddrIn, accept4, bind, connect, getsockopt, + listen as listen_syscall, setsockopt, socket, sockopt, +}; +use nix::sys::time::{TimeVal, TimeValLike}; + +use crate::proto::{self, Frame}; +use crate::telemetry::{self, Telemetry}; +use crate::util; + +// --------------------------------------------------------------------------- +// TCP helpers — IPv4-literal only, matching chatterd.cpp's inet_pton-based +// listen_tcp/connect_tcp (every caller in this example, verify.lua included, +// only ever passes literal addresses, never a hostname to resolve). +// --------------------------------------------------------------------------- + +fn listen_tcp(host: &str, port: u16) -> Result { + let ip: Ipv4Addr = host + .parse() + .map_err(|_| format!("{host}: not an IPv4 address"))?; + let sock = socket( + AddressFamily::Inet, + SockType::Stream, + SockFlag::SOCK_CLOEXEC, + None, + ) + .map_err(|e| format!("socket: {e}"))?; + let _ = setsockopt(&sock, sockopt::ReuseAddr, &true); + let addr = SockaddrIn::from(SocketAddrV4::new(ip, port)); + bind(sock.as_raw_fd(), &addr).map_err(|e| format!("bind: {e}"))?; + listen_syscall( + &sock, + Backlog::new(16).map_err(|e| format!("backlog: {e}"))?, + ) + .map_err(|e| format!("listen: {e}"))?; + Ok(sock) +} + +// Non-blocking connect + poll(2), the syscall-level equivalent of Go's +// net.DialTimeout. +fn connect_tcp(host: &str, port: u16, timeout_ms: i64) -> Result { + let ip: Ipv4Addr = host + .parse() + .map_err(|_| format!("{host}: not an IPv4 address"))?; + let sock = socket( + AddressFamily::Inet, + SockType::Stream, + SockFlag::SOCK_CLOEXEC, + None, + ) + .map_err(|e| format!("socket: {e}"))?; + let addr = SockaddrIn::from(SocketAddrV4::new(ip, port)); + + let orig_flags = + fcntl(sock.as_fd(), FcntlArg::F_GETFL).map_err(|e| format!("fcntl(F_GETFL): {e}"))?; + let mut nb_flags = OFlag::from_bits_truncate(orig_flags); + nb_flags.insert(OFlag::O_NONBLOCK); + fcntl(sock.as_fd(), FcntlArg::F_SETFL(nb_flags)).map_err(|e| format!("fcntl(F_SETFL): {e}"))?; + + match connect(sock.as_raw_fd(), &addr) { + Ok(()) => {} + Err(Errno::EINPROGRESS) => { + let mut fds = [PollFd::new(sock.as_fd(), PollFlags::POLLOUT)]; + let timeout = + PollTimeout::try_from(timeout_ms.max(0) as u64).unwrap_or(PollTimeout::MAX); + let n = poll(&mut fds, timeout).map_err(|e| format!("poll: {e}"))?; + if n <= 0 { + return Err("i/o timeout".to_string()); + } + let soerr: i32 = + getsockopt(&sock, sockopt::SocketError).map_err(|e| format!("getsockopt: {e}"))?; + if soerr != 0 { + return Err(std::io::Error::from_raw_os_error(soerr).to_string()); + } + } + Err(e) => return Err(format!("connect {host}:{port}: {e}")), + } + // Restore blocking mode. + fcntl( + sock.as_fd(), + FcntlArg::F_SETFL(OFlag::from_bits_truncate(orig_flags)), + ) + .map_err(|e| format!("fcntl(F_SETFL restore): {e}"))?; + Ok(sock) +} + +// --------------------------------------------------------------------------- +// Hub — the registry of every connection (local client or bridge worker) +// currently attached to this server. +// --------------------------------------------------------------------------- + +struct PeerConn { + fd: OwnedFd, + nick: String, + is_bridge: bool, + wmu: Mutex<()>, +} + +struct Hub { + clients: Mutex>>, +} + +impl Hub { + fn new() -> Self { + Hub { + clients: Mutex::new(Vec::new()), + } + } + + fn add(&self, c: Arc) { + self.clients.lock().unwrap().push(c); + } + + fn remove(&self, c: &Arc) { + let mut g = self.clients.lock().unwrap(); + g.retain(|x| !Arc::ptr_eq(x, c)); + } + + fn broadcast_deliver( + &self, + exclude: Option<&Arc>, + include_bridges: bool, + nick: &str, + text: &str, + ) { + let targets: Vec> = { + let g = self.clients.lock().unwrap(); + g.iter() + .filter(|c| exclude.map(|e| !Arc::ptr_eq(c, e)).unwrap_or(true)) + .filter(|c| include_bridges || !c.is_bridge) + .cloned() + .collect() + }; + let payload = proto::deliver_payload(nick, text); + for c in targets { + let _wg = c.wmu.lock().unwrap(); + let f = Frame { + typ: proto::TYPE_DELIVER, + payload: payload.clone(), + }; + let _ = proto::write_frame(c.fd.as_raw_fd(), &f); // best-effort, matches Go's `_ = c.send(f)` + } + } +} + +fn handle_client(fd: OwnedFd, hub: Arc, node: String, tel: Arc) { + let raw: RawFd = fd.as_raw_fd(); + let first = match proto::read_frame(raw) { + Ok(f) if f.typ == proto::TYPE_JOIN => f, + _ => return, // fd (owned locally) closes on drop + }; + let nick = String::from_utf8_lossy(&first.payload).into_owned(); + let is_bridge = nick.starts_with("bridge@"); + let pc = Arc::new(PeerConn { + fd, + nick, + is_bridge, + wmu: Mutex::new(()), + }); + hub.add(pc.clone()); + + while let Ok(f) = proto::read_frame(raw) { + if f.typ != proto::TYPE_MSG { + continue; + } + let text = String::from_utf8_lossy(&f.payload).into_owned(); + + match telemetry::start_deliver_span(&tel, &pc.nick, text.len(), &node, pc.is_bridge) { + Some((span, trace_id)) => { + hub.broadcast_deliver(Some(&pc), true, &pc.nick, &text); + span.end_ok(); + if tel.enabled { + println!("chatterd: trace_id={trace_id} node={node} from={}", pc.nick); + } + } + None => hub.broadcast_deliver(Some(&pc), true, &pc.nick, &text), + } + } + hub.remove(&pc); +} + +fn bridge_worker( + peer_addr: String, + local_node: String, + peer_node: String, + hub: Arc, + sig: &'static AtomicBool, +) { + let Some((host, port_str)) = peer_addr.rsplit_once(':') else { + return; + }; + let Ok(port) = port_str.parse::() else { + return; + }; + let backoff = Duration::from_millis(300); + + while !sig.load(Ordering::Relaxed) { + let conn = match connect_tcp(host, port, 3000) { + Ok(c) => c, + Err(_) => { + thread::sleep(backoff); + continue; + } + }; + let raw = conn.as_raw_fd(); + let bridge_nick = format!("bridge@{local_node}"); + let joinf = Frame { + typ: proto::TYPE_JOIN, + payload: bridge_nick.clone().into_bytes(), + }; + if proto::write_frame(raw, &joinf).is_err() { + thread::sleep(backoff); + continue; + } + eprintln!("chatterd: bridge connected peer={peer_addr} as={bridge_nick}"); + + while let Ok(f) = proto::read_frame(raw) { + if f.typ != proto::TYPE_DELIVER { + continue; + } + let Some((mut nick, text)) = proto::split_deliver(&f.payload) else { + continue; + }; + if !nick.contains('@') { + nick.push('@'); + nick.push_str(&peer_node); + } + hub.broadcast_deliver(None, false, &nick, &text); + } + eprintln!("chatterd: bridge disconnected peer={peer_addr} (retrying)"); + thread::sleep(backoff); + // `conn` drops here, closing the fd, before the loop reconnects. + } +} + +pub fn serve( + host: &str, + port: u16, + node: &str, + peer: &str, + peer_node_in: &str, + tel: Arc, +) -> i32 { + let lfd = match listen_tcp(host, port) { + Ok(fd) => fd, + Err(e) => { + eprintln!("chatterd: listen {host}:{port}: {e}"); + return 1; + } + }; + eprintln!("chatterd: listening on {host}:{port} node={node}"); + + let hub = Arc::new(Hub::new()); + let sig = util::install_signal_flag(); + + if !peer.is_empty() { + let peer_node = if peer_node_in.is_empty() { + "remote".to_string() + } else { + peer_node_in.to_string() + }; + let hub2 = hub.clone(); + let peer = peer.to_string(); + let node2 = node.to_string(); + thread::spawn(move || bridge_worker(peer, node2, peer_node, hub2, sig)); + } + + loop { + if sig.load(Ordering::Relaxed) { + eprintln!("chatterd: shutdown"); + return 0; + } + let mut fds = [PollFd::new(lfd.as_fd(), PollFlags::POLLIN)]; + match poll(&mut fds, PollTimeout::from(200u16)) { + Ok(0) => continue, + Ok(_) => {} + Err(Errno::EINTR) => continue, + Err(e) => { + eprintln!("chatterd: poll: {e}"); + return 1; + } + } + let ready = fds[0] + .revents() + .map(|r| r.contains(PollFlags::POLLIN)) + .unwrap_or(false); + if !ready { + continue; + } + let cfd = match accept4(lfd.as_raw_fd(), SockFlag::SOCK_CLOEXEC) { + Ok(fd) => unsafe { OwnedFd::from_raw_fd(fd) }, + Err(_) => continue, + }; + let hub2 = hub.clone(); + let node2 = node.to_string(); + let tel2 = tel.clone(); + thread::spawn(move || handle_client(cfd, hub2, node2, tel2)); + } +} + +// --------------------------------------------------------------------------- +// send / listen — minimal test clients used by verify to prove cross-host +// delivery deterministically (this capstone trims ch27's fuller "loadtest"). +// --------------------------------------------------------------------------- + +pub fn send(host: &str, port: u16, nick: &str, text: &str, timeout_ms: i64) -> i32 { + let conn = match connect_tcp(host, port, timeout_ms) { + Ok(c) => c, + Err(e) => { + eprintln!("chatterd: send: dial {host}:{port}: {e}"); + return 1; + } + }; + let raw = conn.as_raw_fd(); + let joinf = Frame { + typ: proto::TYPE_JOIN, + payload: nick.as_bytes().to_vec(), + }; + if let Err(e) = proto::write_frame(raw, &joinf) { + eprintln!("chatterd: send: join: {e}"); + return 1; + } + let msgf = Frame { + typ: proto::TYPE_MSG, + payload: text.as_bytes().to_vec(), + }; + if let Err(e) = proto::write_frame(raw, &msgf) { + eprintln!("chatterd: send: msg: {e}"); + return 1; + } + thread::sleep(Duration::from_millis(150)); // let the write actually flush before closing + println!("chatterd: sent nick={nick} text={text}"); + 0 +} + +pub fn listen(host: &str, port: u16, nick: &str, timeout_ms: i64) -> i32 { + let conn = match connect_tcp(host, port, 3000) { + Ok(c) => c, + Err(e) => { + eprintln!("chatterd: listen: dial {host}:{port}: {e}"); + return 1; + } + }; + let raw = conn.as_raw_fd(); + let joinf = Frame { + typ: proto::TYPE_JOIN, + payload: nick.as_bytes().to_vec(), + }; + if let Err(e) = proto::write_frame(raw, &joinf) { + eprintln!("chatterd: listen: join: {e}"); + return 1; + } + let _ = setsockopt( + &conn, + sockopt::ReceiveTimeout, + &TimeVal::milliseconds(timeout_ms.max(0)), + ); + + loop { + let f = match proto::read_frame(raw) { + Ok(f) => f, + Err(_) => { + eprintln!("chatterd: listen timeout"); + return 1; + } + }; + if f.typ != proto::TYPE_DELIVER { + continue; + } + let Some((from_nick, text)) = proto::split_deliver(&f.payload) else { + continue; + }; + println!("chatterd: received from={from_nick} text={text}"); + return 0; + } +} diff --git a/examples/41-capstone-fleet/rust/src/fwatch.rs b/examples/41-capstone-fleet/rust/src/fwatch.rs new file mode 100644 index 0000000..07dc942 --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/fwatch.rs @@ -0,0 +1,206 @@ +// fwatch.rs — the book's recurring file watcher (ch07 polling -> ch09 +// inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet +// needs: watch a directory, print one line per raw create/write/delete +// inotify event (no debounce/merge — go/fwatch.go prints every event as it +// arrives, and this port matches that exactly, unlike ex09/ex33's fuller +// debounced watcher), and optionally run under a Landlock ruleset that +// restricts reads to that directory. Landlock is wired exactly as ex33's +// fwatch does (the `landlock` crate for the ruleset, a raw syscall(2) ABI +// probe since landlock_create_ruleset has no libc/crate wrapper for the +// version-query form); the inotify fd comes from ex09's `nix::sys::inotify` +// usage. Full seccomp allow-listing was proven once in ex33 and is not +// re-derived here — capability-bounding-set drop (caps.rs) plus Landlock are +// this capstone's sandboxing layers, applied by pmon before it execs fwatch. +use std::fs; +use std::os::fd::AsFd; +use std::path::Path; +use std::sync::atomic::Ordering; +use std::time::{Duration, Instant}; + +use landlock::{ + AccessFs, PathBeneath, PathFd, Ruleset, RulesetAttr, RulesetCreatedAttr, RulesetStatus, +}; +use nix::errno::Errno; +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sys::inotify::{AddWatchFlags, InitFlags, Inotify}; + +use crate::util; + +// --------------------------------------------------------------------------- +// Landlock — raw syscall for the ABI probe (no libc/crate wrapper exists for +// the version-query form of landlock_create_ruleset); the ruleset itself +// goes through the `landlock` crate, exactly ex33's technique. +// --------------------------------------------------------------------------- + +#[cfg(target_arch = "x86_64")] +const SYS_LANDLOCK_CREATE_RULESET: libc::c_long = 444; +const LANDLOCK_CREATE_RULESET_VERSION: libc::c_int = 1 << 0; + +/// Probes the running kernel's supported Landlock ABI version (0 if +/// Landlock isn't built in / enabled at boot). Never hardcoded. +fn landlock_abi() -> i32 { + // SAFETY: landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION) + // is documented to only ever query the ABI version; it touches no memory. + let v = unsafe { + libc::syscall( + SYS_LANDLOCK_CREATE_RULESET, + std::ptr::null::(), + 0usize, + LANDLOCK_CREATE_RULESET_VERSION, + ) + }; + if v < 0 { 0 } else { v as i32 } +} + +/// Restricts filesystem reads to `dir` (ReadFile + ReadDir, both available +/// since ABI v1) and confirms the kernel actually enforced it (not merely +/// "best-effort, kernel too old"). +fn apply_landlock(dir: &str) -> Result<(), String> { + let access = AccessFs::ReadFile | AccessFs::ReadDir; + let status = Ruleset::default() + .handle_access(access) + .map_err(|e| format!("landlock: handle_access: {e}"))? + .create() + .map_err(|e| format!("landlock: create ruleset: {e}"))? + .add_rule(PathBeneath::new( + PathFd::new(dir).map_err(|e| format!("landlock: open {dir}: {e}"))?, + access, + )) + .map_err(|e| format!("landlock: add_rule: {e}"))? + .restrict_self() + .map_err(|e| format!("landlock: restrict_self: {e}"))?; + if status.ruleset != RulesetStatus::FullyEnforced { + return Err(format!( + "landlock ruleset not fully enforced by this kernel ({:?})", + status.ruleset + )); + } + Ok(()) +} + +// classify mirrors go/fwatch.go's eventName: create, then delete, then +// modify (CLOSE_WRITE or MODIFY), else "other" — checked in that priority +// order because a single inotify event's mask can carry more than one bit. +fn classify(mask: AddWatchFlags) -> &'static str { + if mask.contains(AddWatchFlags::IN_CREATE) { + "create" + } else if mask.contains(AddWatchFlags::IN_DELETE) { + "delete" + } else if mask.intersects(AddWatchFlags::IN_CLOSE_WRITE | AddWatchFlags::IN_MODIFY) { + "modify" + } else { + "other" + } +} + +pub fn snapshot(dir: &str) -> i32 { + let entries = match fs::read_dir(dir) { + Ok(e) => e, + Err(e) => { + eprintln!("fwatch: readdir {dir}: {e}"); + return 1; + } + }; + // os.ReadDir (the Go reference) returns entries sorted by filename; + // std::fs::read_dir makes no such guarantee, so the names are collected + // and sorted explicitly to keep this deterministic across languages. + let mut names: Vec = Vec::new(); + for entry in entries.flatten() { + if entry.file_type().map(|ft| ft.is_file()).unwrap_or(false) { + names.push(entry.file_name().to_string_lossy().into_owned()); + } + } + names.sort(); + for name in names { + println!("{name}"); + } + 0 +} + +pub fn watch(dir: &str, sandbox: bool, timeout_ms: i64) -> i32 { + if sandbox { + let abi = landlock_abi(); + if abi == 0 { + eprintln!("fwatch: Landlock not supported by this kernel"); + return 1; + } + if let Err(e) = apply_landlock(dir) { + eprintln!("fwatch: landlock: {e}"); + return 1; + } + eprintln!("fwatch: landlock ABI={abi} enforced dir={dir}"); + } + + let inotify = match Inotify::init(InitFlags::IN_CLOEXEC | InitFlags::IN_NONBLOCK) { + Ok(i) => i, + Err(e) => { + eprintln!("fwatch: inotify_init1: {e}"); + return 1; + } + }; + let mask = AddWatchFlags::IN_CREATE + | AddWatchFlags::IN_MODIFY + | AddWatchFlags::IN_DELETE + | AddWatchFlags::IN_CLOSE_WRITE; + if let Err(e) = inotify.add_watch(dir, mask) { + eprintln!("fwatch: inotify_add_watch {dir}: {e}"); + return 1; + } + + eprintln!("fwatch: watching {dir} (sandbox={sandbox})"); + + let deadline = + (timeout_ms > 0).then(|| Instant::now() + Duration::from_millis(timeout_ms as u64)); + let sig = util::install_signal_flag(); + + loop { + if sig.load(Ordering::Relaxed) { + println!("(signal)"); + return 0; + } + if let Some(d) = deadline + && Instant::now() > d + { + println!("(timeout)"); + return 0; + } + + let mut fds = [PollFd::new(inotify.as_fd(), PollFlags::POLLIN)]; + match poll(&mut fds, PollTimeout::from(200u16)) { + Ok(0) => continue, + Ok(_) => {} + Err(Errno::EINTR) => continue, + Err(e) => { + eprintln!("fwatch: poll: {e}"); + return 1; + } + } + let ready = fds[0] + .revents() + .map(|r| r.contains(PollFlags::POLLIN)) + .unwrap_or(false); + if !ready { + continue; + } + + loop { + match inotify.read_events() { + Ok(batch) => { + for ev in batch { + let Some(name) = ev.name else { + continue; // event on the directory itself + }; + let kind = classify(ev.mask); + let path = Path::new(dir).join(name.to_string_lossy().into_owned()); + println!("event: {kind} {}", path.display()); + } + } + Err(Errno::EAGAIN) => break, // queue drained + Err(e) => { + eprintln!("fwatch: read: {e}"); + return 1; + } + } + } + } +} diff --git a/examples/41-capstone-fleet/rust/src/main.rs b/examples/41-capstone-fleet/rust/src/main.rs index 9536d12..79d5f4d 100644 --- a/examples/41-capstone-fleet/rust/src/main.rs +++ b/examples/41-capstone-fleet/rust/src/main.rs @@ -1,12 +1,239 @@ -use rustix::system::uname; +// main.rs — 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch +// across two lab hosts, capability-dropped and Landlock-sandboxed, all +// telemetry exported to the host LGTM stack. One binary, four subcommands +// (the same self-re-exec shape ch34's container entrypoint uses): pmon is +// the fleet's init; chatterd/sysagent/fwatch are the services it supervises. +// +// This file is a direct port of go/main.go's flag parsing +// (argFlag/hasFlag/firstPositional/atoiOr) and dispatch table — the usage +// banner text below is byte-identical to the Go reference's (cross-checked +// against cpp/src/main.cpp's std::print of the same banner), since +// verify.lua and every language's demo.sh depend on it matching exactly. +mod caps; +mod chatterd; +mod fwatch; +mod pmon; +mod proto; +mod sysagent; +mod telemetry; +mod util; -fn main() { - let info = uname(); - println!( - "pid {} on {} {} ({})", - std::process::id(), - info.sysname().to_string_lossy(), - info.release().to_string_lossy(), - info.machine().to_string_lossy(), +use std::sync::Arc; + +fn usage() { + eprint!( + "usage: app + pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME] + [--chatterd-port P] [--health-interval-ms N] + chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME] + chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T] + chatterd listen --host H --port P --nick NICK [--timeout-ms T] + sysagent [--node NAME] [--interval-ms N] [--once] + sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M] + fwatch snapshot DIR + fwatch watch DIR [--sandbox] [--timeout-ms T] +" ); } + +/// arg_flag scans args for "--name value" and returns (value, true), or +/// (def, true) if absent, or (_, false) if the flag is present but has no +/// following value — mirrors go/main.go's argFlag exactly. +fn arg_flag(args: &[String], name: &str, def: &str) -> (String, bool) { + for (i, a) in args.iter().enumerate() { + if a == name { + return match args.get(i + 1) { + Some(v) => (v.clone(), true), + None => (String::new(), false), + }; + } + } + (def.to_string(), true) +} + +fn has_flag(args: &[String], name: &str) -> bool { + args.iter().any(|a| a == name) +} + +/// first_positional: walks args, treating every "--xxx" token as consuming +/// exactly one following token (its value), and returns the first token that +/// survives that skipping — mirrors go/main.go's firstPositional exactly, +/// including its "skip the very next token no matter what it is" quirk. +fn first_positional(args: &[String]) -> String { + let mut skip = false; + for a in args { + if skip { + skip = false; + continue; + } + if a.len() >= 2 && a.starts_with("--") { + skip = true; + continue; + } + return a.clone(); + } + String::new() +} + +fn atoi_or(s: &str, def: i64) -> i64 { + if s.is_empty() { + return def; + } + s.parse::().unwrap_or(def) +} + +fn main() { + let args: Vec = std::env::args().collect(); + if args.len() < 2 { + usage(); + std::process::exit(2); + } + let cmd = args[1].as_str(); + let rest: Vec = args[2..].to_vec(); + + let code = match cmd { + "pmon" => { + let (node, ok1) = arg_flag(&rest, "--node", "node1"); + let (sandbox_dir, ok2) = arg_flag(&rest, "--sandbox-dir", "/tmp/fwatch-sandbox"); + let (peer, ok3) = arg_flag(&rest, "--peer", ""); + let (peer_node, ok4) = arg_flag(&rest, "--peer-node", ""); + let (port_s, ok5) = arg_flag(&rest, "--chatterd-port", "47100"); + let (health_s, ok6) = arg_flag(&rest, "--health-interval-ms", "2000"); + if !(ok1 && ok2 && ok3 && ok4 && ok5 && ok6) { + usage(); + std::process::exit(2); + } + pmon::run( + &node, + &sandbox_dir, + &peer, + &peer_node, + atoi_or(&port_s, 47100) as u16, + atoi_or(&health_s, 2000) as u64, + ) + } + + "chatterd" => { + if rest.is_empty() { + usage(); + std::process::exit(2); + } + let args = &rest[1..]; + match rest[0].as_str() { + "serve" => { + let (host, _) = arg_flag(args, "--host", "0.0.0.0"); + let (port_s, _) = arg_flag(args, "--port", "47100"); + let (node, _) = arg_flag(args, "--node", "node1"); + let (peer, _) = arg_flag(args, "--peer", ""); + let (peer_node, _) = arg_flag(args, "--peer-node", ""); + let port = atoi_or(&port_s, 47100) as u16; + let tel = Arc::new(telemetry::init("chatterd", &node)); + let code = chatterd::serve(&host, port, &node, &peer, &peer_node, tel.clone()); + tel.shutdown(); + code + } + "send" => { + let (host, _) = arg_flag(args, "--host", "127.0.0.1"); + let (port_s, _) = arg_flag(args, "--port", "47100"); + let (nick, ok1) = arg_flag(args, "--nick", ""); + let (text, ok2) = arg_flag(args, "--text", ""); + let (timeout_s, _) = arg_flag(args, "--timeout-ms", "3000"); + if !ok1 || !ok2 || nick.is_empty() || text.is_empty() { + usage(); + std::process::exit(2); + } + chatterd::send( + &host, + atoi_or(&port_s, 47100) as u16, + &nick, + &text, + atoi_or(&timeout_s, 3000), + ) + } + "listen" => { + let (host, _) = arg_flag(args, "--host", "127.0.0.1"); + let (port_s, _) = arg_flag(args, "--port", "47100"); + let (nick, ok1) = arg_flag(args, "--nick", ""); + let (timeout_s, _) = arg_flag(args, "--timeout-ms", "5000"); + if !ok1 || nick.is_empty() { + usage(); + std::process::exit(2); + } + chatterd::listen( + &host, + atoi_or(&port_s, 47100) as u16, + &nick, + atoi_or(&timeout_s, 5000), + ) + } + _ => { + usage(); + std::process::exit(2); + } + } + } + + "sysagent" => { + if !rest.is_empty() && rest[0] == "saturate" { + let args = &rest[1..]; + let (resource, _) = arg_flag(args, "--resource", ""); + let (seconds_s, _) = arg_flag(args, "--seconds", "10"); + let (workers_s, _) = arg_flag(args, "--workers", "0"); + let (mb_s, _) = arg_flag(args, "--mb", "0"); + sysagent::saturate( + &resource, + atoi_or(&seconds_s, 10), + atoi_or(&workers_s, 0), + atoi_or(&mb_s, 0), + ) + } else { + let (node, _) = arg_flag(&rest, "--node", "node1"); + let (interval_s, _) = arg_flag(&rest, "--interval-ms", "2000"); + let once = has_flag(&rest, "--once"); + let tel = telemetry::init("sysagent", &node); + let code = sysagent::run(&node, atoi_or(&interval_s, 2000) as u64, once, &tel); + tel.shutdown(); + code + } + } + + "fwatch" => { + if rest.is_empty() { + usage(); + std::process::exit(2); + } + let args = &rest[1..]; + match rest[0].as_str() { + "snapshot" => { + let dir = first_positional(args); + if dir.is_empty() { + usage(); + std::process::exit(2); + } + fwatch::snapshot(&dir) + } + "watch" => { + let dir = first_positional(args); + if dir.is_empty() { + usage(); + std::process::exit(2); + } + let sandbox = has_flag(args, "--sandbox"); + let (timeout_s, _) = arg_flag(args, "--timeout-ms", "0"); + fwatch::watch(&dir, sandbox, atoi_or(&timeout_s, 0)) + } + _ => { + usage(); + std::process::exit(2); + } + } + } + + _ => { + usage(); + std::process::exit(2); + } + }; + + std::process::exit(code); +} diff --git a/examples/41-capstone-fleet/rust/src/pmon.rs b/examples/41-capstone-fleet/rust/src/pmon.rs new file mode 100644 index 0000000..f94e422 --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/pmon.rs @@ -0,0 +1,302 @@ +// pmon.rs — the book's recurring process supervisor (ch11-14, 18-19, 32, +// 34), grown here into the capstone's fleet init: it drops the capability +// bounding set once (caps.rs), then fork/execs (self-re-exec via +// /proc/self/exe, the same technique ch34's container entrypoint uses) +// chatterd, sysagent, and fwatch as children, restarting any of them that +// exits unexpectedly, forwarding SIGTERM/SIGINT to all three on its own +// shutdown, and printing a health line every tick so an operator (or +// verify.lua, over ssh) can poll fleet state without touching /proc. +// +// One thread per supervised service (matching pmon.cpp's std::thread pool), +// each doing its own fork/waitpid retry loop; a dedicated thread blocks +// SIGTERM/SIGINT and consumes them via sigwait(2) (nix's `SigSet::wait`), +// exactly pmon.cpp's rationale for using sigwait over a signal handler: it +// avoids doing anything async-signal-unsafe on delivery. Every forked child +// clears its own inherited copy of that mask right after fork, before +// execve, so a SIGTERM forwarded to it can actually reach its own handling +// (the C++ port documents the same fix, carried over from ch34's container +// entrypoint). +use std::collections::HashMap; +use std::ffi::CString; +use std::fs; +use std::os::unix::ffi::OsStrExt; +use std::sync::atomic::{AtomicBool, AtomicI32, AtomicI64, Ordering}; +use std::sync::{Arc, Mutex}; +use std::thread; +use std::time::Duration; + +use nix::sys::signal::{SigSet, Signal, kill}; +use nix::sys::wait::{WaitStatus, waitpid}; +use nix::unistd::{ForkResult, Pid, execv, fork}; + +use crate::caps; + +struct ChildSpec { + name: &'static str, + args: Vec, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum State { + Starting, + Up, + Restarting, + Down, +} + +impl State { + fn name(self) -> &'static str { + match self { + State::Starting => "starting", + State::Up => "up", + State::Restarting => "restarting", + State::Down => "down", + } + } +} + +struct ChildState { + state: Mutex, + restarts: AtomicI64, + pid: AtomicI32, +} + +impl ChildState { + fn new() -> Self { + ChildState { + state: Mutex::new(State::Starting), + restarts: AtomicI64::new(0), + pid: AtomicI32::new(0), + } + } + fn set_state(&self, s: State) { + *self.state.lock().unwrap() = s; + } + fn get_state(&self) -> State { + *self.state.lock().unwrap() + } +} + +fn describe_wait_status(status: nix::Result) -> String { + match status { + Ok(WaitStatus::Exited(_, code)) => format!("exit status {code}"), + Ok(WaitStatus::Signaled(_, sig, _)) => format!("signal: {sig}"), + Ok(other) => format!("{other:?}"), + Err(e) => format!("waitpid: {e}"), + } +} + +// run_child_supervisor: fork+exec `self_exe args...` in a loop, restarting on +// unexpected exit — the per-service goroutine from go/pmon.go, one OS thread +// per service here too. +fn run_child_supervisor( + self_exe: &CString, + args: &[String], + cs: &ChildState, + name: &str, + shutting_down: &AtomicBool, +) { + // Every CString is built once, before the first fork, so the only work + // the child performs between fork() and execv() is clearing its signal + // mask and calling execv() itself. + let mut argv: Vec = Vec::with_capacity(args.len() + 1); + argv.push(self_exe.clone()); + for a in args { + argv.push(CString::new(a.as_str()).expect("child arg contains NUL")); + } + + loop { + // SAFETY: this process is multi-threaded (one supervisor thread per + // service plus the sigwait thread below); the child branch performs + // only the signal-mask reset and execv() before either replacing + // itself or exiting, matching pmon.cpp's fork-then-immediately-exec + // discipline. + match unsafe { fork() } { + Ok(ForkResult::Child) => { + // Unblock what pmon::run blocked in the parent (see below): + // a forwarded SIGTERM must reach this child's own handling, + // not sit blocked forever. + let empty = SigSet::empty(); + let _ = empty.thread_set_mask(); + match execv(self_exe, &argv) { + Ok(_) => unreachable!(), + Err(_) => unsafe { libc::_exit(127) }, // execv failed + } + } + Ok(ForkResult::Parent { child }) => { + cs.pid.store(child.as_raw(), Ordering::Relaxed); + cs.set_state(State::Up); + println!("pmon: started service={name} pid={}", child.as_raw()); + + let status = waitpid(child, None); + if shutting_down.load(Ordering::Relaxed) { + cs.set_state(State::Down); + return; + } + let n = cs.restarts.fetch_add(1, Ordering::Relaxed) + 1; + cs.set_state(State::Restarting); + let reason = describe_wait_status(status); + println!("pmon: restart service={name} attempt={n} reason={reason}"); + thread::sleep(Duration::from_millis(300)); + } + Err(e) => { + eprintln!("pmon: start service={name}: {e}"); + thread::sleep(Duration::from_millis(500)); + } + } + } +} + +fn print_health(order: &[&str], states: &HashMap<&str, Arc>) { + let mut line = String::from("pmon: health "); + for (i, name) in order.iter().enumerate() { + if i > 0 { + line.push(' '); + } + line.push_str(name); + line.push('='); + line.push_str(states[name].get_state().name()); + } + line.push_str(" restarts="); + for (i, name) in order.iter().enumerate() { + if i > 0 { + line.push(','); + } + line.push_str(name); + line.push(':'); + line.push_str(&states[name].restarts.load(Ordering::Relaxed).to_string()); + } + println!("{line}"); +} + +pub fn run( + node: &str, + sandbox_dir: &str, + peer: &str, + peer_node: &str, + chatterd_port: u16, + health_interval_ms: u64, +) -> i32 { + if let Err(e) = fs::create_dir_all(sandbox_dir) { + eprintln!("pmon: mkdir {sandbox_dir}: {e}"); + return 1; + } + + caps::drop_bounding_set(); + + let self_exe_path = match fs::read_link("/proc/self/exe") { + Ok(p) => p, + Err(e) => { + eprintln!("pmon: readlink /proc/self/exe: {e}"); + return 1; + } + }; + let self_exe = match CString::new(self_exe_path.as_os_str().as_bytes()) { + Ok(c) => c, + Err(_) => { + eprintln!("pmon: self exe path contains NUL"); + return 1; + } + }; + + let mut chatterd_args = vec![ + "chatterd".to_string(), + "serve".to_string(), + "--host".to_string(), + "0.0.0.0".to_string(), + "--port".to_string(), + chatterd_port.to_string(), + "--node".to_string(), + node.to_string(), + ]; + if !peer.is_empty() { + chatterd_args.push("--peer".to_string()); + chatterd_args.push(peer.to_string()); + chatterd_args.push("--peer-node".to_string()); + chatterd_args.push(peer_node.to_string()); + } + + let specs = vec![ + ChildSpec { + name: "chatterd", + args: chatterd_args, + }, + ChildSpec { + name: "sysagent", + args: vec![ + "sysagent".to_string(), + "--node".to_string(), + node.to_string(), + "--interval-ms".to_string(), + "2000".to_string(), + ], + }, + ChildSpec { + name: "fwatch", + args: vec![ + "fwatch".to_string(), + "watch".to_string(), + sandbox_dir.to_string(), + "--sandbox".to_string(), + ], + }, + ]; + + let order: [&'static str; 3] = ["chatterd", "sysagent", "fwatch"]; + let mut states: HashMap<&'static str, Arc> = HashMap::new(); + for name in order { + states.insert(name, Arc::new(ChildState::new())); + } + + let shutting_down = Arc::new(AtomicBool::new(false)); + + // Block SIGTERM/SIGINT in THIS thread before spawning anything: the + // dedicated sigwait thread below is the only one that ever consumes + // them; every forked child unblocks its own inherited copy of the mask + // before execve (see run_child_supervisor). + let mut mask = SigSet::empty(); + mask.add(Signal::SIGTERM); + mask.add(Signal::SIGINT); + mask.thread_block().expect("pthread_sigmask"); + + { + let shutting_down = shutting_down.clone(); + let states_for_signal: Vec> = + order.iter().map(|n| states[n].clone()).collect(); + thread::spawn(move || { + if mask.wait().is_ok() { + shutting_down.store(true, Ordering::Relaxed); + for cs in &states_for_signal { + let pid = cs.pid.load(Ordering::Relaxed); + if pid > 0 { + let _ = kill(Pid::from_raw(pid), Signal::SIGTERM); + } + } + } + }); + } + + let mut worker_handles = Vec::with_capacity(specs.len()); + for spec in specs { + let cs = states[spec.name].clone(); + let self_exe = self_exe.clone(); + let shutting_down = shutting_down.clone(); + worker_handles.push(thread::spawn(move || { + run_child_supervisor(&self_exe, &spec.args, &cs, spec.name, &shutting_down); + })); + } + + loop { + thread::sleep(Duration::from_millis(health_interval_ms)); + if worker_handles.iter().all(|h| h.is_finished()) { + break; + } + print_health(&order, &states); + } + + println!("pmon: shutdown"); + for h in worker_handles { + let _ = h.join(); + } + 0 +} diff --git a/examples/41-capstone-fleet/rust/src/proto.rs b/examples/41-capstone-fleet/rust/src/proto.rs new file mode 100644 index 0000000..96d6311 --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/proto.rs @@ -0,0 +1,107 @@ +// proto.rs — the canonical chatterd chat frame (introduced ch21, unchanged +// since): magic "CH", version 1, one byte type, big-endian u16 length, then +// payload. This capstone speaks only JOIN/MSG/DELIVER, the same three types +// every prior chatterd chapter uses. +use std::os::fd::{BorrowedFd, RawFd}; + +use nix::errno::Errno; +use nix::unistd::{read, write}; + +pub const MAGIC0: u8 = b'C'; +pub const MAGIC1: u8 = b'H'; +pub const VERSION: u8 = 1; + +pub const TYPE_JOIN: u8 = 1; +pub const TYPE_MSG: u8 = 2; +pub const TYPE_DELIVER: u8 = 3; + +pub struct Frame { + pub typ: u8, + pub payload: Vec, +} + +fn write_all(fd: RawFd, buf: &[u8]) -> std::io::Result<()> { + let bfd = unsafe { BorrowedFd::borrow_raw(fd) }; + let mut off = 0; + while off < buf.len() { + match write(bfd, &buf[off..]) { + Ok(0) => return Err(std::io::Error::from(std::io::ErrorKind::WriteZero)), + Ok(n) => off += n, + Err(Errno::EINTR) => continue, + Err(e) => return Err(e.into()), + } + } + Ok(()) +} + +fn read_all(fd: RawFd, buf: &mut [u8]) -> std::io::Result<()> { + let bfd = unsafe { BorrowedFd::borrow_raw(fd) }; + let mut off = 0; + while off < buf.len() { + match read(bfd, &mut buf[off..]) { + Ok(0) => return Err(std::io::Error::from(std::io::ErrorKind::UnexpectedEof)), + Ok(n) => off += n, + Err(Errno::EINTR) => continue, + Err(e) => return Err(e.into()), + } + } + Ok(()) +} + +/// Writes the 6-byte header + payload to fd. Errors on a short/failed write +/// or an oversized payload. +pub fn write_frame(fd: RawFd, f: &Frame) -> std::io::Result<()> { + if f.payload.len() > 0xFFFF { + return Err(std::io::Error::other(format!( + "payload too large: {} bytes", + f.payload.len() + ))); + } + let mut hdr = [0u8; 6]; + hdr[0] = MAGIC0; + hdr[1] = MAGIC1; + hdr[2] = VERSION; + hdr[3] = f.typ; + hdr[4..6].copy_from_slice(&(f.payload.len() as u16).to_be_bytes()); + write_all(fd, &hdr)?; + if !f.payload.is_empty() { + write_all(fd, &f.payload)?; + } + Ok(()) +} + +/// Reads one frame from fd (blocking on a normal, non-nonblocking socket). +/// Errors on EOF/short-read/bad magic. +pub fn read_frame(fd: RawFd) -> std::io::Result { + let mut hdr = [0u8; 6]; + read_all(fd, &mut hdr)?; + if hdr[0] != MAGIC0 || hdr[1] != MAGIC1 || hdr[2] != VERSION { + return Err(std::io::Error::other("bad frame magic/version")); + } + let len = u16::from_be_bytes([hdr[4], hdr[5]]) as usize; + let mut payload = vec![0u8; len]; + if len > 0 { + read_all(fd, &mut payload)?; + } + Ok(Frame { + typ: hdr[3], + payload, + }) +} + +/// Builds the ch21 "nick NUL text" DELIVER payload. +pub fn deliver_payload(nick: &str, text: &str) -> Vec { + let mut out = Vec::with_capacity(nick.len() + 1 + text.len()); + out.extend_from_slice(nick.as_bytes()); + out.push(0); + out.extend_from_slice(text.as_bytes()); + out +} + +/// Reverses deliver_payload; None if there is no NUL. +pub fn split_deliver(payload: &[u8]) -> Option<(String, String)> { + let pos = payload.iter().position(|&b| b == 0)?; + let nick = String::from_utf8_lossy(&payload[..pos]).into_owned(); + let text = String::from_utf8_lossy(&payload[pos + 1..]).into_owned(); + Some((nick, text)) +} diff --git a/examples/41-capstone-fleet/rust/src/sysagent.rs b/examples/41-capstone-fleet/rust/src/sysagent.rs new file mode 100644 index 0000000..7ca4e2c --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/sysagent.rs @@ -0,0 +1,209 @@ +// sysagent.rs — /proc/stat, /proc/meminfo, /proc/loadavg readers and the +// sample loop, ported field-for-field from go/sysagent.go so the printed +// cpu_pct/mem_pct/load1 numbers are computed by the identical formula (not +// just "close enough"): idle is specifically the 4th space-separated value +// on /proc/stat's "cpu " line, and total is the sum of every value on that +// line including guest/guest_nice, exactly as the Go reference sums them. +use std::fs; +use std::sync::atomic::Ordering; +use std::thread; +use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; + +use crate::telemetry::{self, Telemetry}; +use crate::util; + +#[derive(Default, Clone, Copy)] +struct CpuTimes { + idle: u64, + total: u64, +} + +fn read_cpu_times() -> Result { + let text = fs::read_to_string("/proc/stat").map_err(|e| format!("open /proc/stat: {e}"))?; + let line = text.lines().next().ok_or("empty /proc/stat")?; + let fields: Vec<&str> = line.split_whitespace().collect(); + if fields.len() < 5 || fields[0] != "cpu" { + return Err(format!("unexpected /proc/stat format: {line:?}")); + } + let mut total: u64 = 0; + let mut idle: u64 = 0; + for (i, s) in fields[1..].iter().enumerate() { + let Ok(v) = s.parse::() else { continue }; + total += v; + if i == 3 { + // idle is the 4th value after "cpu" + idle = v; + } + } + Ok(CpuTimes { idle, total }) +} + +fn cpu_pct(prev: &CpuTimes, cur: &CpuTimes) -> f64 { + let d_total = cur.total as f64 - prev.total as f64; + let d_idle = cur.idle as f64 - prev.idle as f64; + if d_total <= 0.0 { + return 0.0; + } + (1.0 - d_idle / d_total) * 100.0 +} + +fn read_mem_pct() -> Result { + let text = + fs::read_to_string("/proc/meminfo").map_err(|e| format!("open /proc/meminfo: {e}"))?; + let mut total = 0.0_f64; + let mut avail = 0.0_f64; + for line in text.lines() { + let mut it = line.split_whitespace(); + let (Some(key), Some(val)) = (it.next(), it.next()) else { + continue; + }; + let Ok(v) = val.parse::() else { continue }; + match key.trim_end_matches(':') { + "MemTotal" => total = v, + "MemAvailable" => avail = v, + _ => {} + } + } + if total <= 0.0 { + return Err("MemTotal not found".to_string()); + } + Ok((1.0 - avail / total) * 100.0) +} + +fn read_load1() -> Result { + let text = + fs::read_to_string("/proc/loadavg").map_err(|e| format!("open /proc/loadavg: {e}"))?; + let field = text + .split_whitespace() + .next() + .ok_or("empty /proc/loadavg")?; + field + .parse::() + .map_err(|e| format!("parse load1: {e}")) +} + +#[derive(Default)] +struct Sample { + cpu_pct: f64, + mem_pct: f64, + load1: f64, +} + +fn take_sample(prev: &mut CpuTimes) -> Result { + let cur = read_cpu_times()?; + let mut s = Sample::default(); + if prev.total > 0 { + s.cpu_pct = cpu_pct(prev, &cur); + } + *prev = cur; + s.mem_pct = read_mem_pct()?; + s.load1 = read_load1()?; + Ok(s) +} + +pub fn run(node: &str, interval_ms: u64, once: bool, tel: &Telemetry) -> i32 { + let gauges = telemetry::register_sysagent_gauges(tel, node); + let sig = util::install_signal_flag(); + + // Prime the CPU-time baseline so the first printed sample isn't a bogus + // 0 (matches Go: read once, sleep 200ms, THEN start the sample loop). + let mut prev = read_cpu_times().unwrap_or_default(); + thread::sleep(Duration::from_millis(200)); + + loop { + let s = match take_sample(&mut prev) { + Ok(s) => s, + Err(e) => { + eprintln!("sysagent: sample: {e}"); + return 1; + } + }; + if tel.enabled { + gauges.update(telemetry::GaugeSample { + cpu_pct: s.cpu_pct, + mem_pct: s.mem_pct, + load1: s.load1, + }); + } + let ts = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + println!( + "sysagent: node={node} cpu_pct={:.2} mem_pct={:.2} load1={:.2} ts={ts}", + s.cpu_pct, s.mem_pct, s.load1 + ); + + if once { + break; + } + if sig.load(Ordering::Relaxed) { + break; + } + thread::sleep(Duration::from_millis(interval_ms)); + if sig.load(Ordering::Relaxed) { + break; + } + } + if tel.enabled { + telemetry::print_export_errors("sysagent"); + } + 0 +} + +// saturate — chaos helper: busy-spin threads (cpu) or hold allocated, +// touched memory (mem) for --seconds, so a concurrently-running sysagent's +// cpu_pct/mem_pct visibly rises (this fleet's USE-method callback). +pub fn saturate(resource: &str, seconds: i64, workers: i64, mb: i64) -> i32 { + let seconds = seconds.max(0) as u64; + match resource { + "cpu" => { + let workers = if workers <= 0 { + std::thread::available_parallelism() + .map(|n| n.get() as i64) + .unwrap_or(1) + } else { + workers + }; + println!("sysagent: saturate resource=cpu seconds={seconds} workers={workers} started"); + let stop = Instant::now() + Duration::from_secs(seconds); + let mut handles = Vec::with_capacity(workers as usize); + for _ in 0..workers { + handles.push(thread::spawn(move || { + let mut x = 0.0001_f64; + while Instant::now() < stop { + x = x * 1.0000001 + 1.0; + } + if x == 0.0 { + // never true; keeps the compiler from eliding the loop + println!("{x}"); + } + })); + } + for h in handles { + let _ = h.join(); + } + println!("sysagent: saturate done"); + 0 + } + "mem" => { + let mb = if mb <= 0 { 256 } else { mb }; + println!("sysagent: saturate resource=mem seconds={seconds} mb={mb} started"); + let n = mb as usize * 1024 * 1024; + let mut buf = vec![0u8; n]; + for (i, b) in buf.iter_mut().enumerate() { + *b = i as u8; // touch every page so it's resident, not just reserved + } + thread::sleep(Duration::from_secs(seconds)); + let _ = buf[0]; + println!("sysagent: saturate done"); + 0 + } + _ => { + eprintln!( + "usage: sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M]" + ); + 2 + } + } +} diff --git a/examples/41-capstone-fleet/rust/src/telemetry.rs b/examples/41-capstone-fleet/rust/src/telemetry.rs new file mode 100644 index 0000000..fa0c713 --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/telemetry.rs @@ -0,0 +1,297 @@ +// telemetry.rs — OTel wiring shared by chatterd and sysagent. Both signals +// (traces for chatterd deliveries, metrics for sysagent's /proc samples) are +// exported over OTLP/HTTP. Endpoint comes from the standard +// OTEL_EXPORTER_OTLP_ENDPOINT env var — this init deliberately never calls +// `.with_endpoint()`: opentelemetry-otlp 0.32's exporters only append the +// per-signal "/v1/traces" / "/v1/metrics" suffix when they resolve the +// endpoint themselves from OTEL_EXPORTER_OTLP_ENDPOINT (an explicit +// with_endpoint() is taken as the FULL, already-suffixed URL and posted to +// verbatim). Building with default options and letting the exporter read the +// env var is what makes this match go/telemetry.go, whose exporters read the +// same variable the same way. +// +// The SDK is kept silent on purpose: opentelemetry-rust only surfaces +// export failures through its internal `otel_error!` tracing events, which +// require a `tracing` subscriber to go anywhere. ex38 already established +// that installing one (even a filtered one) risks corrupting the stdout/ +// stderr contract this capstone's verify.lua depends on line-for-line — so +// none is installed here, and (unlike go/telemetry.go's exportErrors counter, +// fed by otel.SetErrorHandler) this port has no live export-failure signal to +// report; the "otel export_errors=" line below is therefore always 0, a +// known, documented gap rather than a claim of a working counter. +use std::sync::Mutex; +use std::sync::atomic::{AtomicI64, Ordering}; + +use anyhow::{Context, Result}; +use opentelemetry::metrics::{Meter, MeterProvider as _}; +use opentelemetry::trace::{Span as _, Status, Tracer as _, TracerProvider as _}; +use opentelemetry::{Context as OtelContext, KeyValue}; +use opentelemetry_otlp::{MetricExporter, SpanExporter}; +use opentelemetry_sdk::Resource; +use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider}; +use opentelemetry_sdk::trace::{ + BatchConfigBuilder, BatchSpanProcessor, Sampler, SdkTracer, SdkTracerProvider, +}; + +const SERVICE_VERSION: &str = "41-capstone-fleet"; +const TRACE_BATCH_DELAY: std::time::Duration = std::time::Duration::from_millis(500); +const METRIC_EXPORT_INTERVAL: std::time::Duration = std::time::Duration::from_secs(2); + +/// Count of OTel export failures. Always 0 in this port (see the module doc +/// comment): opentelemetry-rust 0.32 has no equivalent of Go's +/// `otel.SetErrorHandler` reachable without a `tracing` subscriber. +pub static EXPORT_ERRORS: AtomicI64 = AtomicI64::new(0); + +/// Owns the trace/metric providers (kept alive so shutdown() can flush them) +/// plus the tracer/meter callers record against. When telemetry is disabled +/// every Option is None; callers gate on `enabled` before touching them. +pub struct Telemetry { + pub enabled: bool, + tracer_provider: Option, + meter_provider: Option, + tracer: Option, + meter: Option, +} + +impl Telemetry { + fn disabled() -> Self { + Telemetry { + enabled: false, + tracer_provider: None, + meter_provider: None, + tracer: None, + meter: None, + } + } + + pub fn tracer(&self) -> Option<&SdkTracer> { + self.tracer.as_ref() + } + + pub fn meter(&self) -> Option<&Meter> { + self.meter.as_ref() + } + + /// Flushes and shuts down both providers, best effort. A no-op when + /// telemetry is disabled. + pub fn shutdown(&self) { + if !self.enabled { + return; + } + if let Some(tp) = &self.tracer_provider + && let Err(e) = tp.shutdown() + { + eprintln!("app: telemetry shutdown: {e}"); + } + if let Some(mp) = &self.meter_provider + && let Err(e) = mp.shutdown() + { + eprintln!("app: telemetry shutdown: {e}"); + } + } +} + +/// Sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT is set; +/// otherwise returns a disabled, no-op handle (the service still runs — OTel +/// is additive, never a hard dependency of the fleet coming up). `service` is +/// always one of this binary's own subcommand names ("chatterd"/"sysagent"), +/// passed as a `&'static str` literal by every caller — the Meter/Tracer +/// provider APIs require that lifetime. +pub fn init(service: &'static str, node: &str) -> Telemetry { + let endpoint = std::env::var("OTEL_EXPORTER_OTLP_ENDPOINT").unwrap_or_default(); + if endpoint.is_empty() { + eprintln!("{service}: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)"); + return Telemetry::disabled(); + } + + match init_enabled(service, node, &endpoint) { + Ok(t) => { + eprintln!("{service}: otel enabled endpoint={endpoint} node={node}"); + t + } + Err(e) => { + eprintln!("{service}: otel init: {e:#} (telemetry disabled)"); + Telemetry::disabled() + } + } +} + +fn init_enabled(service: &'static str, node: &str, _endpoint: &str) -> Result { + let resource = Resource::builder_empty() + .with_service_name(service.to_string()) + .with_attribute(KeyValue::new("service.version", SERVICE_VERSION)) + .with_attribute(KeyValue::new("deployment.environment", "lab")) + .with_attribute(KeyValue::new("node.name", node.to_string())) + .build(); + + // Deliberately no `.with_endpoint(...)` — see the module doc comment: + // building with default options lets the exporter resolve + // OTEL_EXPORTER_OTLP_ENDPOINT itself and append the per-signal path. + let span_exporter = SpanExporter::builder() + .with_http() + .build() + .context("trace exporter")?; + let batch_processor = BatchSpanProcessor::builder(span_exporter) + .with_batch_config( + BatchConfigBuilder::default() + .with_scheduled_delay(TRACE_BATCH_DELAY) + .build(), + ) + .build(); + let tracer_provider = SdkTracerProvider::builder() + .with_span_processor(batch_processor) + .with_resource(resource.clone()) + .with_sampler(Sampler::AlwaysOn) + .build(); + + let metric_exporter = MetricExporter::builder() + .with_http() + .build() + .context("metric exporter")?; + let reader = PeriodicReader::builder(metric_exporter) + .with_interval(METRIC_EXPORT_INTERVAL) + .build(); + let meter_provider = SdkMeterProvider::builder() + .with_resource(resource) + .with_reader(reader) + .build(); + + let tracer = tracer_provider.tracer(service); + let meter = meter_provider.meter(service); + + Ok(Telemetry { + enabled: true, + tracer_provider: Some(tracer_provider), + meter_provider: Some(meter_provider), + tracer: Some(tracer), + meter: Some(meter), + }) +} + +/// Starts a "chatterd.deliver" span (chatterd.rs's one instrumented +/// operation), matching go/chatterd.go's handleClient span attributes +/// exactly. Returns the span's lowercase-hex trace id alongside the span +/// itself, since callers print it only when telemetry is enabled. +pub fn start_deliver_span( + tel: &Telemetry, + from: &str, + text_len: usize, + node: &str, + from_bridge: bool, +) -> Option<(SdkSpanHandle, String)> { + let tracer = tel.tracer()?; + let span = tracer + .span_builder("chatterd.deliver") + .with_attributes(vec![ + KeyValue::new("chat.from", from.to_string()), + KeyValue::new("chat.text_len", text_len as i64), + KeyValue::new("chat.node", node.to_string()), + KeyValue::new("chat.from_bridge", from_bridge), + ]) + .start_with_context(tracer, &OtelContext::new()); + let trace_id = format!("{}", span.span_context().trace_id()); + Some((SdkSpanHandle(span), trace_id)) +} + +/// A thin wrapper so chatterd.rs doesn't need to name opentelemetry_sdk's +/// concrete span type or import the `Span` trait itself. +pub struct SdkSpanHandle(opentelemetry_sdk::trace::Span); + +impl SdkSpanHandle { + pub fn end_ok(mut self) { + self.0.set_status(Status::Ok); + self.0.end(); + } +} + +/// A shared, atomically-observed sample the sysagent metric callbacks read +/// from — the Rust analogue of go/sysagent.go's mutex-guarded `latest +/// sample` plus RegisterCallback closure, and sysagent.cpp's GaugeState. +#[derive(Default, Clone, Copy)] +pub struct GaugeSample { + pub cpu_pct: f64, + pub mem_pct: f64, + pub load1: f64, +} + +pub struct SysagentGauges { + state: std::sync::Arc>, + // Keeping the builder-returned instrument handles alive is what keeps + // their callbacks registered with the meter for the process lifetime. + _cpu: Option>, + _mem: Option>, + _load: Option>, +} + +impl SysagentGauges { + pub fn update(&self, s: GaugeSample) { + if let Ok(mut guard) = self.state.lock() { + *guard = s; + } + } +} + +/// Registers the three observable gauges (sysagent_cpu_pct, sysagent_mem_pct, +/// sysagent_load1), each carrying a `node` attribute — the exact names and +/// label verify.lua queries through Grafana's Prometheus proxy. A no-op +/// (empty) registration when telemetry is disabled. +pub fn register_sysagent_gauges(tel: &Telemetry, node: &str) -> SysagentGauges { + let state = std::sync::Arc::new(Mutex::new(GaugeSample::default())); + let Some(meter) = tel.meter() else { + return SysagentGauges { + state, + _cpu: None, + _mem: None, + _load: None, + }; + }; + + let node_cpu = node.to_string(); + let state_cpu = state.clone(); + let cpu = meter + .f64_observable_gauge("sysagent_cpu_pct") + .with_description("CPU utilization percent, from /proc/stat") + .with_callback(move |observer| { + let v = state_cpu.lock().map(|g| g.cpu_pct).unwrap_or(0.0); + observer.observe(v, &[KeyValue::new("node", node_cpu.clone())]); + }) + .build(); + + let node_mem = node.to_string(); + let state_mem = state.clone(); + let mem = meter + .f64_observable_gauge("sysagent_mem_pct") + .with_description("Memory-used percent, from /proc/meminfo") + .with_callback(move |observer| { + let v = state_mem.lock().map(|g| g.mem_pct).unwrap_or(0.0); + observer.observe(v, &[KeyValue::new("node", node_mem.clone())]); + }) + .build(); + + let node_load = node.to_string(); + let state_load = state.clone(); + let load1 = meter + .f64_observable_gauge("sysagent_load1") + .with_description("1-minute load average, from /proc/loadavg") + .with_callback(move |observer| { + let v = state_load.lock().map(|g| g.load1).unwrap_or(0.0); + observer.observe(v, &[KeyValue::new("node", node_load.clone())]); + }) + .build(); + + SysagentGauges { + state, + _cpu: Some(cpu), + _mem: Some(mem), + _load: Some(load1), + } +} + +/// Prints the "otel export_errors=N" line sysagent emits when telemetry is +/// enabled (see the module doc comment for why N is always 0 here). +pub fn print_export_errors(service: &str) { + eprintln!( + "{service}: otel export_errors={}", + EXPORT_ERRORS.load(Ordering::Relaxed) + ); +} diff --git a/examples/41-capstone-fleet/rust/src/util.rs b/examples/41-capstone-fleet/rust/src/util.rs new file mode 100644 index 0000000..039090a --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/util.rs @@ -0,0 +1,38 @@ +// util.rs — small helpers shared across the fleet's subcommands, the Rust +// counterpart of util.go's installSignalFlag: chatterd/sysagent/fwatch each +// run their own poll loop already (accept-with-timeout, interval sleep, +// poll(2) with a timeout), so a level-triggered flag is all any of them +// needs from a signal — no EINTR-sensitive syscall unwinding required, unlike +// ex40's fastpath server. pmon (a supervisor, not a poller) does its own +// dedicated child-reaping loop in pmon.rs instead of using this. +use std::sync::OnceLock; +use std::sync::atomic::{AtomicBool, Ordering}; + +use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction}; + +static SIGNAL_FLAG: AtomicBool = AtomicBool::new(false); +static INSTALLED: OnceLock<()> = OnceLock::new(); + +extern "C" fn on_signal(_: libc::c_int) { + SIGNAL_FLAG.store(true, Ordering::Relaxed); +} + +/// Installs a SIGINT/SIGTERM handler (once per process; safe to call from +/// multiple subcommands' code paths) that flips a process-wide flag to true +/// the first time either signal arrives. Returns a reference to that flag. +pub fn install_signal_flag() -> &'static AtomicBool { + INSTALLED.get_or_init(|| { + let sa = SigAction::new( + SigHandler::Handler(on_signal), + SaFlags::empty(), + SigSet::empty(), + ); + // SAFETY: on_signal only performs a relaxed atomic store, which is + // async-signal-safe. + unsafe { + let _ = sigaction(Signal::SIGINT, &sa); + let _ = sigaction(Signal::SIGTERM, &sa); + } + }); + &SIGNAL_FLAG +} From 9adee07d7f47a10f3cd3a8b25f40d57b6cb1a46b Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 24 Jul 2026 11:00:42 -0400 Subject: [PATCH 6/7] docs(r10): example 41 fully gated across cpp/go/rust (S5 done) test-all-examples.py --only 41-capstone-fleet --mode vm prints PASS PASS PASS (verify.lua PASS 31/0 per language). Only chapter 41 (S6) remains. --- _plans/r10-capstone-fleet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index 7bc2732..b372561 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -166,5 +166,5 @@ the dispatcher, rather than one 1300-line drop. | S2 verify.lua (go gate) | done — `PASS 31 / FAIL 0` for `LSP_LANG=go` through the runner. Following ex23's convention the verify runs entirely on LOCAL loopback (fleet, cross-node bridge on two 127.0.0.1 ports, telemetry to the local LGTM) rather than orchestrating two guests over ssh — far more tractable, and the bridge logic is identical on loopback; the real two-host run is the chapter's demo | | S3 C++ port | done — `PASS 31 / FAIL 0` for `LSP_LANG=cpp` through the runner, first gate. Multi-file (pmon/chatterd/sysagent/fwatch/proto/caps/telemetry/util), reusing ex33/34/36/38/40; fixed a real telemetry-scaffold bug (`shared_ptr` vs `nostd::shared_ptr`) and the OTLP CMake linkage. Builds offline | | S4 Rust port | done — `PASS 31 / FAIL 0` for `LSP_LANG=rust` through the runner, first gate. Same file split as C++, reusing ex33/09/36/38/40. nix (inotify) + landlock 0.4 + opentelemetry-otlp 0.32, all offline. Documented divergence: opentelemetry-rust 0.32 surfaces export errors only via tracing events (needs a subscriber that would break the output contract), so the `export_errors` counter stays 0 — verify does not assert its value | -| S5 whole-fleet gate | pending | +| S5 whole-fleet gate | done — `test-all-examples.py --only 41-capstone-fleet --mode vm` prints `PASS PASS PASS` (3 passed, 0 failed; verify.lua PASS 31/0 per language). Example 41 fully ported and gated across cpp/go/rust | | S6 chapter 41 | pending | From 8cdd54ebb38aa3171ffd9c47f090223c521b2780 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 24 Jul 2026 17:59:10 -0400 Subject: [PATCH 7/7] =?UTF-8?q?docs(r10):=20author=20chapter=2041=20?= =?UTF-8?q?=E2=80=94=20the=20capstone=20fleet=20(S6,=20capstone=20complete?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Writes _docs/41-capstone-fleet.md, the book's finale (part "Performance and Low Latency"): pmon supervising chatterd + sysagent + fwatch as a capability-dropped, Landlock-sandboxed, OTLP-observed fleet across two hosts. Ties the six-program arc together — the supervisor (ch11-14), the chat daemon (ch21-27), the /proc agent (ch36), the file watcher (ch07-09/33), the two-VM lab (ch02), the sandbox layers (ch32-34), and OTLP to the LGTM stack (ch03/38). Three verbatim codetabs (capability-bounding-set drop, pmon's supervise-and-restart loop, chatterd's cross-host bridge worker) across cpp/go/rust. Adds Figures 41.1 (two-VM fleet topology) and 41.2 (pmon supervise lifecycle) + catalogue entries. Verified footer cites both the loopback gate (test-all-examples PASS PASS PASS; verify.lua PASS 31/0 per language) and a real two-VM run this session: cross-host chat received from=dave@target, both guests reaching health=up, Landlock ABI=7 on the guest vs 9 on the host, and sysagent_cpu_pct{node=peer}=1.75 reaching the host's Prometheus over the libvirt gateway. Marks S6 done; the capstone (example 41 + chapter 41) is complete. validate.py OK. --- _docs/41-capstone-fleet.md | 827 +++++++++ _plans/r10-capstone-fleet.md | 2 +- assets/diagrams/41-fleet-topology.excalidraw | 1582 +++++++++++++++++ assets/diagrams/41-fleet-topology.svg | 51 + .../41-supervise-lifecycle.excalidraw | 1026 +++++++++++ assets/diagrams/41-supervise-lifecycle.svg | 37 + assets/diagrams/README.md | 2 + 7 files changed, 3526 insertions(+), 1 deletion(-) create mode 100644 _docs/41-capstone-fleet.md create mode 100644 assets/diagrams/41-fleet-topology.excalidraw create mode 100644 assets/diagrams/41-fleet-topology.svg create mode 100644 assets/diagrams/41-supervise-lifecycle.excalidraw create mode 100644 assets/diagrams/41-supervise-lifecycle.svg diff --git a/_docs/41-capstone-fleet.md b/_docs/41-capstone-fleet.md new file mode 100644 index 0000000..dd21b78 --- /dev/null +++ b/_docs/41-capstone-fleet.md @@ -0,0 +1,827 @@ +--- +title: "The capstone fleet: pmon supervises chatterd, sysagent, and fwatch across two capability-dropped, Landlock-sandboxed, OTLP-observed hosts" +order: 41 +part: "Performance and Low Latency" +description: "41-capstone-fleet is one binary with four subcommands that assembles four of the book's six recurring programs into a single supervised system: pmon drops the capability bounding set and sets no_new_privs, then self-re-execs chatterd, sysagent, and fwatch as restartable children, forwarding SIGTERM/SIGINT and printing a pmon: health line every tick; chatterd (ch21-27) is bridged across two hosts so a message sent on one node is delivered on the other tagged from=NICK@node; sysagent (ch36) exports cpu%/mem%/load1 as OTel gauges over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; and fwatch (ch07-09, 33) watches its sandbox directory under an optional Landlock ruleset whose enforced ABI — 7 on the lab guest's 6.19.10 kernel, 9 on the Fedora 44 host — is itself a live cross-check of the kernel-version probing discipline Chapter 33 established. Verified on the systems-target/systems-peer two-VM lab, with OTLP reaching the host's Grafana LGTM stack over the libvirt gateway." +duration: "60 minutes" +--- + +Chapter 0's course map promised six programs that would grow chapter by +chapter into real systems software. By now four of them have grown enough to +stand on their own: `pmon` learned to fork, exec, wait, drop privilege, and +walk namespaces (Chapters 11–14, 18–19, 32, 34); `chatterd` learned TCP, then +epoll at scale, then UDP peer discovery, then async, then a pinned-core fast +path (21–27, 34, 38, 40); `sysagent` learned to read `/proc` and +`/sys/fs/cgroup` directly and then to export what it read (36–38); and +`fwatch` learned to watch a directory with polling, then inotify, then a +Landlock sandbox around itself (07–10, 33). This chapter does not teach a new +syscall. It assembles: one binary, four subcommands, where `pmon` is the +fleet's own init — it drops the capability bounding set for itself and every +descendant, then self-re-execs `chatterd`, `sysagent`, and `fwatch` as +children it restarts on unexpected exit, forwards its own signals to, and +reports on with a `pmon: health` line every tick. + +Two copies of that fleet run in this chapter — one on loopback, for +`verify.lua`, and one for real, on the two-VM lab Chapter 2 built. +`chatterd` is bridged between `systems-target` and `systems-peer` so a +message sent on one host is delivered on the other; each guest's `sysagent` +exports gauges over OTLP, through the libvirt gateway, to the LGTM stack +Chapter 3 stood up on the host; and `fwatch` runs sandboxed under Landlock +inside each fleet, its enforced ABI itself differing between the host kernel +and the guest kernel — a real, verified number, not a claim. Nearly every +part of this book from Setting Up through Performance and Low Latency shows +up somewhere in the run below. + +{% include excalidraw.html + file="41-fleet-topology" + alt="Two side-by-side VM boxes labeled systems-target (192.168.124.7) and systems-peer (192.168.124.95), each drawn as a large rounded rectangle representing one fleet. Inside each box, a top band labeled 'pmon: capability bounding set dropped, no_new_privs=1' encloses three child boxes side by side: chatterd, sysagent, and fwatch, each connected upward to a small pmon box with a supervise/restart arrow labeled 'fork+exec, restart on unexpected exit, forward SIGTERM'. The fwatch box additionally sits inside its own smaller nested box labeled 'Landlock ruleset: reads restricted to sandbox dir' with its enforced ABI number shown as a variable (7 on this guest's 6.19.10 kernel). A bidirectional arrow runs directly between the two chatterd boxes, labeled 'bridge: bridge@target / bridge@peer, DELIVER re-broadcast with from=NICK@node'. Below both VM boxes, a horizontal band represents the libvirt virbr0 bridge and its gateway address 192.168.124.1; two upward arrows from each guest's sysagent box pass through the gateway band labeled 'OTLP/HTTP :4318' and converge on a host-side box labeled 'LGTM stack (Grafana + Prometheus + Tempo), Chapter 3' sitting outside and above both VM boxes. A small caption near the gateway notes 'same route deploy-to-vm.sh forwards OTEL_EXPORTER_OTLP_ENDPOINT over.'" + caption="Figure 41.1 — the two-VM fleet: pmon supervises chatterd/sysagent/fwatch on each of systems-target and systems-peer inside a capability-bounding-set-drop boundary, fwatch additionally sandboxed under Landlock, chatterd bridged directly between the two guests, and both guests' sysagent exporting OTLP over the libvirt gateway to the host's LGTM stack" %} + +> **Tools used** — `ssh`/`scp` (host, to stage the built binary onto both +> `systems-target` and `systems-peer` and drive `pmon` on each — the same +> `scripts/lab/deploy-to-vm.sh` transport every VM example uses), `curl` +> (host, querying Grafana's Prometheus proxy for the `sysagent_cpu_pct` +> gauge). The libvirt bridge's gateway address (`192.168.124.1`) is the +> route guest-side OTLP traffic takes to reach the host's LGTM stack — a +> network path, not a tool. Capability-bounding-set drop and Landlock are +> syscalls the program makes itself (`prctl(2)`, `landlock_create_ruleset(2)` +> and friends); seccomp was already proven alongside Landlock in Chapter 33 +> and is not re-derived here. `ssh`/`scp`/`curl` ship in Fedora's base +> install and aren't separately checked by `scripts/check-host.sh`, the same +> way Chapter 36 notes for `curl` itself. + +## One binary, four programs: pmon as fleet init + +`app`'s four subcommands share one binary and one usage banner: + +``` +pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME] + [--chatterd-port P] [--health-interval-ms N] +chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME] +chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T] +chatterd listen --host H --port P --nick NICK [--timeout-ms T] +sysagent [--node NAME] [--interval-ms N] [--once] +sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M] +fwatch snapshot DIR +fwatch watch DIR [--sandbox] [--timeout-ms T] +``` + +`pmon` never execs a *different* binary for chatterd, sysagent, or fwatch — +it re-execs **itself**, with a different `argv[0]` command word, the same +self-re-exec technique Chapter 34's container entrypoint used. Concretely: +`mkdir` the sandbox directory, drop the capability bounding set once, resolve +the running binary's own path (`os.Executable()` in Go, +`readlink("/proc/self/exe")` in C++ and Rust), then fork+exec that same path +three times with three different argument lists — `chatterd serve …`, +`sysagent --node … --interval-ms 2000`, `fwatch watch DIR --sandbox`. + +{% include excalidraw.html + file="41-supervise-lifecycle" + alt="A state-machine diagram for one supervised child. A box labeled starting has an arrow to a box labeled up, annotated 'fork+exec succeeded, pid recorded, pmon: started service=NAME pid=PID printed'. From up, a dashed arrow labeled 'blocking wait (waitpid/cmd.Wait/nix::waitpid)' loops back into a decision diamond: 'shutting_down flag set?'. The no-branch leads to a box labeled restarting, annotated 'pmon: restart service=NAME attempt=N reason=... printed, restarts counter incremented, 300ms backoff sleep', with an arrow looping back up to starting. The yes-branch leads to a box labeled down. Off to the side, a separate small loop shows a clock icon labeled 'health tick every --health-interval-ms' with an arrow into a box printing 'pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0'. A third flow at the bottom shows SIGTERM/SIGINT arriving at a box labeled 'signal thread (sigwait / signal.Notify channel / SigSet::wait)', with an arrow to 'shutting_down.store(true)', then a fan-out arrow to 'kill(child_pid, SIGTERM) for every recorded pid', ending in a box labeled 'pmon: shutdown, exit 0' once every per-child supervisor has returned." + caption="Figure 41.2 — pmon's per-child supervise/restart lifecycle (starting → up → wait → restarting → up again on an unexpected exit), the health tick alongside it, and the SIGTERM → forward-to-children → shutdown path" %} + +Before any of that forking happens, `pmon` drops every capability it might +hold from its own bounding set and sets `no_new_privs`. `PR_CAPBSET_DROP` +removes a capability from the *bounding set*, not the current effective set — +the guarantee it gives is that no future `execve` of a setuid-root binary, +by this process or any descendant, can ever regain that capability, because +the bounding set only shrinks across `fork`/`exec`. That guarantee holds +even when the calling process is already unprivileged, which is exactly +`pmon`'s situation on an unprivileged lab-guest account: a loopback run +prints `pmon: capabilities bounding_set_dropped=0 no_new_privs=1`, and `0` is +the *correct* number here — there was nothing in the bounding set to drop, +and the line still proves the `prctl` calls ran and `no_new_privs` is set. + +{% include codetabs.html langs="C++|Go|Rust" %} + +```go +func dropCapabilityBoundingSet() { + dropped := 0 + for c := 0; c <= lastKnownCap; c++ { + if err := unix.Prctl(unix.PR_CAPBSET_DROP, uintptr(c), 0, 0, 0); err == nil { + dropped++ + } + } + if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil { + fmt.Fprintf(os.Stderr, "pmon: prctl(PR_SET_NO_NEW_PRIVS): %v\n", err) + return + } + fmt.Fprintf(os.Stderr, "pmon: capabilities bounding_set_dropped=%d no_new_privs=1\n", dropped) +} +``` + +```cpp +void drop_bounding_set() { + int dropped = 0; + for (int c = 0; c <= kLastKnownCap; ++c) { + if (::prctl(PR_CAPBSET_DROP, c, 0, 0, 0) == 0) { + ++dropped; + } + } + if (::prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0) { + std::fprintf(stderr, "pmon: prctl(PR_SET_NO_NEW_PRIVS) failed\n"); + return; + } + std::fprintf(stderr, "pmon: capabilities bounding_set_dropped=%d no_new_privs=1\n", dropped); +} +``` + +```rust +pub fn drop_bounding_set() { + let mut dropped = 0; + for c in 0..=LAST_KNOWN_CAP { + // SAFETY: prctl(PR_CAPBSET_DROP, c, 0, 0, 0) has no memory-safety + // implications; only its return value is inspected. + let rc = unsafe { libc::prctl(libc::PR_CAPBSET_DROP, c, 0, 0, 0) }; + if rc == 0 { + dropped += 1; + } + } + // SAFETY: same as above. + let rc = unsafe { libc::prctl(libc::PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) }; + if rc != 0 { + eprintln!( + "pmon: prctl(PR_SET_NO_NEW_PRIVS): {}", + std::io::Error::last_os_error() + ); + return; + } + eprintln!("pmon: capabilities bounding_set_dropped={dropped} no_new_privs=1"); +} +``` + +All three loop `prctl(PR_CAPBSET_DROP, c, …)` from `0` through +`CAP_CHECKPOINT_RESTORE` (40, current as of a 6.x kernel) and ignore `EINVAL` +from probing past whatever the running kernel's actual last capability is — +the loop is future-proof by construction rather than by a version check. +Once the bounding set is shrunk, every child `pmon` forks inherits the +narrower set automatically; nothing in `chatterd`, `sysagent`, or `fwatch` +has to know this happened. + +With the bounding set dropped, `pmon` spawns and supervises the three +children — one goroutine per service in Go, one `std::thread` per service in +C++, one `std::thread` per service in Rust, each running the same +restart-on-unexpected-exit loop Figure 41.2 diagrams: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```go + go func() { + defer wg.Done() + for { + cmd := exec.Command(selfExe, spec.args...) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + cmd.Env = os.Environ() + if err := cmd.Start(); err != nil { + fmt.Fprintf(os.Stderr, "pmon: start service=%s: %v\n", spec.name, err) + time.Sleep(500 * time.Millisecond) + continue + } + cs.pid.Store(int64(cmd.Process.Pid)) + cs.state.Store("up") + fmt.Printf("pmon: started service=%s pid=%d\n", spec.name, cmd.Process.Pid) + + waitErr := cmd.Wait() + if shuttingDown.Load() { + cs.state.Store("down") + return + } + n := cs.restarts.Add(1) + cs.state.Store("restarting") + fmt.Printf("pmon: restart service=%s attempt=%d reason=%v\n", spec.name, n, waitErr) + time.Sleep(300 * time.Millisecond) + } + }() +``` + +```cpp +void run_child_supervisor(const std::string& self_exe, const ChildSpec& spec, ChildState& cs, + std::atomic& shutting_down) { + for (;;) { + const pid_t pid = ::fork(); + if (pid < 0) { + std::println(stderr, "pmon: start service={}: {}", spec.name, std::strerror(errno)); + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + continue; + } + if (pid == 0) { + // Unblock what pmon::run blocked in the parent (see pmon::run): + // a forwarded SIGTERM must reach this child's own handler, not + // sit blocked forever. + sigset_t empty; + sigemptyset(&empty); + ::pthread_sigmask(SIG_SETMASK, &empty, nullptr); + + std::vector argv; + argv.reserve(spec.args.size() + 2); + argv.push_back(const_cast(self_exe.c_str())); + for (const auto& a : spec.args) { + argv.push_back(const_cast(a.c_str())); + } + argv.push_back(nullptr); + ::execv(self_exe.c_str(), argv.data()); + ::_exit(127); // execv failed + } + + cs.pid.store(pid, std::memory_order_relaxed); + cs.state.store(State::Up, std::memory_order_relaxed); + std::println("pmon: started service={} pid={}", spec.name, pid); + std::fflush(stdout); + + int status = 0; + ::waitpid(pid, &status, 0); + + if (shutting_down.load(std::memory_order_relaxed)) { + cs.state.store(State::Down, std::memory_order_relaxed); + return; + } + + const long n = cs.restarts.fetch_add(1, std::memory_order_relaxed) + 1; + cs.state.store(State::Restarting, std::memory_order_relaxed); + std::string reason; + if (WIFEXITED(status)) { + reason = "exit status " + std::to_string(WEXITSTATUS(status)); + } else if (WIFSIGNALED(status)) { + reason = std::string("signal: ") + ::strsignal(WTERMSIG(status)); + } else { + reason = "unknown"; + } + std::println("pmon: restart service={} attempt={} reason={}", spec.name, n, reason); + std::fflush(stdout); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + } +} +``` + +```rust +fn run_child_supervisor( + self_exe: &CString, + args: &[String], + cs: &ChildState, + name: &str, + shutting_down: &AtomicBool, +) { + // Every CString is built once, before the first fork, so the only work + // the child performs between fork() and execv() is clearing its signal + // mask and calling execv() itself. + let mut argv: Vec = Vec::with_capacity(args.len() + 1); + argv.push(self_exe.clone()); + for a in args { + argv.push(CString::new(a.as_str()).expect("child arg contains NUL")); + } + + loop { + // SAFETY: this process is multi-threaded (one supervisor thread per + // service plus the sigwait thread below); the child branch performs + // only the signal-mask reset and execv() before either replacing + // itself or exiting, matching pmon.cpp's fork-then-immediately-exec + // discipline. + match unsafe { fork() } { + Ok(ForkResult::Child) => { + // Unblock what pmon::run blocked in the parent (see below): + // a forwarded SIGTERM must reach this child's own handling, + // not sit blocked forever. + let empty = SigSet::empty(); + let _ = empty.thread_set_mask(); + match execv(self_exe, &argv) { + Ok(_) => unreachable!(), + Err(_) => unsafe { libc::_exit(127) }, // execv failed + } + } + Ok(ForkResult::Parent { child }) => { + cs.pid.store(child.as_raw(), Ordering::Relaxed); + cs.set_state(State::Up); + println!("pmon: started service={name} pid={}", child.as_raw()); + + let status = waitpid(child, None); + if shutting_down.load(Ordering::Relaxed) { + cs.set_state(State::Down); + return; + } + let n = cs.restarts.fetch_add(1, Ordering::Relaxed) + 1; + cs.set_state(State::Restarting); + let reason = describe_wait_status(status); + println!("pmon: restart service={name} attempt={n} reason={reason}"); + thread::sleep(Duration::from_millis(300)); + } + Err(e) => { + eprintln!("pmon: start service={name}: {e}"); + thread::sleep(Duration::from_millis(500)); + } + } + } +} +``` + +Go delegates the fork+exec mechanics to `os/exec.Command`, which is why its +version is shorter — `cmd.Start()`/`cmd.Wait()` hide the raw syscalls C++ and +Rust make directly (`fork()`, then `execv()` in the child, then `waitpid()` +in the parent). But the *shape* is identical across all three: start, record +the pid, print `pmon: started`, block until the child exits, check whether +this exit was requested (`shutting_down`) or not, and if not, count a +restart, print `pmon: restart … reason=…`, back off 300 ms, and loop. A +child that fails to even start (`cmd.Start()` erroring, `fork()` returning +`< 0`) is not fatal to `pmon` — it logs, waits 500 ms, and retries, because a +supervisor's job is to keep trying, not to give up on the first bad attempt. + +## chatterd, bridged across two hosts + +`chatterd` (Chapters 21–27) speaks the same wire frame it always has — +magic `"CH"`, version 1, a type byte, a big-endian `u16` length, then payload +(`proto.go`/`.hpp`/`.rs`, unchanged) — and this chapter adds exactly one new +idea on top: with `--peer HOST:PORT` set on both sides, a message sent on +one node is delivered to a listener on the other. The mechanism needs no +protocol change at all. A **bridge worker** is just an ordinary client of +the *other* node's server, joined under the nick `bridge@`: + +``` +target dials peer, joins as "bridge@target" (lives in peer's client registry) +peer dials target, joins as "bridge@peer" (lives in target's client registry) +``` + +A local `MSG` is broadcast as `DELIVER` to every registered connection — +including a bridge connection dialed in from the other side — exactly like +any other client. When a bridge *worker* itself receives a `DELIVER` over the +connection it dialed out, it re-broadcasts locally with +`includeBridges=false`, so the message never goes back out over any bridge: +no ping-pong between two nodes forwarding the same line back and forth +forever. The one piece of new logic lives in the bridge worker's receive +loop — tag the nick with the *sending* node's name, if it isn't tagged +already, before re-broadcasting locally: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```go +func bridgeWorker(peerAddr, localNode, peerNode string, h *hub, sig interface{ Load() bool }) { + backoff := 300 * time.Millisecond + for !sig.Load() { + conn, err := net.DialTimeout("tcp", peerAddr, 3*time.Second) + if err != nil { + time.Sleep(backoff) + continue + } + bridgeNick := "bridge@" + localNode + if err := writeFrame(conn, frame{typ: typeJoin, payload: []byte(bridgeNick)}); err != nil { + conn.Close() + time.Sleep(backoff) + continue + } + fmt.Fprintf(os.Stderr, "chatterd: bridge connected peer=%s as=%s\n", peerAddr, bridgeNick) + + for { + f, err := readFrame(conn) + if err != nil { + break + } + if f.typ != typeDeliver { + continue + } + nick, text, ok := splitDeliver(f.payload) + if !ok { + continue + } + if !strings.Contains(nick, "@") { + nick = nick + "@" + peerNode + } + h.broadcastDeliver(nil, false, nick, text) + } + conn.Close() + fmt.Fprintf(os.Stderr, "chatterd: bridge disconnected peer=%s (retrying)\n", peerAddr) + time.Sleep(backoff) + } +} +``` + +```cpp +void bridge_worker(std::string peer_addr, std::string local_node, std::string peer_node, Hub& hub, + std::atomic& sig) { + const auto colon = peer_addr.rfind(':'); + if (colon == std::string::npos) { + return; + } + const std::string host = peer_addr.substr(0, colon); + const int port = std::atoi(peer_addr.substr(colon + 1).c_str()); + constexpr auto kBackoff = std::chrono::milliseconds(300); + + while (!sig.load(std::memory_order_relaxed)) { + auto conn = connect_tcp(host, port, 3000); + if (!conn) { + std::this_thread::sleep_for(kBackoff); + continue; + } + const std::string bridge_nick = "bridge@" + local_node; + proto::Frame joinf{proto::kJoin, {bridge_nick.begin(), bridge_nick.end()}}; + if (auto r = proto::write_frame(conn->get(), joinf); !r) { + std::this_thread::sleep_for(kBackoff); + continue; + } + std::println(stderr, "chatterd: bridge connected peer={} as={}", peer_addr, bridge_nick); + std::fflush(stderr); + + for (;;) { + auto f = proto::read_frame(conn->get()); + if (!f) { + break; + } + if (f->type != proto::kDeliver) { + continue; + } + std::string nick; + std::string text; + if (!proto::split_deliver(f->payload, nick, text)) { + continue; + } + if (nick.find('@') == std::string::npos) { + nick += "@" + peer_node; + } + hub.broadcast_deliver(nullptr, false, nick, text); + } + std::println(stderr, "chatterd: bridge disconnected peer={} (retrying)", peer_addr); + std::fflush(stderr); + std::this_thread::sleep_for(kBackoff); + } +} +``` + +```rust +fn bridge_worker( + peer_addr: String, + local_node: String, + peer_node: String, + hub: Arc, + sig: &'static AtomicBool, +) { + let Some((host, port_str)) = peer_addr.rsplit_once(':') else { + return; + }; + let Ok(port) = port_str.parse::() else { + return; + }; + let backoff = Duration::from_millis(300); + + while !sig.load(Ordering::Relaxed) { + let conn = match connect_tcp(host, port, 3000) { + Ok(c) => c, + Err(_) => { + thread::sleep(backoff); + continue; + } + }; + let raw = conn.as_raw_fd(); + let bridge_nick = format!("bridge@{local_node}"); + let joinf = Frame { + typ: proto::TYPE_JOIN, + payload: bridge_nick.clone().into_bytes(), + }; + if proto::write_frame(raw, &joinf).is_err() { + thread::sleep(backoff); + continue; + } + eprintln!("chatterd: bridge connected peer={peer_addr} as={bridge_nick}"); + + while let Ok(f) = proto::read_frame(raw) { + if f.typ != proto::TYPE_DELIVER { + continue; + } + let Some((mut nick, text)) = proto::split_deliver(&f.payload) else { + continue; + }; + if !nick.contains('@') { + nick.push('@'); + nick.push_str(&peer_node); + } + hub.broadcast_deliver(None, false, &nick, &text); + } + eprintln!("chatterd: bridge disconnected peer={peer_addr} (retrying)"); + thread::sleep(backoff); + // `conn` drops here, closing the fd, before the loop reconnects. + } +} +``` + +`peer_node` here is the *name of the node this worker dialed into* — so when +`dave` sends a message locally on `target`, `target`'s hub broadcasts +`DELIVER` to every local client including the `bridge@peer` connection that +`peer`'s own bridge worker dialed in with; that worker reads the `DELIVER` +back over the connection it opened, sees `dave` has no `@` in it yet, tags it +`dave@target`, and re-broadcasts locally on `peer` with +`includeBridges=false`. A listener on `peer` sees +`chatterd: received from=dave@target text=…` — the `@target` suffix is the +proof the message actually crossed the wire, not that it was echoed +somewhere local. The reconnect loop (dial, backoff, retry) means either side +can start first, or the other side can restart under `pmon`, without the +bridge ever needing a fixed startup order. + +## sysagent and fwatch: what's observed, what's sandboxed + +`sysagent` under the fleet is the same `/proc`-reading collector Chapter 36 +built, trimmed to the three signals a fleet operator watches continuously: +`cpu_pct` (a `/proc/stat` tick delta), `mem_pct` (`/proc/meminfo`), and +`load1` (`/proc/loadavg`), printed every `--interval-ms` as +`sysagent: node=… cpu_pct=… mem_pct=… load1=… ts=…`. New in this chapter: +when `OTEL_EXPORTER_OTLP_ENDPOINT` is set, those same three numbers are also +registered as OTel **observable gauges** (`sysagent.cpu.pct`, +`sysagent.mem.pct`, `sysagent.load1`) with a callback that reads the latest +sample under a mutex and reports it tagged with a `node` attribute, exported +over OTLP/HTTP on a periodic reader. When the endpoint is unset, `sysagent` +prints `sysagent: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)` and +runs exactly the same otherwise — telemetry here is additive, never a +precondition for the service coming up. `sysagent saturate --resource +cpu|mem --seconds N` is the chaos half: it spins worker threads doing +nothing but arithmetic, or holds a touched, resident memory allocation, so a +concurrently-running `sysagent`'s own `cpu_pct`/`mem_pct` visibly moves — the +USE-method callback this fleet can point at itself. + +`fwatch` is Chapter 07–10's file watcher — `inotify_init1`, one +non-recursive watch on a directory, `create`/`modify`/`delete` lines — with +Chapter 33's Landlock ruleset available as `--sandbox`. The ruleset +restricts filesystem *reads* to the watched directory: `pmon` passes fwatch +`watch SANDBOX_DIR --sandbox`, and before the inotify loop starts, `fwatch` +probes the running kernel's Landlock ABI with +`landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, builds a +ruleset scoped to that one directory, and calls `landlock_restrict_self` +after setting `no_new_privs` — the identical sequence Chapter 33 proved, +reused rather than re-derived. `fwatch: landlock ABI=N enforced dir=…` is +printed only once the kernel has actually accepted the restriction, and — as +the cross-check below shows — `N` is not a constant baked into the program; +it is whatever the kernel underneath happens to support. Capability-bounding- +set drop (applied once, by `pmon`, before any child exists) and Landlock +(applied per-process, by `fwatch` itself) are this capstone's two sandbox +layers; seccomp, Chapter 33's third layer, was proven there and is not +reapplied here. + +## How the code works + +`main` in all three languages dispatches on `argv[1]` — `pmon`, `chatterd`, +`sysagent`, or `fwatch` — through the same hand-rolled `argFlag`/`hasFlag`/ +`firstPositional` scan the rest of the book's argument parsing uses, funnel- +ing every malformed invocation through one `usage()` call. `chatterd serve` +and `sysagent` (without `saturate`) each call `initTelemetry`, which is +additive by design: it reads `OTEL_EXPORTER_OTLP_ENDPOINT`, and if it's +unset, returns a no-op `telemetry` struct backed by `otel.Tracer`/ +`otel.Meter` no-op implementations rather than failing the subcommand — a +service the fleet depends on for chatting and sampling `/proc` should never +refuse to start because an observability backend is unreachable or +unconfigured. When the endpoint *is* set, `chatterd` wraps every delivered +message in a `chatterd.deliver` span tagged with the sender's nick, text +length, node, and whether it arrived over a bridge, and prints +`chatterd: trace_id=… node=… from=…` for each one; `sysagent` registers its +three gauges and lets a periodic OTLP/HTTP exporter push them on its own +schedule. Self-re-exec is the one place the three languages genuinely +diverge in mechanism: Go's `os.Executable()` plus `filepath.Abs` resolves +the running binary through `/proc/self/exe` internally, while C++ and Rust +call `readlink("/proc/self/exe", …)` directly — same symlink, three ways to +read it, all landing on the same absolute path that every forked child then +`execv`s. + +## Errors, three ways + +`app` has two usage-error surfaces and one runtime-error surface. The **top- +level usage error** — no command, an unrecognized subcommand, `chatterd`/ +`fwatch` with no sub-subcommand, `chatterd send` missing `--nick`/`--text` — +prints the shared `usage: app ` banner and exits **2**. +`sysagent saturate` has its **own**, narrower usage error: a missing +`--resource` prints `usage: sysagent saturate --resource cpu|mem --seconds N +[--workers K|--mb M]` — a second banner, not the top-level one, because +`saturate` is itself a small program with its own contract. **Runtime +errors** are exit **1** with a `service: error message` line: `chatterd: +listen H:P: …` on a bind failure, `fwatch: Landlock not supported by this +kernel` when the probe returns ABI 0, `pmon: mkdir DIR: …` if the sandbox +directory can't be created. What `pmon` does *not* treat as a runtime error +is a single failed fork or exec attempt for one of its children — that's +logged (`pmon: start service=…: …`) and retried after a backoff, never +surfaced as a fatal error, because a process supervisor exists precisely to +absorb transient start failures rather than propagate them. A clean `pmon` +shutdown, a successful one-shot `sysagent --once`, or a `fwatch watch` that +runs to its `--timeout-ms` deadline or a delivered signal all exit **0**. + +## Concurrency lens + +Every piece of this book's other running thread — which concurrency model +each language reaches for — shows up in `pmon`'s two jobs: consuming exactly +one `SIGTERM`/`SIGINT` no matter how many children are running, and +supervising each child without one child's restart blocking another's. + +Go's version is the one most readers will recognize: `signal.Notify` hands a +buffered channel to the runtime's own signal machinery, and a dedicated +goroutine blocks on `<-sigCh`, sets `shuttingDown`, and kills every recorded +child pid. Each of the three services gets its own goroutine running the +restart loop shown above, all three joined with a `sync.WaitGroup`; the main +goroutine waits on a `select` between the `WaitGroup`'s completion (via a +`done` channel) and a health-interval `time.Ticker`, printing +`pmon: health …` each time the ticker fires and returning once every child +has actually stopped. + +C++ and Rust reach for the same alternative to a signal *handler*: a +dedicated **`sigwait`-consuming thread**. Both block `SIGTERM` and `SIGINT` +in the main thread with `pthread_sigmask`/`SigSet::thread_block()` *before* +spawning any other thread — a blocked mask is inherited by every thread +created afterward, so only the one thread that later calls +`sigwait`/`SigSet::wait()` (nix's wrapper over the same syscall) ever +consumes the signal; no async-signal-unsafe work happens in a handler at +all, because there is no handler. Each of the three services then gets its +own `std::thread`, and — unlike Go's `exec.Command`-based children — that +thread performs the `fork()`/`execv()` **itself**, directly: `pmon`'s C++ +and Rust ports are "fork+threads" in the sense the task names them, one OS +thread owning one child's entire fork/wait/restart lifecycle. Every forked +child clears its own inherited copy of the blocked signal mask +(`pthread_sigmask(SIG_SETMASK, &empty, …)` in C++, `SigSet::empty(). +thread_set_mask()` in Rust) immediately after `fork()` and before `execv()` +— without this, the child would inherit a *blocked* `SIGTERM`/`SIGINT` and +its own shutdown-signal handling (the same `install_signal_flag` +mechanism `chatterd`/`sysagent`/`fwatch` use standalone) could never fire, +the exact bug Chapter 34's container entrypoint first ran into and +documented. Where C++ and Rust part ways is the health-tick wait: C++'s main +thread holds a `std::condition_variable` and calls `wait_for(lock, interval, +predicate)`, waking early the moment every worker thread's `active` counter +reaches zero and otherwise printing health once per timeout; Rust's main +thread has no condvar at all — it just `thread::sleep`s the health interval +and polls `JoinHandle::is_finished()` on every worker, a coarser but simpler +wake pattern that trades a small amount of shutdown latency (up to one +health interval) for not needing a shared mutex/condvar pair at all. + +## Build, run, observe + +```bash +[host]$ cd examples/41-capstone-fleet && ./demo.sh build +``` + +`verify.lua` follows the convention Chapter 23 established for `vm-peer` +examples: the assertion itself runs entirely on loopback — the fleet, the +cross-node bridge across two `127.0.0.1` ports, and telemetry to a local +LGTM stack — while the real two-host run is the chapter's demo. A loopback +fleet, driven by hand the same way, shows every piece in one place: + +```console +[host]$ ./go/bin/app pmon --node p1 --sandbox-dir /tmp/fwatch-sandbox +pmon: capabilities bounding_set_dropped=0 no_new_privs=1 +pmon: started service=chatterd pid= +chatterd: listening on 0.0.0.0:47100 node=p1 +pmon: started service=sysagent pid= +sysagent: node=p1 cpu_pct= mem_pct= load1= ts= +pmon: started service=fwatch pid= +fwatch: landlock ABI=9 enforced dir=/tmp/fwatch-sandbox +pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0 +^C +pmon: shutdown +``` + +(`ABI=9` here is the reference host's own Landlock version — see the +cross-check below for the guest's different number.) A local listener +receiving a message sent by another client on the same node, and the +cross-node bridge with two nodes on loopback, are exactly the shapes +`verify.lua` asserts and that produced this session's evidence: + +```console +chatterd: received from=alice text=hello fleet +chatterd: received from=dave@A text=cross node +``` + +```console +[host]$ python3 scripts/test-all-examples.py --only 41-capstone-fleet --mode vm +verifying... + verify 41-capstone-fleet [cpp]: PASS + verify 41-capstone-fleet [go]: PASS + verify 41-capstone-fleet [rust]: PASS + +example cpp go rust +41-capstone-fleet PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +Each language's `verify.lua` run reported `PASS 31 / FAIL 0`. + +The real payoff is the two-host run this chapter has been building toward: +`pmon` staged and started on both `systems-target` (192.168.124.7) and +`systems-peer` (192.168.124.95), bridged to each other, both exporting OTLP +through the libvirt gateway (`192.168.124.1`) to the host's LGTM stack: + +```console +[host]$ scp app fedora@192.168.124.7:~/app +[host]$ scp app fedora@192.168.124.95:~/app +[vm]$ OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1:4318 \ + ./app pmon --node target --peer 192.168.124.95:47100 --peer-node peer +[peer]$ OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1:4318 \ + ./app pmon --node peer --peer 192.168.124.7:47100 --peer-node target +``` + +Both guests reached full health: + +```console +[vm]$ pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0 +[peer]$ pmon: health chatterd=up sysagent=up fwatch=up restarts=chatterd:0,sysagent:0,fwatch:0 +``` + +And the cross-host bridge delivered a message sent on `target` to a listener +on `peer`: + +```console +[vm]$ ./app chatterd send --host 127.0.0.1 --port 47100 --nick dave --text "hello across hosts" +[peer]$ ./app chatterd listen --host 127.0.0.1 --port 47100 --nick carol +chatterd: received from=dave@target text=hello across hosts +``` + +## Cross-check: an ABI the kernel decides, and a gauge that crosses the gateway + +Two independent proofs back this chapter's two biggest claims, and neither +comes from a line the program prints about its own intentions. + +The first is Landlock's enforced ABI. `fwatch --sandbox` never hardcodes a +version — it asks the running kernel with +`landlock_create_ruleset(NULL, 0, LANDLOCK_CREATE_RULESET_VERSION)`, the +exact probe Chapter 33 established, and prints whatever comes back. On the +Fedora 44 reference host (kernel `7.1.3-200.fc44`), where this chapter's +loopback `verify.lua` runs, that probe returns **ABI 9**. On the +`systems-target` lab guest (kernel `6.19.10-300.fc44`) this session, the +identical code, the identical syscall, returned **ABI 7**. That is not a +bug and not a discrepancy to reconcile — it is the accurate, verified +consequence of two different kernel versions supporting two different sets +of Landlock features, and it is the same lesson Chapter 33 taught about +attach targets and struct offsets, reached through a sandbox ABI instead: a +program that hardcodes "ABI 9" would be *wrong* on the guest, and `fwatch`'s +own probe is what keeps it right on both. + +The second is the OTLP path itself. `sysagent: otel enabled endpoint=…` +proves the exporter was configured; it does not prove a single byte reached +anywhere. The real proof is on the other end of the gateway: after the +two-host bridge run above, `sysagent_cpu_pct{node="peer"}` — queried through +Grafana's Prometheus proxy — was present with value **1.75**, a gauge that +started life inside an unprivileged process on `systems-peer`, was +exported over OTLP/HTTP to `http://192.168.124.1:4318`, crossed the libvirt +bridge's gateway, and landed in the host's own Prometheus, labeled by the +node that produced it. That is the guest→host telemetry path Chapter 3's +LGTM stack exists to receive, closed end to end. + +## What you learned + +- **`pmon` is the fleet's own init**: it drops the capability bounding set + and sets `no_new_privs` once, then self-re-execs its own binary three + times with three different `argv[0]` commands, restarting whichever child + exits unexpectedly and forwarding its own `SIGTERM`/`SIGINT` to every + recorded child pid on shutdown. +- **`PR_CAPBSET_DROP` is meaningful even on an already-unprivileged + process.** `bounding_set_dropped=0` on the lab guest's unprivileged + account is the *correct* number — the guarantee (no future `execve` of a + setuid binary can regain a dropped capability) holds regardless of what + was there to begin with. +- **A bridge is just a client with a special nick.** `chatterd`'s cross-host + bridging needed no protocol change: `bridge@` is an ordinary + connection in the peer's client registry, `includeBridges=false` on + re-broadcast is what stops a two-node ping-pong, and the `@node` suffix on + a delivered nick is the observable proof a message actually crossed. +- **A sandbox's enforced feature set is a kernel fact, not a program's + assumption.** The identical Landlock probe returned ABI 9 on the Fedora + 44 host and ABI 7 on the lab guest's 6.19.10 kernel this session — the + same class of version drift Chapter 33 taught, reached through a + capability-negotiation syscall instead of an attach point. +- **Telemetry is additive, and the guest→host OTLP path is real, not + assumed.** Every service ran identically with `OTEL_EXPORTER_OTLP_ENDPOINT` + unset; with it set, `sysagent_cpu_pct{node="peer"}` — 1.75 on this + session's run — reached the host's Prometheus over the libvirt gateway, + the same route `deploy-to-vm.sh` forwards for every OTLP-emitting VM demo. +- **Identical supervision, three concurrency shapes.** Go leans on + `os/exec.Command`, goroutines, a `sync.WaitGroup`, and a `signal.Notify` + channel; C++ and Rust both fork and `execv` directly inside a dedicated + `std::thread` per child and consume signals on a separate `sigwait`/ + `SigSet::wait` thread rather than a handler — and even between those two, + C++'s health tick wakes early off a condition variable while Rust's just + polls `is_finished()` on a sleep — the same restart/health/shutdown + contract, four genuinely different mechanisms underneath. + +This closes the six-program arc that began in Chapter 7: four of the six — +`pmon`, `chatterd`, `sysagent`, `fwatch` — now run together, supervised, +sandboxed, and observed, across two real hosts. The book continues into +three deep dives — embedding Lua, Rust macros for systems code, the Go +runtime — but every syscall, sandbox layer, and signal-handling discipline +those chapters lean on was proven here, running for real. + +--- + +

verified — on the Fedora 44 +reference host (kernel 7.1.3-200.fc44) and the two lab guests +systems-target (192.168.124.7) / systems-peer +(192.168.124.95, kernel 6.19.10-300.fc44) this session: +python3 scripts/test-all-examples.py --only 41-capstone-fleet --mode +vm printed PASS PASS PASS (3 passed, 0 failed; +verify.lua reported PASS 31 / FAIL 0 for each of +cpp/go/rust, exercised entirely on loopback per the Chapter 23 vm-peer +convention). On the real two-host run, pmon on both +systems-target and systems-peer reached +pmon: health chatterd=up sysagent=up fwatch=up; a message sent +on target (nick dave, text "hello across hosts") +was delivered to a listener on peer as chatterd: received +from=dave@target text=hello across hosts, proving the cross-host +bridge; fwatch --sandbox on the guest printed fwatch: +landlock ABI=7 enforced, against ABI=9 on the host's +loopback run — the same probe, two different kernels, a real and verified +difference; and with OTEL_EXPORTER_OTLP_ENDPOINT=http://192.168.124.1: +4318 set on both guests, sysagent_cpu_pct{node="peer"} +reached the host's Prometheus (queried through Grafana's proxy) with value +1.75, confirming the guest→host OTLP path over the libvirt +gateway. Not exercised: seccomp filtering of any fleet child (Chapter 33 +proved seccomp alongside Landlock; this capstone reuses only the +capability-bounding-set-drop and Landlock layers, not seccomp) and a run +where the LGTM stack was unreachable (that path reports SKIP for the +telemetry assertions, not a failing example, per the manifest's +requires: [lgtm]).

diff --git a/_plans/r10-capstone-fleet.md b/_plans/r10-capstone-fleet.md index b372561..eec02ec 100644 --- a/_plans/r10-capstone-fleet.md +++ b/_plans/r10-capstone-fleet.md @@ -167,4 +167,4 @@ the dispatcher, rather than one 1300-line drop. | S3 C++ port | done — `PASS 31 / FAIL 0` for `LSP_LANG=cpp` through the runner, first gate. Multi-file (pmon/chatterd/sysagent/fwatch/proto/caps/telemetry/util), reusing ex33/34/36/38/40; fixed a real telemetry-scaffold bug (`shared_ptr` vs `nostd::shared_ptr`) and the OTLP CMake linkage. Builds offline | | S4 Rust port | done — `PASS 31 / FAIL 0` for `LSP_LANG=rust` through the runner, first gate. Same file split as C++, reusing ex33/09/36/38/40. nix (inotify) + landlock 0.4 + opentelemetry-otlp 0.32, all offline. Documented divergence: opentelemetry-rust 0.32 surfaces export errors only via tracing events (needs a subscriber that would break the output contract), so the `export_errors` counter stays 0 — verify does not assert its value | | S5 whole-fleet gate | done — `test-all-examples.py --only 41-capstone-fleet --mode vm` prints `PASS PASS PASS` (3 passed, 0 failed; verify.lua PASS 31/0 per language). Example 41 fully ported and gated across cpp/go/rust | -| S6 chapter 41 | pending | +| S6 chapter 41 | done — `_docs/41-capstone-fleet.md` (827 lines) to the spine, part "Performance and Low Latency"; 3 verbatim codetabs (cap-drop, pmon supervise loop, chatterd bridge worker), Figures 41.1 (topology) + 41.2 (supervise lifecycle). Verified footer cites the loopback gate (PASS 31/0 ×3) AND the real two-VM run (cross-host `from=dave@target`, dual-guest health, Landlock ABI 7-guest/9-host, `sysagent_cpu_pct{node=peer}`=1.75 guest→host). validate.py OK. **Capstone complete.** | diff --git a/assets/diagrams/41-fleet-topology.excalidraw b/assets/diagrams/41-fleet-topology.excalidraw new file mode 100644 index 0000000..48c97d9 --- /dev/null +++ b/assets/diagrams/41-fleet-topology.excalidraw @@ -0,0 +1,1582 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r899895451", + "type": "rectangle", + "x": 24, + "y": 48, + "width": 452, + "height": 300, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 834608374, + "versionNonce": 1469783630, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1902705596", + "type": "text", + "x": 36, + "y": 54, + "width": 184.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1335880292, + "versionNonce": 1874795626, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-target \u00b7 node=target", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-target \u00b7 node=target", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r237367847", + "type": "rectangle", + "x": 524, + "y": 48, + "width": 452, + "height": 300, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1551918244, + "versionNonce": 1752990218, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1017458792", + "type": "text", + "x": 536, + "y": 54, + "width": 158.4, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1236009909, + "versionNonce": 1097358908, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-peer \u00b7 node=peer", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-peer \u00b7 node=peer", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r975767240", + "type": "rectangle", + "x": 44, + "y": 78, + "width": 412, + "height": 52, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 430217618, + "versionNonce": 319613954, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t298468713", + "type": "text", + "x": 54, + "y": 86, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1932518549, + "versionNonce": 1861179545, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmon \u2014 supervisor", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmon \u2014 supervisor", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1288964876", + "type": "text", + "x": 54, + "y": 106, + "width": 272.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 124059615, + "versionNonce": 1572890115, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "drops caps \u00b7 no_new_privs \u00b7 restarts \u00b7 health", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "drops caps \u00b7 no_new_privs \u00b7 restarts \u00b7 health", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r739935985", + "type": "rectangle", + "x": 44, + "y": 162, + "width": 126, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1881593090, + "versionNonce": 1897971527, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t914362493", + "type": "text", + "x": 54, + "y": 170, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 928842946, + "versionNonce": 1688221852, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fwatch", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fwatch", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t858599877", + "type": "text", + "x": 54, + "y": 190, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1504130085, + "versionNonce": 659977890, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inotify + Landlock", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inotify + Landlock", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1498123781", + "type": "rectangle", + "x": 180, + "y": 162, + "width": 126, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 151218572, + "versionNonce": 1670355337, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t373838131", + "type": "text", + "x": 190, + "y": 170, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 985340740, + "versionNonce": 356990577, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1136854719", + "type": "text", + "x": 190, + "y": 190, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 478154106, + "versionNonce": 1460409482, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc \u2192 OTLP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc \u2192 OTLP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r22156890", + "type": "rectangle", + "x": 316, + "y": 162, + "width": 140, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1612767158, + "versionNonce": 1977064104, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1532800324", + "type": "text", + "x": 326, + "y": 170, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1518944066, + "versionNonce": 746498872, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chatterd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chatterd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t964947630", + "type": "text", + "x": 326, + "y": 190, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 844032747, + "versionNonce": 1009504354, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chat + bridge", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chat + bridge", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1192342793", + "type": "rectangle", + "x": 544, + "y": 78, + "width": 412, + "height": 52, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1158698755, + "versionNonce": 375413712, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t618984021", + "type": "text", + "x": 554, + "y": 86, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1115714099, + "versionNonce": 13249070, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmon \u2014 supervisor", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmon \u2014 supervisor", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t37551861", + "type": "text", + "x": 554, + "y": 106, + "width": 272.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 328034515, + "versionNonce": 936202818, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "drops caps \u00b7 no_new_privs \u00b7 restarts \u00b7 health", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "drops caps \u00b7 no_new_privs \u00b7 restarts \u00b7 health", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r622221395", + "type": "rectangle", + "x": 544, + "y": 162, + "width": 140, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 889582472, + "versionNonce": 815149810, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1572482465", + "type": "text", + "x": 554, + "y": 170, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1305207514, + "versionNonce": 284090082, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chatterd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chatterd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1070797839", + "type": "text", + "x": 554, + "y": 190, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1483230971, + "versionNonce": 1476818669, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "chat + bridge", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "chat + bridge", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r946102318", + "type": "rectangle", + "x": 694, + "y": 162, + "width": 126, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 506099524, + "versionNonce": 824958106, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t867087299", + "type": "text", + "x": 704, + "y": 170, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 810861933, + "versionNonce": 616560891, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1413681397", + "type": "text", + "x": 704, + "y": 190, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 545240425, + "versionNonce": 861768860, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc \u2192 OTLP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc \u2192 OTLP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1665347924", + "type": "rectangle", + "x": 830, + "y": 162, + "width": 126, + "height": 66, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 859704604, + "versionNonce": 655179138, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t449694979", + "type": "text", + "x": 840, + "y": 170, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1195887567, + "versionNonce": 502630597, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fwatch", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fwatch", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1456748557", + "type": "text", + "x": 840, + "y": 190, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 562187299, + "versionNonce": 95162100, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inotify + Landlock", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inotify + Landlock", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1987230863", + "type": "rectangle", + "x": 396, + "y": 378, + "width": 208, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 955413121, + "versionNonce": 1462389745, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1958507841", + "type": "text", + "x": 406, + "y": 386, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1918605557, + "versionNonce": 1087508139, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "host LGTM stack", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "host LGTM stack", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1102944863", + "type": "text", + "x": 406, + "y": 406, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1805025582, + "versionNonce": 272710576, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTLP :4318 via libvirt gateway", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTLP :4318 via libvirt gateway", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1980728160", + "type": "arrow", + "x": 130, + "y": 130, + "width": 23, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1835592721, + "versionNonce": 1531657573, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -23, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1749614568", + "type": "arrow", + "x": 240, + "y": 130, + "width": 3, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1154612234, + "versionNonce": 976880473, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 3, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a480779252", + "type": "arrow", + "x": 360, + "y": 130, + "width": 26, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1257622007, + "versionNonce": 503253929, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 26, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1998459078", + "type": "arrow", + "x": 640, + "y": 130, + "width": 26, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1753290799, + "versionNonce": 1016432110, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -26, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1228292860", + "type": "arrow", + "x": 760, + "y": 130, + "width": 3, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1200521872, + "versionNonce": 1452794454, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -3, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1606797766", + "type": "arrow", + "x": 870, + "y": 130, + "width": 23, + "height": 32, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1607023953, + "versionNonce": 1307612363, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 23, + 32 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a74856703", + "type": "arrow", + "x": 456, + "y": 195, + "width": 88, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 293373439, + "versionNonce": 1217119, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 88, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t231285176", + "type": "text", + "x": 500.0, + "y": 181.0, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1658533610, + "versionNonce": 18908090, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bridge from=NICK@node", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bridge from=NICK@node", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a947614368", + "type": "arrow", + "x": 243, + "y": 228, + "width": 197, + "height": 150, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1462674964, + "versionNonce": 630720771, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 197, + 150 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1266511448", + "type": "text", + "x": 341.5, + "y": 289.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 833152023, + "versionNonce": 399115703, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTLP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTLP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1149995420", + "type": "arrow", + "x": 757, + "y": 228, + "width": 197, + "height": 150, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 328281048, + "versionNonce": 1425019773, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -197, + 150 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t82351294", + "type": "text", + "x": 658.5, + "y": 289.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1262876870, + "versionNonce": 176148247, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTLP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTLP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1386587141", + "type": "text", + "x": 250, + "y": 240, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 482790657, + "versionNonce": 1712000026, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sandbox: cap-drop + Landlock ABI=7 (guest)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sandbox: cap-drop + Landlock ABI=7 (guest)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t702000783", + "type": "text", + "x": 750, + "y": 240, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 360831270, + "versionNonce": 1797062191, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sandbox: cap-drop + Landlock ABI=7 (guest)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sandbox: cap-drop + Landlock ABI=7 (guest)", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/41-fleet-topology.svg b/assets/diagrams/41-fleet-topology.svg new file mode 100644 index 0000000..15fdcde --- /dev/null +++ b/assets/diagrams/41-fleet-topology.svg @@ -0,0 +1,51 @@ + + + +systems-target · node=target + +systems-peer · node=peer + +pmon — supervisor +drops caps · no_new_privs · restarts · health + +fwatch +inotify + Landlock + +sysagent +/proc → OTLP + +chatterd +chat + bridge + +pmon — supervisor +drops caps · no_new_privs · restarts · health + +chatterd +chat + bridge + +sysagent +/proc → OTLP + +fwatch +inotify + Landlock + +host LGTM stack +OTLP :4318 via libvirt gateway + + + + + + + +bridge from=NICK@node +bridge from=NICK@node + +OTLP +OTLP + +OTLP +OTLP +sandbox: cap-drop + Landlock ABI=7 (guest) +sandbox: cap-drop + Landlock ABI=7 (guest) + \ No newline at end of file diff --git a/assets/diagrams/41-supervise-lifecycle.excalidraw b/assets/diagrams/41-supervise-lifecycle.excalidraw new file mode 100644 index 0000000..decc1b1 --- /dev/null +++ b/assets/diagrams/41-supervise-lifecycle.excalidraw @@ -0,0 +1,1026 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r706796686", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 860, + "height": 218, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1063506182, + "versionNonce": 1972266540, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1392269431", + "type": "text", + "x": 32, + "y": 50, + "width": 217.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1182009881, + "versionNonce": 1849888018, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmon \u2014 one child's supervise loop", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmon \u2014 one child's supervise loop", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r788089855", + "type": "rectangle", + "x": 36, + "y": 96, + "width": 112, + "height": 58, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1687617416, + "versionNonce": 1090189043, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1278345019", + "type": "text", + "x": 46, + "y": 104, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 294187050, + "versionNonce": 732624675, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pmon", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pmon", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t434340863", + "type": "text", + "x": 46, + "y": 124, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 11333135, + "versionNonce": 9332965, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "per child", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "per child", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r778946335", + "type": "rectangle", + "x": 186, + "y": 98, + "width": 118, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 676628099, + "versionNonce": 279145869, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t673397769", + "type": "text", + "x": 196, + "y": 106, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 380515534, + "versionNonce": 856986407, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "starting", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "starting", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1018798784", + "type": "rectangle", + "x": 342, + "y": 98, + "width": 118, + "height": 52, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 124386357, + "versionNonce": 281824724, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1018592285", + "type": "text", + "x": 352, + "y": 106, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1350648422, + "versionNonce": 1306427528, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "up", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "up", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1332004948", + "type": "rectangle", + "x": 498, + "y": 98, + "width": 132, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1291946956, + "versionNonce": 381194407, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1381252265", + "type": "text", + "x": 508, + "y": 106, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 771455407, + "versionNonce": 1887492272, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wait on child", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wait on child", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1141169940", + "type": "rectangle", + "x": 498, + "y": 194, + "width": 132, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1228709155, + "versionNonce": 1235803716, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1591840122", + "type": "text", + "x": 508, + "y": 202, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 190126203, + "versionNonce": 154147611, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "restarting", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "restarting", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t86539477", + "type": "text", + "x": 508, + "y": 222, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1843696801, + "versionNonce": 21378591, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "+300ms \u00b7 attempt++", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "+300ms \u00b7 attempt++", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r650776715", + "type": "rectangle", + "x": 700, + "y": 98, + "width": 150, + "height": 52, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 43275299, + "versionNonce": 896307561, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t158055501", + "type": "text", + "x": 710, + "y": 106, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1129855102, + "versionNonce": 1487038998, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "down", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "down", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1660136875", + "type": "text", + "x": 710, + "y": 126, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1559685045, + "versionNonce": 34833239, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SIGTERM \u2192 forwarded", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SIGTERM \u2192 forwarded", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a189656933", + "type": "arrow", + "x": 148, + "y": 125, + "width": 38, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1400765840, + "versionNonce": 90980812, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 38, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1637447370", + "type": "text", + "x": 167.0, + "y": 111.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 142413863, + "versionNonce": 458849800, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fork+exec", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fork+exec", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1349148143", + "type": "arrow", + "x": 304, + "y": 124, + "width": 38, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 396473973, + "versionNonce": 185145556, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 38, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t416978340", + "type": "text", + "x": 323.0, + "y": 110.0, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 238163855, + "versionNonce": 746414086, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "started pid=P", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "started pid=P", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a376343143", + "type": "arrow", + "x": 460, + "y": 124, + "width": 38, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 477457464, + "versionNonce": 1106502706, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 38, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a687161559", + "type": "arrow", + "x": 564, + "y": 150, + "width": 0, + "height": 44, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1396618348, + "versionNonce": 276325456, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 44 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1378592462", + "type": "text", + "x": 564.0, + "y": 158.0, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 316994119, + "versionNonce": 1586282603, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "unexpected exit", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "unexpected exit", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a41525888", + "type": "arrow", + "x": 498, + "y": 220, + "width": 97, + "height": 70, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 100266617, + "versionNonce": 561068715, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -97, + -70 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1133863665", + "type": "text", + "x": 449.5, + "y": 171.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1699216055, + "versionNonce": 641269690, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "re-exec", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "re-exec", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a848254917", + "type": "arrow", + "x": 630, + "y": 124, + "width": 70, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1186919576, + "versionNonce": 348041137, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 70, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t995228177", + "type": "text", + "x": 665.0, + "y": 110.0, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1447265479, + "versionNonce": 445014086, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shuttingDown", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shuttingDown", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t375958736", + "type": "text", + "x": 450, + "y": 278, + "width": 1113.2, + "height": 13.75, + "angle": 0, + "strokeColor": "#666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1051884483, + "versionNonce": 65292773, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "a health tick prints each child's state (starting|up|restarting|down) every --health-interval-ms; on SIGTERM/SIGINT pmon forwards the signal to all children, then prints pmon: shutdown", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "a health tick prints each child's state (starting|up|restarting|down) every --health-interval-ms; on SIGTERM/SIGINT pmon forwards the signal to all children, then prints pmon: shutdown", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/41-supervise-lifecycle.svg b/assets/diagrams/41-supervise-lifecycle.svg new file mode 100644 index 0000000..22c22cd --- /dev/null +++ b/assets/diagrams/41-supervise-lifecycle.svg @@ -0,0 +1,37 @@ + + + +pmon — one child's supervise loop + +pmon +per child + +starting + +up + +wait on child + +restarting ++300ms · attempt++ + +down +SIGTERM → forwarded + +fork+exec +fork+exec + +started pid=P +started pid=P + + +unexpected exit +unexpected exit + +re-exec +re-exec + +shuttingDown +shuttingDown +a health tick prints each child's state (starting|up|restarting|down) every --health-interval-ms; on SIGTERM/SIGINT pmon forwards the signal to all children, then prints pmon: shutdown + \ No newline at end of file diff --git a/assets/diagrams/README.md b/assets/diagrams/README.md index 071c083..4d4bb14 100644 --- a/assets/diagrams/README.md +++ b/assets/diagrams/README.md @@ -76,6 +76,8 @@ here as you add them: | `39-benchmark-pipeline` | 39 | benchlab's real path (warmup discard → per-iteration monotonic timing → sort/percentile and the parallel `co_correct` branch → the printed table) beside `--lie`'s one unwarmed wall-clock call straight to `elapsed_ns` — same codec, same fixed workload, two harnesses telling a different story | | `38-otlp-pipeline` | 38 | lsp-otel's telemetry path — serve's per-request parent span `request` wrapping `work` + `respond` children plus the `requests_total` counter and `request_duration` histogram, through the OTLP/HTTP exporter (batched ~2s, flushed on SIGINT) posting to the OTel collector on :4318, fanned to Tempo (traces) and Prometheus (metrics) and queried back through Grafana's datasource proxy — with cpp/go/rust each tagging service.name=lsp-otel- so the three never collide in the shared stack | | `37-use-method-checklist` | 37 | loadmix analyze as a pipeline — four one-shot pre-samples (uptime/dmesg/top/ss) then five interval samplers (vmstat/mpstat/iostat/sar/pidstat) spawned concurrently so an N-second checklist costs ~N seconds not 5N, each parsed by column name from its own header row into the 4×3 USE signal matrix (cpu/mem/io/net × Utilization/Saturation/Errors), and the max-Saturation-ratio verdict — with the saturate side shown as a worker pool the concurrent analyze attributes | +| `41-fleet-topology` | 41 | the capstone's two-VM fleet — pmon on systems-target and systems-peer each supervising fwatch/sysagent/chatterd (cap-dropped + Landlock-sandboxed), the two chatterds bridged across hosts (from=NICK@node), and both guests exporting OTLP over the libvirt gateway to the host LGTM stack | +| `41-supervise-lifecycle` | 41 | pmon's per-child supervise loop — fork+exec → starting → up → wait-on-child, an unexpected exit looping through restarting (+300ms, attempt++) back to up, and SIGTERM forwarding to all children then pmon: shutdown, with a health tick reading each child's state | | `40-fastpath-datapath` | 40 | one echo round trip, naive versus fastpath: the naive server's per-message heap buffer and blocking `read(2)` that sleeps through the kernel scheduler (one voluntary context switch per message) against the fastpath server's single reused buffer on a `sched_setaffinity`-pinned core busy-polling `recv(2)` over `EAGAIN`, never entering the scheduler (≈ zero voluntary context switches) — the measured 20500-vs-3 contrast that the chapter reads from `/proc//status` | | `36-proc-datasource-map` | 36 | the six /proc + cgroup PSI sources sysagent reads (stat, loadavg, meminfo, diskstats, net/dev, PSI), counters marked apart from gauges, feeding `take_snapshot`'s t0-read / concurrent-reader / sleep / t1-read / delta sequence that turns two cumulative `/proc/stat` reads into one `cpu_util_pct` | | `36-agent-architecture` | 36 | sysagent's `sample`/`serve` subcommands sharing one procfs reader layer, against the row of cross-check tools run at the same moment (vmstat, mpstat, free -m, ss -s, a direct `/proc/meminfo` read, `kill -INT`), with forward references to Chapter 37 (USE method), 38 (OTLP export), and 45 (PCP pmrep) |