diff --git a/CLAUDE.md b/CLAUDE.md index acf3c07..27d745d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,7 +51,15 @@ Ruby), examples in `examples/NN-slug/{cpp,go,rust}/`, chapters in (`status--verified` / `status--unverified`) state the behavioral evidence. Every number quoted as "from a run" must come from a real run. - **Banned words**: never use "honest", "honestly", "to be honest" — write - "real", "practical", "accurate" instead. + "real", "practical", "accurate" instead. Same rule for "lie"/"lies"/"lying" + **when the subject is the work, the method, or the reader** ("benchmarking + without lies", "the lie this chapter corrects") — it implies the rest of the + book is dishonest and calls the reader a liar. Keep "lie" when the subject is + a machine reporting a wrong value ("the mtime can lie", + "`hardware_concurrency()` lies about its container") — accurate idiom, no + accusation. Name a deliberately-bad variant after its defect (`--naive`, + `--unwarmed`), never `--lie`: flag names propagate into all three languages, + `verify.lua`, the recorded chapter output, and the example slug. ## Examples @@ -73,3 +81,9 @@ Ruby), examples in `examples/NN-slug/{cpp,go,rust}/`, chapters in - CI (`validate.yml`) checks site integrity + build smoke only; the authoritative gate is a host run of the runner with the lab up, recorded in `_plans/`. +- `_plans` is a **published** collection (`output: true`), and + `_plans/iteration-plan.md` is reader-facing — it is linked as "Roadmap" from + `_includes/footer.html` and `_layouts/part_index.html`, so never unpublish or + rename it without fixing both links. **Internal working plans and capture + dumps must set `published: false` in their front matter** so they stay in git + for resumability without shipping to the site. diff --git a/_docs/36-proc-sys-and-the-agent.md b/_docs/36-proc-sys-and-the-agent.md new file mode 100644 index 0000000..e44f702 --- /dev/null +++ b/_docs/36-proc-sys-and-the-agent.md @@ -0,0 +1,657 @@ +--- +title: "/proc, /sys, and the agent: a USE-method snapshot read straight from the kernel, no tools required" +order: 36 +part: "Observability" +description: "sysagent v0 is a USE-method metrics collector that reads /proc and /sys/fs/cgroup directly — no root, no shelled-out tools — computing CPU utilization from a two-read /proc/stat delta spanning --interval-ms, run-queue and load from /proc/loadavg, memory from /proc/meminfo, per-disk I/O from /proc/diskstats, network counters from /proc/net/dev, and cgroup PSI (falling back to system-wide /proc/pressure) gathered concurrently with the sampling sleep, exposed identically by a one-shot sample --json and a serve --port /metrics HTTP endpoint, with byte-for-byte identical field names across C++, Go, and Rust." +duration: "50 minutes" +--- + +Chapter 35's `overheadbench` measured syscall, memory, and I/O overhead across +a bare process, a container, and a VM — but every one of those numbers came +from watching `strace -c` and hand-timed loops wrapped around each run. That's +the right instrument for a one-off measurement campaign; it is not how a +program watches a system continuously, in production, for the months between +deploys. This chapter builds the tool that does: `sysagent`, a USE-method +metrics collector that reads `/proc` and `/sys/fs/cgroup` directly — no root, +no subprocess, no tool you'd have to trust is still installed on the box next +year. Chapter 37's `loadmix` builds the opposite kind of tool: an `analyze` +mode that shells out to `vmstat`, `mpstat`, `iostat`, `sar`, and `pidstat` +exactly the way an on-call engineer would, and parses their plain-text tables +by column name instead of reading kernel counters directly. The split between +the two chapters is deliberate — this one is the case for reading the +kernel's own accounting straight from the source, and the numbers it produces +are the ones a long-running agent can afford to compute every second without +forking a process to do it. + +The code is in `examples/36-proc-sys-and-the-agent/`. The run script there +builds/sets up and runs it; its `README.md` covers what it does and how to +drive it. + +{% include excalidraw.html + file="36-proc-datasource-map" + alt="A pipeline diagram of sysagent's take_snapshot. Left, a single /proc/stat read at t0 landing in a small box labeled 'cumulative counters since boot, not a percentage.' Center, a band labeled 'gathered concurrently with the --interval-ms sleep' containing five parallel boxes — /proc/loadavg, /proc/meminfo, /proc/diskstats, /proc/net/dev, and cgroup PSI (cpu.pressure/memory.pressure/io.pressure, falling back to /proc/pressure) — each writing into its own field of one shared Snapshot, all joined before the sleep's deadline is allowed to pass. A second /proc/stat read at t1 sits to the right of the band; arrows from both t0 and t1 feed a delta box computing cpu_util_pct/cpu_user_pct/cpu_system_pct as 100 times (busy1-busy0)/(total1-total0). A note reads 'two cumulative reads, --interval-ms apart, become one percentage.' Below, the completed Snapshot feeds two consumers drawn side by side: sample --json printing the object once to stdout, and serve --port P's GET /metrics handler calling take_snapshot fresh on every request. A small inset box shows the PSI fallback path: this process's own cgroup v2 delegate (resolved via the '0::' line in /proc/self/cgroup) tried first, the system-wide /proc/pressure/* files tried second, and psi_available:false — not an error — if neither is readable." + caption="Figure 36.1 — sysagent's take_snapshot: a t0/t1 /proc/stat delta bracketing five concurrently-gathered /proc + cgroup PSI sources, all landing in one Snapshot that sample --json and serve /metrics both emit" %} + +> **Tools used** — none, by design: `sysagent` reads `/proc/*` and +> `/sys/fs/cgroup/*` directly, with no subprocess, no bcc-tools/bpftrace, and +> no root. The only tool this chapter reaches for to observe it from the +> outside is `curl` (host), against the `serve --port P` `/metrics` endpoint +> — `curl` ships in Fedora's base install and isn't separately checked by +> `scripts/check-host.sh`, the same way Chapter 39 notes for `cpupower` and +> `objdump`. + +## Six sources, one deterministic schema + +`sysagent` has exactly two subcommands, and both produce the same shape of +data: + +``` +sysagent sample [--json] [--interval-ms N] # one snapshot to stdout +sysagent serve --port P [--interval-ms N] # the same snapshot over HTTP +``` + +`sample` takes one reading and prints it — `key=value` lines by default, +a single-line JSON object with `--json`. `serve` exposes the identical +snapshot over `GET /metrics`, taking a fresh reading on every request; any +other path 404s. Both walk the same six sources: `/proc/stat` (as a delta, +covered next), `/proc/loadavg`, `/proc/meminfo`, `/proc/diskstats`, +`/proc/net/dev`, and cgroup PSI. What makes this example worth its own +chapter isn't any one of those parsers — each is a plain-text file, `open`, +split fields, done — it's that the field names landing in the JSON are a +**deterministic, cross-language schema**: `cpu_util_pct`, `mem_total_kb`, +`disks[].reads`, `psi_available`, and every other key are byte-identical +whether the binary is the C++, Go, or Rust build. Point one Grafana +dashboard, one `jq` filter, or one test suite at any of the three and it +works unmodified — the entire point of README.md's field table, which this +chapter's `verify.lua` checks key by key. + +## The `/proc/stat` delta: two cumulative reads, one percentage + +`/proc/stat`'s `cpu` line is not a gauge. It's eight cumulative tick counters +— `user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal` — +that only ever grow, counted in USER_HZ ticks since the machine booted. A +single read of that line tells you nothing about *current* utilization; it +tells you the total CPU-seconds this machine has ever spent in each state. +`cpu_util_pct` only exists as the ratio of two such readings, `--interval-ms` +apart: `busy` is `user + nice + system + irq + softirq + steal`, `total` is +`busy + idle + iowait`, and the percentage is `100 × Δbusy / Δtotal` over +that window. `take_snapshot` reads `/proc/stat` once at the start (`t0`), +gathers everything else while `--interval-ms` elapses, reads `/proc/stat` +again (`t1`), and only then computes the delta: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp + auto t1 = read_cpu_ticks(); + if (!t1) return std::unexpected(t1.error()); + + auto busy0 = t0->user + t0->nice + t0->system; + auto busy1 = t1->user + t1->nice + t1->system; + auto idle0 = t0->idle + t0->iowait; + auto idle1 = t1->idle + t1->iowait; + auto total0 = busy0 + idle0 + t0->irq + t0->softirq + t0->steal; + auto total1 = busy1 + idle1 + t1->irq + t1->softirq + t1->steal; + + auto d_total = static_cast(total1 - total0); + auto d_busy = static_cast((busy1 - busy0) + (t1->irq - t0->irq) + + (t1->softirq - t0->softirq) + (t1->steal - t0->steal)); + auto d_user = static_cast(t1->user - t0->user); + auto d_system = static_cast(t1->system - t0->system); + + if (d_total > 0.0) { + snap.cpu_util_pct = 100.0 * d_busy / d_total; + snap.cpu_user_pct = 100.0 * d_user / d_total; + snap.cpu_system_pct = 100.0 * d_system / d_total; + } +``` + +```go + t1, err := readCPUTicks() + if err != nil { + return Snapshot{}, err + } + + busy0 := t0.user + t0.nice + t0.system + busy1 := t1.user + t1.nice + t1.system + idle0 := t0.idle + t0.iowait + idle1 := t1.idle + t1.iowait + total0 := busy0 + idle0 + t0.irq + t0.softirq + t0.steal + total1 := busy1 + idle1 + t1.irq + t1.softirq + t1.steal + + dTotal := float64(total1 - total0) + dBusy := float64((busy1 - busy0) + (t1.irq - t0.irq) + (t1.softirq - t0.softirq) + (t1.steal - t0.steal)) + dUser := float64(t1.user - t0.user) + dSystem := float64(t1.system - t0.system) + + if dTotal > 0 { + snap.CPUUtilPct = 100 * dBusy / dTotal + snap.CPUUserPct = 100 * dUser / dTotal + snap.CPUSystemPct = 100 * dSystem / dTotal + } +``` + +```rust + let t1 = read_cpu_ticks()?; + + let busy0 = t0.user + t0.nice + t0.system; + let busy1 = t1.user + t1.nice + t1.system; + let idle0 = t0.idle + t0.iowait; + let idle1 = t1.idle + t1.iowait; + let total0 = busy0 + idle0 + t0.irq + t0.softirq + t0.steal; + let total1 = busy1 + idle1 + t1.irq + t1.softirq + t1.steal; + + let d_total = (total1 - total0) as f64; + let d_busy = + ((busy1 - busy0) + (t1.irq - t0.irq) + (t1.softirq - t0.softirq) + (t1.steal - t0.steal)) + as f64; + let d_user = (t1.user - t0.user) as f64; + let d_system = (t1.system - t0.system) as f64; + + if d_total > 0.0 { + snap.cpu_util_pct = 100.0 * d_busy / d_total; + snap.cpu_user_pct = 100.0 * d_user / d_total; + snap.cpu_system_pct = 100.0 * d_system / d_total; + } +``` + +`irq`, `softirq`, and `steal` fold into `d_busy` alongside `user`/`nice`/ +`system` because all three represent CPU time genuinely spent — interrupt +handling, softirq processing, and (on a VM) time stolen by the hypervisor for +another guest — none of it is idle time, even though none of it is `user` or +`system` either. The `d_total > 0.0`/`dTotal > 0` guard exists because a +sufficiently short `--interval-ms` on an otherwise-idle single-CPU host could +in principle produce a zero-tick window (the kernel's tick granularity is +coarser than a few milliseconds); when that happens, all three percentages +stay at their zero-initialized default rather than dividing by zero. + +## Graceful degradation: PSI is optional, not an error + +Pressure Stall Information — the `some avg10=`/`avg60=`/`avg300=` lines in +`cpu.pressure`, `memory.pressure`, and `io.pressure` — tells you what +fraction of the last 10/60/300 seconds *some* task was stalled waiting on +that resource, which is a genuinely different signal from utilization: a CPU +can be 100% utilized with zero pressure (one thread pinning one core, nothing +else waiting) or 40% utilized with real pressure (many threads taking turns, +each one stalled part of the time). `read_psi` looks in two places, in +order: this process's own cgroup v2 delegate first — resolved by reading the +single `0::` line out of `/proc/self/cgroup` and joining it onto +`/sys/fs/cgroup/{cpu,memory,io}.pressure` — falling back to the +system-wide `/proc/pressure/{cpu,memory,io}` files if the cgroup-scoped ones +aren't there. Neither location is guaranteed: PSI can be compiled out of the +kernel, or a container can be handed a cgroup namespace with no delegated +`.pressure` files at all. When that happens, `sysagent` doesn't fail the +whole snapshot — `psi_available` comes back `false`, the three `avg10` +fields default to `0.0`, and every other field in the schema is unaffected. +A missing PSI source is data, not an error; the schema has a bit for exactly +that case so a dashboard built against it never has to special-case "did +this host even expose PSI." + +## How the code works + +`main` dispatches on `argv[1]` — `sample` or `serve`, anything else prints +the usage line and exits — and both subcommands share one hand-rolled +`Flags` scan (`--json`, `--interval-ms N`, `--port P`) kept in the same +shape across all three languages precisely so the argument-parsing code +never becomes the interesting part of the diff. `cmd_sample` calls +`take_snapshot(interval_ms)` once and prints either `to_text` or `to_json`; +`cmd_serve` hands `port` and `interval_ms` straight to `serve`, which owns +the HTTP loop and calls `take_snapshot` again on every `GET /metrics`. + +`to_json`/`ToJSON`/`to_json` is the piece that actually delivers the +cross-language guarantee the schema promises, and it earns that guarantee by +being deliberately unglamorous: none of the three languages hands the +`Snapshot` to a generic serializer. C++ builds the object with an +`ostringstream` and literal `"key":value,` fragments in a fixed order; Go +does the same with `strings.Builder` and `fmt.Fprintf`; Rust with `String` +and `push_str`. A generic serializer — `nlohmann::json`, `encoding/json`, +`serde_json` — would have been less code, but none of the three guarantees +the *same* key order or the same float formatting as the others out of the +box, and a schema that's supposed to be byte-identical across languages +can't leave that up to three different libraries' defaults. Hand-writing the +same fixed field order three times, with the same `%.2f`-style two-decimal +formatting (`fmt2`/`f2`/`{:.2}`) for every float, is what actually makes +`verify.lua`'s key-by-key checks — and a `diff` between any two languages' +output — meaningful. + +The HTTP layer is the one place the three implementations genuinely diverge +in mechanism rather than just syntax, and it follows the book's established +chatterd pattern: Go leans on `net/http`'s stdlib server, registering +`/metrics` on a `ServeMux` and letting `http.Server.Shutdown` handle a clean +`SIGINT`/`SIGTERM` stop. C++ and Rust hand-roll the same tiny surface +instead — a raw listening socket, `poll(2)` waiting on both the listener and +a `signalfd`/`SignalFd` doorbell, and one inline HTTP/1.1 response writer — +because pulling in a framework for one endpoint would hide the exact +mechanism this book has been building toward since Chapter 21's `chatterd` +and Chapter 24's signal-driven shutdown, not simplify it. + +## Errors, three ways + +`sysagent` has one usage-error surface and one runtime-error surface, and +they map to exit codes **2** and **1** respectively across all three +languages. A **usage error** — no subcommand, an unrecognized subcommand, +`serve` with no `--port`, or a non-positive `--interval-ms`/`--port` — +prints the single usage line to stderr and exits 2; every language funnels +every parse failure through the same `usage()` call before returning. A +**runtime error** — `/proc/stat`, `/proc/loadavg`, `/proc/meminfo`, +`/proc/diskstats`, or `/proc/net/dev` failing to open or parse — surfaces as +`sysagent: error: ` on stderr and exit 1; `/proc/stat` itself is +checked first, before any concurrent gathering even begins, since the whole +snapshot brackets around its two readings. A successful `sample` or a clean +signal-driven `serve` shutdown exits **0**. + +PSI sits outside that error surface entirely, and Rust's type signatures +make the guarantee visible at compile time rather than just by convention: +`read_loadavg`, `read_meminfo`, `read_diskstats`, and `read_netdev` all +return `Result` — they *can* fail, and `take_snapshot` propagates +that failure. `read_psi` returns `Option` — there is no `Err` variant +for it to construct, so a PSI-read failure literally cannot become a runtime +error; the type system rules it out before `take_snapshot` is ever called. +C++ mirrors the same asymmetry by discipline rather than by type: every +other worker thread calls `note_error(...)` on failure, but `psi_thread` +never does, its `if` guards only the success path. Go hardcodes the same +asymmetry directly into the fan-in loop, which explicitly `continue`s past +`r.source == "psi"` before counting it toward the required sources or a +possible `firstErr`. Three different enforcement mechanisms — a type the +compiler checks, a code path that's simply never written, a loop that +explicitly skips one name — for the same one-sentence rule: a missing PSI +source degrades the snapshot, it never fails it. + +## Concurrency lens + +Five of `sysagent`'s six sources are independent of each other and of the +`/proc/stat` delta they're bracketed by, so `take_snapshot` overlaps all five +reads with the `--interval-ms` sleep instead of paying for each sequentially +— the wall-clock cost of one `sample` is `max(interval_ms, gather_time)`, not +`interval_ms + gather_time`. Every language spawns the same five workers +(`loadavg`, `meminfo`, `diskstats`, `netdev`, `psi`), sleeps, and joins before +`t1` is read, guaranteeing every field is filled in before the delta is +computed — but *how* each language gets a result out of a worker back into +the shared `Snapshot` is where the three genuinely part ways: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +std::expected take_snapshot(int interval_ms) { + auto t0 = read_cpu_ticks(); + if (!t0) return std::unexpected(t0.error()); + + // Gather the non-CPU sources concurrently with the sampling sleep, using + // jthreads (auto-joining, cooperatively cancellable) rather than raw + // pthreads; ok_count is an atomic so every worker can report success + // without a mutex. + Snapshot snap{}; + std::atomic ok_count{0}; + std::string first_error; + std::mutex err_mu; + auto note_error = [&](std::string msg) { + std::lock_guard lock(err_mu); + if (first_error.empty()) first_error = std::move(msg); + }; + + std::jthread load_thread([&] { + if (auto la = read_loadavg()) { + snap.load1 = la->load1; + snap.load5 = la->load5; + snap.load15 = la->load15; + snap.runnable = la->runnable; + snap.total_threads = la->total_threads; + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(la.error()); + } + }); + std::jthread mem_thread([&] { + if (auto mi = read_meminfo()) { + snap.mem_total_kb = mi->total_kb; + snap.mem_available_kb = mi->available_kb; + snap.mem_used_kb = mi->total_kb - mi->available_kb; + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(mi.error()); + } + }); + std::jthread disk_thread([&] { + if (auto ds = read_diskstats()) { + snap.disks = std::move(*ds); + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(ds.error()); + } + }); + std::jthread net_thread([&] { + if (auto ns = read_netdev()) { + snap.net = std::move(*ns); + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(ns.error()); + } + }); + std::jthread psi_thread([&] { + if (auto psi = read_psi()) { + snap.psi_available = true; + snap.psi_cpu_some_avg10 = psi->cpu_some_avg10; + snap.psi_mem_some_avg10 = psi->mem_some_avg10; + snap.psi_io_some_avg10 = psi->io_some_avg10; + } + ok_count.fetch_add(1, std::memory_order_relaxed); + }); + + std::this_thread::sleep_for(std::chrono::milliseconds(interval_ms)); + + // jthread destructors join as each goes out of scope at the end of this + // block; explicitly joining here just makes the ordering-before-t1 clear. + load_thread.join(); + mem_thread.join(); + disk_thread.join(); + net_thread.join(); + psi_thread.join(); +``` + +```go +func TakeSnapshot(intervalMs int) (Snapshot, error) { + t0, err := readCPUTicks() + if err != nil { + return Snapshot{}, err + } + + var snap Snapshot + results := make(chan namedErr, 5) + + go func() { + la, err := readLoadAvg() + if err == nil { + snap.Load1, snap.Load5, snap.Load15 = la.load1, la.load5, la.load15 + snap.Runnable, snap.TotalThreads = la.runnable, la.totalThreads + } + results <- namedErr{"loadavg", err} + }() + go func() { + mi, err := readMemInfo() + if err == nil { + snap.MemTotalKB, snap.MemAvailableKB = mi.totalKB, mi.availableKB + snap.MemUsedKB = mi.totalKB - mi.availableKB + } + results <- namedErr{"meminfo", err} + }() + go func() { + disks, err := readDiskStats() + if err == nil { + snap.Disks = disks + } + results <- namedErr{"diskstats", err} + }() + go func() { + net, err := readNetDev() + if err == nil { + snap.Net = net + } + results <- namedErr{"netdev", err} + }() + go func() { + if p, ok := readPSI(); ok { + snap.PSIAvailable = true + snap.PSICPUSomeAvg10, snap.PSIMemSomeAvg10, snap.PSIIOSomeAvg10 = + p.cpuSomeAvg10, p.memSomeAvg10, p.ioSomeAvg10 + } + results <- namedErr{"psi", nil} + }() + + time.Sleep(time.Duration(intervalMs) * time.Millisecond) + + var firstErr error + required := 0 + for i := 0; i < 5; i++ { + r := <-results + if r.source == "psi" { + continue + } + required++ + if r.err != nil && firstErr == nil { + firstErr = fmt.Errorf("%s: %w", r.source, r.err) + } + } + if firstErr != nil { + return Snapshot{}, firstErr + } + _ = required +``` + +```rust +pub fn take_snapshot(interval_ms: u64) -> Result { + let t0 = read_cpu_ticks()?; + + let mut snap = Snapshot::default(); + let mut first_err: Option = None; + + thread::scope(|s| { + let load_h = s.spawn(read_loadavg); + let mem_h = s.spawn(read_meminfo); + let disk_h = s.spawn(read_diskstats); + let net_h = s.spawn(read_netdev); + let psi_h = s.spawn(read_psi); + + thread::sleep(Duration::from_millis(interval_ms)); + + match load_h.join().unwrap() { + Ok(la) => { + snap.load1 = la.load1; + snap.load5 = la.load5; + snap.load15 = la.load15; + snap.runnable = la.runnable; + snap.total_threads = la.total_threads; + } + Err(e) => { + first_err.get_or_insert(e); + } + } + match mem_h.join().unwrap() { + Ok(mi) => { + snap.mem_total_kb = mi.total_kb; + snap.mem_available_kb = mi.available_kb; + snap.mem_used_kb = mi.total_kb - mi.available_kb; + } + Err(e) => { + first_err.get_or_insert(e); + } + } + match disk_h.join().unwrap() { + Ok(disks) => snap.disks = disks, + Err(e) => { + first_err.get_or_insert(e); + } + } + match net_h.join().unwrap() { + Ok(net) => snap.net = net, + Err(e) => { + first_err.get_or_insert(e); + } + } + if let Some(psi) = psi_h.join().unwrap() { + snap.psi_available = true; + snap.psi_cpu_some_avg10 = psi.cpu_some_avg10; + snap.psi_mem_some_avg10 = psi.mem_some_avg10; + snap.psi_io_some_avg10 = psi.io_some_avg10; + } + }); +``` + +C++ and Go both mutate the *same* shared `snap` directly from inside each +worker — `load_thread` writes `snap.load1`/`load5`/`load15`, `mem_thread` +writes the three `mem_*` fields, and so on. This is race-free, but only +because every worker touches a *disjoint* set of fields: distinct struct +fields are distinct objects for data-race purposes in both languages' +memory models, so five threads writing to five non-overlapping field groups +of one struct never race — Go additionally needs the channel send/receive +in `results <- namedErr{...}` / `r := <-results` to establish the +happens-before edge that makes each goroutine's writes visible to the +receiver, which the buffered channel provides for free. Rust's `thread::scope` +takes a structurally different shape for a reason that isn't stylistic: the +five spawned closures (`s.spawn(read_loadavg)`, etc.) don't capture `snap` at +all, because two closures each holding `&mut snap` to write disjoint fields +would still require *two simultaneous mutable borrows of the same value* — +something the borrow checker refuses regardless of whether the underlying +fields actually overlap, since it reasons about the whole object, not the +fields a closure happens to touch. So Rust's workers return owned values +through their `JoinHandle`s instead, and every assignment into `snap` happens +in the *scope* thread itself, sequentially, immediately after each +`.join().unwrap()` — safe by construction, where C++ and Go are safe by the +programmer having correctly kept every worker's field set disjoint. + +## Build, run, observe + +```bash +[host]$ cd examples/36-proc-sys-and-the-agent && ./demo.sh build +``` + +A real snapshot, gathered on the reference host while the JSON output was +being captured for this chapter: + +```console +[host]$ ./demo.sh cpp run sample --json +{"cpu_util_pct":6.27,"cpu_user_pct":5.33,"cpu_system_pct":0.63,"load1":0.80, + "load5":0.88,"load15":1.04,"runnable":7,"total_threads":3961, + "mem_total_kb":65545860,"mem_available_kb":33268760,"mem_used_kb":32277100, + "disks":[{"name":"nvme0n1","reads":1092534,"writes":28109458,...}, ...], + "net":[{"iface":"lo","rx_bytes":1456453981,"tx_bytes":1456453981}, ...], + "psi_available":true,"psi_cpu_some_avg10":0.00,"psi_mem_some_avg10":0.00, + "psi_io_some_avg10":0.00} +``` + +`mem_total_kb=65545860` is ~62.5 GiB, this host's actual installed RAM; +`psi_available:true` because this box's cgroup v2 delegate exposes PSI, so +the process's own `cpu.pressure`/`memory.pressure`/`io.pressure` files were +readable — no fallback to `/proc/pressure` was needed on this run. With no +arguments at all, all three builds print the usage line to stderr and exit 2: + +```console +[host]$ ./demo.sh go run +usage: sysagent sample [--json] [--interval-ms N] | serve --port P [--interval-ms N] +``` + +`serve` exposes the same schema over HTTP, one fresh sample per request: + +```console +[host]$ ./demo.sh rust run serve --port 9100 & +sysagent: listening on 0.0.0.0:9100 +[host]$ curl -s http://127.0.0.1:9100/metrics | head -c 40 +{"cpu_util_pct":7.21,"cpu_user_pct":6.02, +[host]$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9100/nope +404 +[host]$ kill -INT %1 +sysagent: shutting down +``` + +The second `cpu_util_pct` (7.21) differs from the first (6.27) because +`/metrics` takes a genuinely new `--interval-ms`-apart sample on every +request rather than caching one — the same live measurement `sample` +prints, just reachable over the network. The unknown path 404s, and `SIGINT` +prints the shutdown line and exits 0. + +The runner drives all three languages on the local host: + +```console +[host]$ python3 scripts/test-all-examples.py --only 36-proc-sys-and-the-agent --mode local +verifying... + verify 36-proc-sys-and-the-agent [cpp]: PASS + verify 36-proc-sys-and-the-agent [go]: PASS + verify 36-proc-sys-and-the-agent [rust]: PASS + +example cpp go rust +36-proc-sys-and-the-agent PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: a static fact that has to match exactly, and a load that has to move + +Two different cross-checks back two different kinds of claim. The first is a +**static fact**: `mem_total_kb` is `MemTotal` straight out of +`/proc/meminfo`, unaffected by system load or timing, so `verify.lua` reads +it two ways in the same run — once through `serve`'s `/metrics` response, +once through a direct `awk '/^MemTotal:/{print $2}' /proc/meminfo` — and +requires the two numbers to agree **exactly**, not approximately. On this +host that's `65545860` from both sources, every time; any mismatch would +mean the parser is reading the wrong field or the wrong file, not measurement +noise, since nothing about this number is supposed to move between two +reads a few milliseconds apart. + +The second cross-check backs a **dynamic** claim instead: that `cpu_util_pct` +actually tracks real CPU load rather than returning a constant or a +miscomputed delta. `verify.lua` takes an idle `sample --json` baseline, spins +up `nproc` tight busy-loop children for slightly longer than a +`--interval-ms 500` window, samples again, and checks that `cpu_util_pct` +rose by a real margin — retried up to three times, with a 12-percentage-point +floor (or an outright ≥55% reading, for the case where the host is already +loaded before the check even starts). The margin and the retries exist +because this runs on a real, possibly-noisy development box, not an isolated +benchmark rig — the check has to distinguish "this host happened to be busy +for an unrelated reason" from "the parser is broken," and a single +non-retried comparison couldn't. + +## What you learned + +- **`/proc/stat` is a cumulative counter, not a gauge.** A percentage only + exists as the delta between two readings `--interval-ms` apart — + `take_snapshot` reads it once before the concurrent gather, once after, + and computes `cpu_util_pct` only from the difference. +- **Gather independent `/proc` sources concurrently with the sampling + sleep, not sequentially after it.** Five sources, one shared + `--interval-ms` window: the wall-clock cost of a sample is + `max(interval_ms, gather_time)`, not the sum. +- **A missing signal should degrade the data, not fail the call.** PSI + absent — kernel support missing, or a container with no cgroup v2 + delegate — sets `psi_available:false` rather than returning an error; + Rust enforces this at the type level (`Option`, no `Err` variant to + construct), C++ and Go enforce the same rule by never writing the failure + path for that one source. +- **A cross-language schema is only real if you hand-write the + serializer.** All three `to_json` implementations build the JSON object + field-by-field in a fixed order with the same float formatting, rather + than delegating to a generic library whose key order and number + formatting would drift between languages. +- **Rust's borrow checker forces a different concurrency shape, not just + different syntax.** C++ and Go both mutate one shared struct directly + from five worker threads — safe because the field sets are disjoint, but + only by the programmer's discipline. Rust can't express that pattern at + all without extra ceremony, so `thread::scope`'s workers return owned + values and every write into `snap` happens sequentially in the parent, + after each `.join()` — safe by construction instead. +- **`mem_total_kb` served over HTTP matched a direct `/proc/meminfo` read + exactly**, and a busy-loop child measurably raised `cpu_util_pct` over an + idle baseline — two cross-checks confirming the schema reports what the + kernel actually has, not what the code merely claims. + +Chapter 37 keeps the same Utilization/Saturation/Errors lens but inverts the +source of truth entirely: instead of reading `/proc` directly the way this +chapter does, `loadmix analyze` shells out to `uptime`, `vmstat`, `mpstat`, +`iostat`, `sar`, and `pidstat` exactly as an on-call engineer would, and +parses each tool's plain-text table by column name rather than reading +kernel counters itself. Read the two chapters back to back — same checklist, +same USE method, two structurally opposite ways of answering it. + +--- + +

verified — on the Fedora 44 +reference host (kernel 7.1.3-200.fc44, 16 CPUs, ~62 GiB RAM) this session: +python3 scripts/test-all-examples.py --only 36-proc-sys-and-the-agent +--mode local printed PASS PASS PASS (3 passed, 0 failed; +verify.lua reported PASS 32 / FAIL 0 for each of +cpp/go/rust). A real sample --json run printed +cpu_util_pct:6.27, mem_total_kb:65545860, +psi_available:true, and populated disks/net +arrays, exactly matching the schema in README.md. No arguments +printed the usage line on stderr and exited 2. serve --port P +printed sysagent: listening on 0.0.0.0:P; a second real sample +over GET /metrics read cpu_util_pct:7.21, confirming +each request takes a fresh reading; an unknown path returned HTTP 404; +SIGINT printed sysagent: shutting down and exited 0. +verify.lua's cross-check confirmed mem_total_kb +served over /metrics agrees exactly with a direct +awk '/^MemTotal:/{print $2}' /proc/meminfo read +(65545860 both ways), and its retried, margin-based check +confirmed a busy-loop child measurably raised cpu_util_pct above +an idle baseline. Not exercised: the psi_available:false +degraded path (this host's cgroup v2 delegate exposes PSI, so every sample +this session read psi_available:true) and any run against the +lab VM — this is a local-mode example (mode: local in the +manifest) that reads the current host's own /proc, not a +VM's.

diff --git a/_docs/37-use-method-60s-analysis.md b/_docs/37-use-method-60s-analysis.md new file mode 100644 index 0000000..416ee3e --- /dev/null +++ b/_docs/37-use-method-60s-analysis.md @@ -0,0 +1,608 @@ +--- +title: "The USE method in 60 seconds: a checklist that runs the real tools, not a shortcut around them" +order: 37 +part: "Observability" +description: "loadmix is Brendan Gregg's Utilization/Saturation/Errors checklist as a runnable program — a saturation generator that pegs one resource at a time, and an analyze mode that shells out to uptime, vmstat, mpstat, iostat, sar, and pidstat exactly as a human would, parses each tool's plain-text table by column name from its own header row, runs the five interval samplers concurrently so a 60-second checklist costs 60 seconds and not five minutes, and names the resource whose Saturation signal fired hardest." +duration: "50 minutes" +--- + +Chapter 36's `sysagent` built a USE-method collector from first principles — +reading `/proc/stat`, `/proc/net`, `/proc/diskstats`, and cgroup PSI directly, +no external process, no tool you'd have to trust. That is the right call for a +program that runs continuously in production. But it is not how most engineers +actually triage a system that's in trouble *right now*: they SSH in and reach +for `uptime`, `vmstat 1`, `mpstat -P ALL 1`, `iostat -xz 1`, `pidstat 1`, and +`free`, running the checklist Brendan Gregg popularized as "the USE method" — +for every resource, ask whether it's **Utilized**, **Saturated**, or +**Erroring** — because those tools already exist, are already trusted, and +already parse a live system correctly. `loadmix` is that checklist wired into +a program: a saturation generator that pegs one resource on command, and an +`analyze` mode that runs the real tools exactly as a human would, reads their +plain-text tables the way a human reads them — by column *name*, from each +tool's own header row — and tells you which resource is in trouble and why. + +The code is in `examples/37-use-method-60s-analysis/`. The run script there +builds/sets up and runs it; its `README.md` covers what it does and how to +drive it. + +{% include excalidraw.html + file="37-use-method-checklist" + alt="A pipeline diagram of loadmix analyze. Left column, four sequential pre-samples run once at start: uptime, dmesg tail, top -b -n1, and ss -s, each landing in a single printed 'tool' line. Center, a band labeled 'five interval samplers, spawned concurrently' containing five parallel boxes — vmstat 1 N, mpstat -P ALL 1 N, iostat -xz 1 N, sar -n DEV 1 N, and pidstat 1 N — each writing to its own temp file, all five joined by a single wait so the band's total wall time is N seconds, not 5N; a note under the band reads 'N-second checklist costs ~N seconds'. Right, each sampler's output feeds a column-name parser (Row::get against the tool's own header row) into a 4x3 signal matrix: rows cpu/mem/io/net, columns Utilization/Saturation/Errors, each cell a fired=true-or-false verdict against a fixed threshold — cpu's Saturation threshold shown as cpus+2 to make the guest-relative scaling visible. An arrow from the matrix's Saturation column into a final verdict box reads 'resource with the highest Saturation-value/threshold ratio wins'. A small inset box below the diagram shows the saturate side: app --resource cpu|mem|io|net --seconds N spinning a worker pool that the analyze pipeline, run concurrently against it, is able to see and attribute." + caption="Figure 37.1 — the analyze pipeline: four one-shot pre-samples, five interval samplers run concurrently, column-name parsing into a 4×3 USE signal matrix, and the highest-Saturation-ratio verdict" %} + +> **Tools used** — `uptime`, `dmesg`, `top`, `ss`, `vmstat`, `free` +> (procps-ng/util-linux/iproute2, base Fedora on the `systems-target` guest) +> plus `mpstat`, `iostat`, `sar`, `pidstat` (the `sysstat` package, installed +> by the guest's own cloud-init) — all on the `systems-target` VM, which +> `loadmix analyze` invokes directly as subprocesses. No tool here is +> `loadmix`-specific; this chapter's whole premise is that the checklist is +> the same nine commands you'd type by hand. + +## Two modes, one binary + +`loadmix` has exactly two things it does, and the split is deliberate: one +half creates a problem, the other half diagnoses it, so the same binary can +demonstrate both the checklist and its payoff. + +``` +app --resource cpu|mem|io|net --seconds N saturate that resource +app analyze [--seconds N] run the checklist and name + the saturated resource +``` + +`--resource cpu` spins a pool of worker threads doing nothing but xorshift +arithmetic — no syscalls, no allocation in the hot loop, pure CPU. `--resource +mem` touches 1.35× the guest's `MemTotal` across a worker pool, deliberately +overshooting available RAM to push real pages into swap. `--resource io` +opens one file per worker with `O_DSYNC` and `pwrite`s a 16 KiB buffer of +xorshift-generated (non-compressible) bytes to the same offset in a loop, +forcing a synchronous write to the block device on every call. `--resource +net` floods UDP datagrams over loopback from a pool of senders to one +receiver. Each saturator prints a `loadmix: start ...` line with the +resource, requested duration, and worker count, runs for (approximately) +that duration, and prints a `loadmix: done ... ops= bytes=` line. `analyze` +does none of that — it induces no load of its own, and instead spends its +`--seconds` window watching the system through the same nine tools listed +above. + +## Parsing by column name, not position + +The single design decision that makes `analyze` durable — the reason it +survives a `sysstat` or `procps-ng` version bump that reorders columns — is +that every parser reads a tool's header row first and looks fields up by +*name* against that row, never by a hardcoded index. `Row` (C++), `row` (Go), +and `Row` (Rust) all express the same idea: a table row is a name list zipped +with a value list, and `get`/`getOr` walks the name list linearly to find the +column a caller actually asked for. + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +// A parsed table row: column names (from a tool's own header line) zipped +// with this row's values, so every parser below reads fields *by name* +// rather than by a hardcoded position — robust to sysstat/procps column +// reordering across versions. +struct Row { + std::vector names; + std::vector values; + + [[nodiscard]] std::optional get(const std::string& name) const { + for (size_t i = 0; i < names.size() && i < values.size(); ++i) { + if (names[i] == name) { + try { + return std::stod(values[i]); + } catch (...) { + return std::nullopt; + } + } + } + return std::nullopt; + } +}; + +Row make_row(const std::vector& header_tokens, size_t header_skip, + const std::vector& data_tokens, size_t data_skip) { + Row r; + for (size_t i = header_skip; i < header_tokens.size(); ++i) r.names.push_back(header_tokens[i]); + for (size_t i = data_skip; i < data_tokens.size(); ++i) r.values.push_back(data_tokens[i]); + return r; +} +``` + +```go +// row is a parsed table row: column names (from a tool's own header line) +// zipped with this row's values, so every parser below reads fields *by +// name* rather than by a hardcoded position — robust to sysstat/procps +// column reordering across versions. +type row struct { + names []string + values []string +} + +func (r row) getOr(name string, def float64) float64 { + n := len(r.names) + if len(r.values) < n { + n = len(r.values) + } + for i := 0; i < n; i++ { + if r.names[i] == name { + v, err := strconv.ParseFloat(r.values[i], 64) + if err != nil { + return def + } + return v + } + } + return def +} + +func makeRow(headerTokens []string, headerSkip int, dataTokens []string, dataSkip int) row { + var r row + if headerSkip < len(headerTokens) { + r.names = append(r.names, headerTokens[headerSkip:]...) + } + if dataSkip < len(dataTokens) { + r.values = append(r.values, dataTokens[dataSkip:]...) + } + return r +} +``` + +```rust +// A parsed table row: column names (from a tool's own header line) zipped +// with this row's values, so every parser below reads fields *by name* +// rather than by a hardcoded position -- robust to sysstat/procps column +// reordering across versions. +struct Row { + names: Vec, + values: Vec, +} + +impl Row { + fn get(&self, name: &str) -> Option { + let n = self.names.len().min(self.values.len()); + for i in 0..n { + if self.names[i] == name { + return self.values[i].parse::().ok(); + } + } + None + } +} + +fn make_row( + header_tokens: &[String], + header_skip: usize, + data_tokens: &[String], + data_skip: usize, +) -> Row { + let names = header_tokens.get(header_skip..).unwrap_or(&[]).to_vec(); + let values = data_tokens.get(data_skip..).unwrap_or(&[]).to_vec(); + Row { names, values } +} +``` + +`header_skip`/`data_skip` exist because a header row and its data rows don't +always start aligned — `mpstat -P ALL`'s `Average:` rows both begin with a +label token (`Average: CPU` / `Average: all`) that has to be skipped before +`%idle` in the header lines up with the actual idle percentage in the data. +`parse_mpstat` puts this together end to end: it scans every line looking for +exactly those two `Average:` rows (sysstat's own aggregate over the whole +sampling window, which conveniently excludes the since-boot first report for +you), builds a `Row` from them, and reads `%idle` and `%iowait` by name: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +struct MpstatResult { + double busy_pct = 0.0; + double iowait_pct = 0.0; +}; + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically — sysstat's job, not ours). +MpstatResult parse_mpstat(const std::string& text) { + MpstatResult res; + std::vector header, data; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.size() < 2) continue; + if (toks[0] == "Average:" && toks[1] == "CPU") header = toks; + else if (toks[0] == "Average:" && toks[1] == "all") data = toks; + } + if (header.empty() || data.empty()) return res; + Row r = make_row(header, 2, data, 2); + double idle = r.get("%idle").value_or(100.0); + res.iowait_pct = r.get("%iowait").value_or(0.0); + res.busy_pct = 100.0 - idle; + return res; +} +``` + +```go +type mpstatResult struct { + busyPct float64 + iowaitPct float64 +} + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically — sysstat's job, not ours). +func parseMpstat(text string) mpstatResult { + res := mpstatResult{} + var header, data []string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) < 2 { + continue + } + if toks[0] == "Average:" && toks[1] == "CPU" { + header = toks + } else if toks[0] == "Average:" && toks[1] == "all" { + data = toks + } + } + if header == nil || data == nil { + return res + } + r := makeRow(header, 2, data, 2) + idle := r.getOr("%idle", 100.0) + res.iowaitPct = r.getOr("%iowait", 0.0) + res.busyPct = 100.0 - idle + return res +} +``` + +```rust +#[derive(Default)] +struct MpstatResult { + busy_pct: f64, + iowait_pct: f64, +} + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically -- sysstat's job, not ours). +fn parse_mpstat(text: &str) -> MpstatResult { + let mut res = MpstatResult::default(); + let mut header: Vec = Vec::new(); + let mut data: Vec = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.len() < 2 { + continue; + } + if toks[0] == "Average:" && toks[1] == "CPU" { + header = toks; + } else if toks[0] == "Average:" && toks[1] == "all" { + data = toks; + } + } + if header.is_empty() || data.is_empty() { + return res; + } + let r = make_row(&header, 2, &data, 2); + let idle = r.get("%idle").unwrap_or(100.0); + res.iowait_pct = r.get("%iowait").unwrap_or(0.0); + res.busy_pct = 100.0 - idle; + res +} +``` + +`busy_pct` is computed as `100.0 - %idle` rather than summing `%usr` + `%sys` ++ `%nice` + ... by name, because `%idle`'s complement is definitionally +"everything else," immune to a future sysstat release adding a new +utilization sub-category this parser has never heard of. The other three +parsers (`parse_vmstat`, `parse_iostat`, `parse_sar_pkts`) each carry one more +quirk worth knowing before you read the source: `vmstat` has no `Average:` +line at all, and its first sample is a since-boot average rather than a live +interval, so `parse_vmstat` scans every data row, discards row 0, and takes +the *max* of `r` (the run queue) across the rest — the steady-state +saturation signal, not one diluted by averaging across the whole window. +`iostat -xz` has the identical since-boot-first-row quirk, plus `-z` hides +idle devices entirely, so `parse_iostat` tracks block boundaries with its own +`avg-cpu:` counter and skips block 0 by index rather than by content. + +## How the code works + +`cmd_analyze` runs in three phases. First, four one-shot pre-samples — +`uptime`, `dmesg | tail -n 5`, `top -b -n1`, `ss -s` — each captured through +`run_capture`, the one function in the file with a real failure mode (the +shell itself failing to start), which is why it's the one that returns +`std::expected`/`error`/`Result` per this book's convention. `dmesg` on an +unprivileged account returns `Operation not permitted`; `cmd_analyze` checks +for that string and reports `status=denied` rather than treating it as a +capture failure, because an unprivileged `analyze` run is the common case, +not an edge case. + +Second, the five interval samplers — `vmstat`, `mpstat -P ALL`, `iostat -xz`, +`sar -n DEV`, `pidstat`, each invoked as ` 1 N` — are spawned +**concurrently**, each redirected to its own temp file, and only then +joined. This matters more than it looks: each of those five commands blocks +for `N` seconds sampling at a 1-second interval, so running them one after +another would make a 60-second checklist take five minutes. Running them +together makes it take 60 seconds, matching the chapter's own title. Third, +`free -m`'s single-shot output and the five sampler files are parsed (all by +column name, as above), assembled into eight `analyze: metric ...` lines, and +checked against the fixed threshold table that produces the USE signal +matrix and the verdict. + +That threshold table is worth reading directly, because it *is* the USE +method encoded as data: + +``` +cpu Utilization: busy_pct >= 60% Saturation: run_queue >= cpus+2 +mem Utilization: used_pct >= 75% Saturation: swap_io >= 300 KB/s +io Utilization: util_pct >= 40% Saturation: iowait_pct >= 8% +net Utilization: pkts/s >= 2000 Saturation: pkts/s >= 8000 +``` + +cpu's Saturation threshold is the one that scales with the machine it's +running on rather than being a flat number: `cpus + 2`, so a run queue longer +than the guest's own core count plus a little headroom means threads are +genuinely waiting for a CPU that isn't there — the textbook definition of +saturation, not an arbitrary constant that would be wrong on a differently +sized host. Errors is printed for every resource but always `fired=false`, +because this chapter's saturators induce load, not kernel-logged faults; the +line exists so the U/S/E triple is complete and so a future chapter that +*does* wire in `dmesg`-sourced error counts has somewhere to plug them in. +The verdict is whichever resource has the highest `Saturation value / +Saturation threshold` ratio — not the highest raw Saturation value, which +would let net's `8000`-scale threshold always lose to cpu's `4`-scale one +regardless of which resource is actually worse off. + +## Errors, three ways + +`loadmix` has one failure surface and one exit code for it: a **usage +error** — no `--resource`/`--seconds` given at all, an unrecognized +`--resource` value, `--seconds 0` or negative, or an unrecognized flag to +`analyze` — prints the two-line usage banner to stderr and exits **2**. C++ +funnels every parse path through `usage()` and returns `2` from `main`; Go +calls `usage()` then `os.Exit(2)`; Rust calls `usage()` then +`std::process::exit(2)`. There is no separate "runtime error" exit code here, +unlike Chapter 39's codec-bug path or Chapter 40's connection-refused path — +`analyze`'s subprocess calls are designed to degrade gracefully instead of +failing the whole run: a denied `dmesg`, an empty `ss -s`, or a sampler that +produces no data rows all show up as a `0.00`/`denied`/empty field in the +output rather than a non-zero exit, because the entire point of a diagnostic +checklist is that it still reports *something* under the same restricted +privileges a real on-call engineer would have. A successful run of either +mode — saturate or analyze — exits **0**. + +## Concurrency lens + +Every saturator uses the same three-part shape: compute a `deadline`, spawn a +worker pool that loops until `stop_requested() || now >= deadline`, then join. +C++'s pool is a `std::vector`; Go's is a `sync.WaitGroup` over +goroutines with `sync/atomic` counters; Rust's is `std::thread::scope`, the +`jthread` analogue where worker threads are RAII-joined when the scope block +ends. All three converge on identical worker-count formulas — `max(4, cpus*3)` +for cpu, `max(4, cpus*4)` for io and net, `max(2, cpus)` for mem — and +identical atomic counters accumulated across the pool. + +Where the three languages genuinely diverge is in how the parent thread stays +alive long enough for that deadline to matter. C++'s `std::jthread` runs +`request_stop()` *before* `join()` in its destructor, which means the instant +the enclosing scope's spawn loop finishes, every worker is told to stop — +before the deadline the workers are checking has had any chance to arrive. +Left alone, this was a real bug in the first cut of this example: every +saturator exited in single-digit milliseconds regardless of `--seconds`, +because `request_stop()` fired essentially at thread-creation time. The fix +is `std::this_thread::sleep_until(deadline)` in the parent, holding the +enclosing scope open for the full requested duration before it's allowed to +close and trigger the RAII-driven stop-then-join: + +```cpp + { + std::vector pool; + pool.reserve(workers); + for (unsigned w = 0; w < workers; ++w) { + pool.emplace_back([&total_iters, &checksum, deadline, w](std::stop_token st) { + uint64_t x = 0x9E3779B97F4A7C15ULL ^ (w + 1); + uint64_t iters = 0; + while (!st.stop_requested()) { + x ^= x << 7; + x ^= x >> 9; + ++iters; + if ((iters & 0xFFFFF) == 0 && std::chrono::steady_clock::now() >= deadline) break; + } + total_iters.fetch_add(iters, std::memory_order_relaxed); + checksum.fetch_add(x, std::memory_order_relaxed); // keeps x provably live (no DCE) + }); + } + // Hold the scope open for the full duration. ~jthread runs + // request_stop() *before* join(), so without this wait the workers are + // told to stop the instant the spawn loop ends and the deadline check + // below never fires. The sleep makes that stop request the intended + // post-deadline signal rather than an immediate kill. + std::this_thread::sleep_until(deadline); + } // jthreads join here (RAII) +``` + +Go and Rust never needed this hack, because in both ports each worker checks +its *own* deadline directly inside its loop condition (`time.Now().Before(deadline)`, +`Instant::now() < deadline`) rather than waiting on an external stop signal +raised by the parent — the parent simply calls `wg.Wait()` / lets +`thread::scope` return, which blocks until every worker's own loop condition +has already become false on its own. There is no analogue of `jthread`'s +stop-before-join ordering to fight, because nothing external tells these +workers to stop early; they stop themselves, on schedule, and the parent's +join is purely a rendezvous rather than part of the termination signal. + +The second, unrelated concurrency story in this chapter is `analyze`'s five +interval samplers, spawned as five real OS processes — `fork`+`execvp` in +C++, `exec.Command` on five goroutines in Go, `Command::spawn` in Rust — with +a single joint wait afterward. This isn't thread concurrency inside the +program at all; it's five independent subprocesses racing the wall clock +together so that, as the source comment puts it, "a 12s analyze window costs +~12s wall-clock, not 5x that from running them one at a time." + +## Build, run, observe + +```bash +[host]$ cd examples/37-use-method-60s-analysis && ./demo.sh build +``` + +This is a **VM example**: the saturation signals only mean something measured +against a real, otherwise-quiet kernel, so `verify.lua` stages the built +binary directly to `/tmp/lsp37-/app` on `systems-target` via plain +`scp`/`ssh` and drives it there — not through `demo.sh run +TARGET=...`, whose `deploy-to-vm.sh` wraps commands in `ssh -t` and injects +CRLF that breaks the tool's plain-text output. + +An idle checklist first, so the baseline is visible: + +```console +[vm]$ ./app analyze --seconds 10 +analyze: start seconds=10 cpus=2 +analyze: tool uptime raw="05:51:20 up 3:39, 1 user, load average: 1.78, 0.45, 0.15" +analyze: tool dmesg status=denied +analyze: tool top cpu="%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st" tasks="Tasks: 149 total, 2 running, 147 sleep, 0 d-sleep, 0 stopped, 0 zombie" +analyze: tool ss raw="Total: 167" +analyze: tool pidstat active_processes=4 +analyze: metric resource=cpu name=busy_pct value=1.34 unit=pct +analyze: metric resource=cpu name=run_queue value=3.00 unit=procs +... +analyze: signal resource=cpu type=Utilization fired=false value=1.34 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=false value=3.00 threshold=4.00 +... +analyze: verdict resource=cpu ratio=0.75 +analyze: done +``` + +`dmesg status=denied` is expected — the guest's unprivileged `fedora` user +gets `Operation not permitted` from `dmesg`, and `analyze` reports that +plainly rather than failing. This run finished in **10.417s** wall-clock for +a `--seconds 10` window, confirming the five samplers really do run +concurrently — five serial ` 1 10` invocations would have taken close +to 50 seconds. + +Now the payoff: a background saturator, then a concurrent checklist run +against it. + +```console +[vm]$ nohup ./app --resource cpu --seconds 25 >/tmp/bg.out 2>&1 & +[vm]$ sleep 2 && ./app analyze --seconds 12 +analyze: tool top cpu="%Cpu(s): 97.8 us, 2.2 sy, 0.0 ni, 0.0 id, ..." +analyze: metric resource=cpu name=busy_pct value=100.00 unit=pct +analyze: metric resource=cpu name=run_queue value=8.00 unit=procs +analyze: signal resource=cpu type=Utilization fired=true value=100.00 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=true value=8.00 threshold=4.00 +analyze: verdict resource=cpu ratio=2.00 +analyze: done +``` + +`busy_pct` rose from the idle baseline of **1.34** to **100.00**; `run_queue` +rose from **3.00** to **7.00–8.00** across three independent captures on this +2-vCPU guest — comfortably clearing both the `busy_pct >= 60` and +`run_queue >= cpus+2 = 4` thresholds by a wide margin, every time. +Utilization fired true, and the verdict landed on `cpu`: `analyze`, run +concurrently against a background `loadmix --resource cpu` saturator it knows +nothing about, correctly identified the resource in trouble by reading +exactly the same five tools a person would have run by hand. + +```console +[vm]$ ./app --resource cpu --seconds 5 +loadmix: start resource=cpu seconds=5 workers=6 +loadmix: done resource=cpu seconds=5 workers=6 ops=8801461550 bytes=11315899737308079569 +``` + +`workers=6` matches the documented formula, `max(4, cpus*3)` with `cpus=2` → +`max(4,6)=6`; this run's wall time was **5.002s** for a requested +`--seconds 5`, tracking the request within about 0.2 seconds. + +The runner drives all three languages against `systems-target`: + +```console +[host]$ python3 scripts/test-all-examples.py --only 37-use-method-60s-analysis --mode vm +verifying... + verify 37-use-method-60s-analysis [cpp]: PASS + verify 37-use-method-60s-analysis [go]: PASS + verify 37-use-method-60s-analysis [rust]: PASS + +example cpp go rust +37-use-method-60s-analysis PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: the same load, seen two ways + +The load-attribution run above is itself a cross-check — `analyze`'s verdict +is only trustworthy if it agrees with what you already know is true, since +you launched the saturator yourself. But it's worth pairing `analyze`'s own +report against `top`'s independent view, captured in the very same run: `top +-b -n1`'s `%Cpu(s)` line read `97.8 us, 2.2 sy, 0.0 id` while `mpstat`'s +`busy_pct` (100.0 − `%idle`) read `100.00`. The two numbers come from +different tools sampling at different instants over different windows — +`top` a single one-shot snapshot, `mpstat` an average over the whole +`analyze --seconds` window — so they are not expected to match to the decimal +place, and they don't; what they agree on is the story: this guest's CPUs are +saturated, not idle, not lightly loaded. A verdict that `mpstat` alone +disagreed with `top` about — one reporting near-idle while the other reported +saturated — would be the sign of a parsing bug, not measurement noise, since +both tools are reading the same underlying `/proc/stat` counters through +different sampling windows. + +## What you learned + +- **The USE method is a checklist, not a tool** — for every resource, ask + whether it's Utilized, Saturated, or Erroring, against a threshold you + choose up front. `loadmix analyze` is that checklist wired into a program + that runs the same nine commands (`uptime`, `dmesg`, `top`, `ss`, `vmstat`, + `mpstat`, `iostat`, `sar`, `pidstat`, `free`) a person would run by hand. +- **Parse plain-text tool output by column name, from the tool's own header + row, never by hardcoded position** — `Row::get`/`row.getOr` walk a name + list to find the field a caller asked for, which is what keeps every + parser correct across a `sysstat` or `procps-ng` version that reorders + columns. +- **Run independent interval samplers concurrently, not serially** — five + ` 1 N` processes spawned together and joined once made a + `--seconds 10` checklist finish in 10.417s instead of the ~50s five serial + invocations would have cost. +- **A saturation threshold should scale with the machine it runs on where + that's the accurate signal** — cpu's `run_queue >= cpus + 2` uses the + guest's own core count, unlike net's flat `8000 pkts/s`, because "threads + waiting for a CPU" only means something relative to how many CPUs exist. +- **`std::jthread`'s RAII order (`request_stop()` before `join()`) can silence + a worker before its own deadline ever fires** — the fix is holding the + parent scope open with `sleep_until(deadline)`; Go and Rust never needed + the workaround because each worker checks its own deadline directly rather + than waiting on an externally raised stop signal. +- **`analyze`, run concurrently against a background `--resource cpu` + saturator it knows nothing about, correctly attributed the load**: + `busy_pct` 1.34 → 100.00, `run_queue` 3.00 → 7.00–8.00, Utilization fired + true, verdict `cpu` — stable across three independent attempts on the lab + guest. + +Chapter 38 takes the numbers this checklist prints to a terminal and sends +them somewhere durable: OpenTelemetry into the LGTM stack Chapter 3 stood up, +so the next time a resource saturates you don't have to be watching when it +happens. + +--- + +

verified — on the +systems-target lab guest (Fedora 44, kernel +6.19.10-300.fc44, 2 vCPU nested KVM) this session: +python3 scripts/test-all-examples.py --only +37-use-method-60s-analysis --mode vm printed PASS PASS +PASS (3 passed, 0 failed; verify.lua reported +PASS 50 / FAIL 0 for each of cpp/go/rust). cpu/io/net +saturators tracked their requested --seconds within ~0.2s (a +--resource cpu --seconds 5 run printed workers=6, +matching max(4, cpus*3) for this guest's 2 cores, and completed +in 5.002s wall). analyze --seconds 10 against an idle guest +completed in 10.417s, not the ~50s five serial sampler invocations would +cost, confirming concurrent execution. The compound load-attribution run (a +background --resource cpu saturator, a concurrent +analyze --seconds 12) moved busy_pct from an idle +1.34 to 100.00 and run_queue from 3.00 to 7.00–8.00, with +Utilization fired=true and verdict resource=cpu, +stable across 3 independent attempts. --resource mem was +confirmed non-OOM on the guest at workers=2, +bytes=5525480448 (1.35×this guest's MemTotal), with +free -m/dmesg/journalctl -k checks +showing no OOM-kill and full memory/swap recovery after exit. Not exercised: +a privileged dmesg capture (the guest's unprivileged +fedora account reports status=denied, which +analyze treats as an expected, non-failing path) and any run +outside the lab guest — this is a VM example whose saturation signals are +only meaningful against a real, otherwise-quiet kernel.

diff --git a/_docs/38-opentelemetry-to-lgtm.md b/_docs/38-opentelemetry-to-lgtm.md new file mode 100644 index 0000000..5d4ac63 --- /dev/null +++ b/_docs/38-opentelemetry-to-lgtm.md @@ -0,0 +1,454 @@ +--- +title: "OpenTelemetry to LGTM: instrument a service, then prove the spans and metrics actually arrived" +order: 38 +part: "Observability" +description: "lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the LGTM stack from Chapter 3 — a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram — and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name." +duration: "50 minutes" +--- + +Chapter 37's `loadmix analyze` printed a USE-method verdict to a terminal, and +Chapter 36's `sysagent` served one JSON snapshot per HTTP request. Both make you +*present* to read the number: someone has to be running the command, or scraping +the endpoint, at the moment the system is in trouble. This chapter closes that +gap. `lsp-otel` is a small traced server that emits the three OpenTelemetry +signal shapes — a distributed **trace** per request, a monotonic **counter**, and +a latency **histogram** — and exports them over OTLP/HTTP into the LGTM stack +(Loki, Grafana, Tempo, Mimir plus an OpenTelemetry collector) that Chapter 3 +stood up, so the next time a request is slow you can go *look* at when it +happened instead of needing to have been watching. And because "I sent +telemetry" is the easiest claim in observability to get quietly wrong, the whole +point of the verification here is that it queries the stack back — Tempo for the +traces, Prometheus for the metrics — and refuses to believe the export happened +until the data is actually there to read. + +The code is in `examples/38-opentelemetry-to-lgtm/`. It is a **local-mode +example that requires the LGTM stack**: `serve` and `drive` run on the host, and +the exporter posts to the collector on `localhost:4318`. + +{% include excalidraw.html + file="38-otlp-pipeline" + alt="A left-to-right pipeline. Left, a band labeled 'serve — one traced request per input line' containing: a parent span box 'request (attr request.seq)' with two child span boxes below it, 'work' (sleep 1+seq%4 ms) and 'respond' (write ok seq), plus two metric boxes, 'requests_total (+1 per request)' and 'request_duration (ms)'. An arrow labeled OTLP leaves the band into an 'OTLP/HTTP exporter' box annotated 'batch ~2s, flush on SIGINT', which POSTs, over an amber arrow labeled POST /v1/{traces,metrics}, into an 'OTel collector :4318' box. The collector fans out along two arrows — 'spans' up to a 'Tempo — traces' box and 'metrics' across to a 'Prometheus — metrics' box — and both of those feed a 'Grafana query :3000' box on the right. A caption along the bottom reads: cpp · go · rust — each tags service.name = lsp-otel-, so the three never collide in the shared stack." + caption="Figure 38.1 — lsp-otel's telemetry path: a per-request span tree plus two metrics, batched through the OTLP/HTTP exporter to the collector on :4318, fanned to Tempo and Prometheus, and read back through Grafana — which is exactly how the export is verified" %} + +> **Tools used** — `podman` (host, runs the `lsp-lgtm` observability stack — +> OTLP on `:4318`, Grafana on `:3000`) and `curl` (host, the only tool the +> verification needs: it queries Prometheus and Tempo through Grafana's +> datasource proxy). The stack is the same `lsp-lgtm` container the +> `lgtm-podman-stack` setup from Chapter 3 provides; nothing else is required on +> the host. + +## Three signals, one request + +Every OpenTelemetry story is told in three signal types, and `lsp-otel` emits +all three from a single unit of work — one line of input to `serve`, treated as +one "request": + +- A **trace** describes the shape and timing of that request as a tree of + spans. `lsp-otel` starts a parent span named `request`, and *inside* it two + child spans: `work` (a deliberately tiny, variable `1 + seq%4` ms sleep + standing in for real computation) and `respond` (writing the `ok ` reply). + The parent/child nesting is what lets Tempo later show you that the request's + time was spent in `work`, not `respond`, or vice versa. +- A **counter** — `requests_total`, a monotonic `Int64` that goes up by one per + request — answers "how many?" It only ever increases, which is what makes it + a counter rather than a gauge, and what lets Prometheus compute a rate from it. +- A **histogram** — `request_duration`, in milliseconds — answers "how long?" + across the whole population, recording one observation (the parent span's + wall-clock duration) per request, so the backend can later report a p50, a + p99, or a bucketed distribution without the server having to keep every sample. + +The second subcommand, `drive --n N --addr HOST:PORT`, is a plain TCP client +that sends `N` requests and reads `N` replies. It is deliberately **untraced**: +it generates the load that makes `serve`'s telemetry interesting, but emits none +of its own, because the service under observation is `serve`, not the thing +poking it. + +## The request pipeline, three ways + +The heart of the instrumentation is one function — `handleRequest` — that starts +the parent span, nests the two children inside it, writes the reply, and records +both metrics. It reads almost identically across the three languages because the +OpenTelemetry API is deliberately the same shape in each; the differences are +idiomatic, not structural: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +bool handle_request(const telemetry::Handle& t, int fd, std::int64_t seq) { + const auto start = std::chrono::steady_clock::now(); + + auto parent_span = t.tracer->StartSpan("request", {{"request.seq", seq}}); + trace_api::Scope parent_scope(parent_span); + + // work: a tiny, deterministic-but-variable amount of simulated cost. + auto work_span = t.tracer->StartSpan("work", {{"request.seq", seq}}); + std::this_thread::sleep_for(std::chrono::milliseconds(1 + seq % 4)); + work_span->End(); + + // respond: write the reply line. + auto respond_span = t.tracer->StartSpan("respond", {{"request.seq", seq}}); + const std::string reply = "ok " + std::to_string(seq) + "\n"; + const bool wrote = write_all(fd, reply); + if (!wrote) { + respond_span->SetStatus(trace_api::StatusCode::kError, "write reply failed"); + } + respond_span->End(); + parent_span->End(); + + const double elapsed_ms = + std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + t.requests_total->Add(1); + // Histogram::Record(T) alone is an ABI-v2-only overload; this build is + // pinned to ABI v1 (matching the rest of the conan cache), so an + // explicit empty Context is required to select an ABI-v1 overload. + t.request_duration->Record(elapsed_ms, opentelemetry::context::Context{}); + + return wrote; +} +``` + +```go +func handleRequest(ctx context.Context, t *telemetry, w *bufio.Writer, seq int) error { + start := time.Now() + ctx, parent := t.tracer.Start(ctx, "request", + trace.WithAttributes(attribute.Int("request.seq", seq))) + defer parent.End() + + // work: a tiny, deterministic-but-variable amount of simulated cost. + _, work := t.tracer.Start(ctx, "work", + trace.WithAttributes(attribute.Int("request.seq", seq))) + time.Sleep(time.Duration(1+seq%4) * time.Millisecond) + work.End() + + // respond: write the reply line. + _, respond := t.tracer.Start(ctx, "respond", + trace.WithAttributes(attribute.Int("request.seq", seq))) + if _, err := fmt.Fprintf(w, "ok %d\n", seq); err != nil { + respond.RecordError(err) + respond.End() + return fmt.Errorf("write reply: %w", err) + } + if err := w.Flush(); err != nil { + respond.RecordError(err) + respond.End() + return fmt.Errorf("flush reply: %w", err) + } + respond.End() + + elapsedMs := float64(time.Since(start)) / float64(time.Millisecond) + t.requestsTotal.Add(ctx, 1) + t.requestDur.Record(ctx, elapsedMs) + return nil +} +``` + +```rust +fn handle_request(t: &Telemetry, w: &mut TcpStream, seq: i64) -> std::io::Result<()> { + let start = Instant::now(); + let parent_span = t + .tracer + .span_builder("request") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &Context::new()); + let parent_cx = Context::current_with_span(parent_span); + + // work: a tiny, deterministic-but-variable amount of simulated cost. + let mut work_span = t + .tracer + .span_builder("work") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &parent_cx); + thread::sleep(Duration::from_millis((1 + seq % 4) as u64)); + work_span.end(); + + // respond: write the reply line. + let mut respond_span = t + .tracer + .span_builder("respond") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &parent_cx); + let result = (|| -> std::io::Result<()> { + writeln!(w, "ok {seq}")?; + w.flush() + })(); + if let Err(ref e) = result { + respond_span.record_error(e); + } + respond_span.end(); + parent_cx.span().end(); + result?; + + let elapsed_ms = start.elapsed().as_secs_f64() * 1000.0; + t.requests_total.add(1, &[]); + t.request_duration.record(elapsed_ms, &[]); + Ok(()) +} +``` + +The parent/child relationship is carried differently in each, and it is the one +part worth reading closely. Go threads a `context.Context`: `tracer.Start` +returns a *new* context carrying the parent span, and passing that context into +the two child `Start` calls is what makes them children. Rust makes the same +context explicit — `Context::current_with_span(parent_span)` builds a context +holding the parent, and each child is started `start_with_context(..., &parent_cx)`. +C++ uses a `Scope` object: constructing `trace_api::Scope parent_scope(parent_span)` +installs the parent as the current span for the lifetime of that RAII object, so +the two children, started while the scope is alive, attach to it implicitly. +Three mechanisms — an explicit context value, an explicit context value again, +and a scope guard — for the identical span tree, which Tempo later shows as +`request` with `work` and `respond` beneath it. + +## How the code works + +Each language's `initTelemetry`/`init_telemetry`/`telemetry::init` wires up two +exporters against a shared **resource** — the set of attributes that identify +*this* service in the stack. The critical one is `service.name`, and it is +deliberately different per language: `lsp-otel-cpp`, `lsp-otel-go`, +`lsp-otel-rust`. Because all three export into the *same* shared LGTM stack, a +distinct `service.name` is what keeps their signals from colliding and lets the +verification query each language's telemetry by its own name. + +Both exporters speak **OTLP over HTTP/protobuf** to the collector, and both are +**batched on a short (~2 second) interval** — a batch span processor for traces, +a periodic reader for metrics. That interval is a deliberate choice: the SDK +defaults (5-second span batches, 60-second metric batches) are fine for a +long-running service but far too slow for a demo that starts, drives ten +requests, and stops within a couple of seconds. A short interval plus a **flush +on shutdown** is what guarantees the last batch — often *all* the batches, at +this scale — actually leaves the process before it exits. That shutdown path is +the subject of the Concurrency lens below, because it turned out to be where the +three languages diverge most. + +## Errors, three ways + +`lsp-otel` has two exit codes for failure. A **usage error** — no subcommand, an +unknown subcommand, `serve` without `--port`, or `drive` with `--n 0` or missing +— prints the two-line usage banner to stderr and exits **2**: + +``` +usage: app serve --port PORT [--otel-endpoint URL] + app drive --n N --addr HOST:PORT +``` + +A **runtime error** is exit **1** with an `app: error: …` message: the one +`verify.lua` exercises is `drive` against an address nothing is listening on, +which fails to connect and prints `app: error: dial …: connection refused`. This +is the one message that is *not* byte-identical across the three languages — +each surfaces its own dial/`errno` wording, the same way Chapter 40's `connect` +errors carry each platform's `strerror` text — so `verify.lua` asserts only the +shared `app: error:` prefix and the exit code, not the full string. A clean run +of either subcommand exits **0**: `serve` after a signal-driven shutdown, `drive` +after `N/N` successful replies. + +## Concurrency lens + +`serve` has to wait on two fundamentally different things at once: a new +connection arriving, and a shutdown signal arriving. All three ports solve this +the same way in shape — a signal source and an accept loop both feed a *single* +channel that the main loop consumes one event at a time — and all three write it +with the idiom their language reaches for: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp + for (;;) { + Event e = channel.pop(); + if (e.kind == Event::Kind::Signal) { + std::println("app: shutting down ({})", e.signal_name); + std::fflush(stdout); + const bool ok = t.flush_and_shutdown(std::chrono::milliseconds(5000)); + if (!ok) { + std::println(stderr, "app: telemetry shutdown: one or more providers failed to flush/shutdown"); + } + return 0; + } + serve_conn(t, e.conn_fd); + } +``` + +```go + for { + select { + case sig := <-sigCh: + fmt.Printf("app: shutting down (%s)\n", sig) + ln.Close() + shutCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + if err := shutdown(shutCtx); err != nil { + fmt.Fprintln(os.Stderr, "app: telemetry shutdown:", err) + } + cancel() + return 0 + case conn := <-connCh: + serveConn(t, conn) + } + } +``` + +```rust + loop { + match rx.recv() { + Ok(Event::Signal(name)) => { + println!("app: shutting down ({name})"); + t.shutdown(); + return 0; + } + Ok(Event::Conn(stream)) => { + serve_conn(&t, stream); + } + Err(_) => return 0, + } + } +} +``` + +Go's `select` over `sigCh` and `connCh` is the native version — the language has +a built-in construct for exactly "wait on whichever of these channels is ready +first." Rust rebuilds the same thing from an `mpsc::channel` carrying an `Event` +enum (`Signal` or `Conn`), fed by a `signal-hook` thread and an accept thread. +C++ does likewise with a hand-written `EventChannel` fed by a `sigwait()` thread +and an accept thread. Same architecture, three spellings. + +Getting there surfaced three portability lessons worth keeping, because each is +the kind of bug that leaves the program looking fine while silently doing +nothing: + +1. **The OTLP/HTTP exporter must POST to the full per-signal URLs** — + `http://localhost:4318/v1/traces` and `…/v1/metrics`, not the bare base + `http://localhost:4318`. Both the Rust and C++ ports first tried the base URL; + the collector answered `404`, the export silently failed, and the *only* + outward sign was a shutdown-time flush error. The Go SDK hides this by + appending the path for you; the others make you spell it out. +2. **Each SDK logs internally by default, and that breaks the output contract.** + `serve`'s output must be exactly two lines. opentelemetry-cpp writes + Info/Debug to stdout and Warning/Error to stderr unless you install a + `NoopLogHandler` at `LogLevel::None`; the Rust build had to *not* register a + `tracing-subscriber` at all. Silence is part of the port. +3. **C++ only: a background job inherits `SIGINT` set to `SIG_IGN`.** A + non-interactive shell running `serve &` — exactly how `verify.lua` starts it — + sets `SIGINT`/`SIGQUIT` to *ignore* in the child before `exec`, and that + disposition is inherited. `pthread_sigmask` + `sigwait()` alone cannot consume + a signal whose disposition is "ignore"; the port has to install a real (no-op) + `sigaction` first to move the disposition off `SIG_IGN` — precisely what Go's + `signal.Notify` and Rust's `signal-hook` do under the hood, which is why only + the C++ port had to do it by hand. (Two smaller C++-only wrinkles rounded it + out: `Histogram::Record` needs the `(value, Context{})` overload under this + build's ABI v1, and the OTLP static archives have a genuine link cycle that + needs `-Wl,--start-group`.) + +## Build, run, observe + +```bash +[host]$ cd examples/38-opentelemetry-to-lgtm && ./demo.sh build +``` + +Run the server, drive ten requests through it, and stop it — the SIGINT flush is +what pushes the final batch out: + +```console +[host]$ ./go/bin/app serve --port 8080 & +app: serve listening on 127.0.0.1:8080 +[host]$ ./go/bin/app drive --n 10 --addr 127.0.0.1:8080 +app: drive sent 10/10 ok +[host]$ kill -INT %1 +app: shutting down (interrupt) +``` + +Now read the telemetry back out of the stack — not out of the program. Metrics +land in Prometheus; querying `requests_total` through Grafana's datasource proxy +shows the counter for whichever language just ran: + +```console +[host]$ curl -s -u admin:admin \ + 'http://localhost:3000/api/datasources/proxy/uid/prometheus/api/v1/query?query=requests_total{service_name="lsp-otel-go"}' +... "value":[ ..., "10"] ... +``` + +The runner drives all three languages against the live stack and asserts the +full contract — including the stack queries — for each: + +```console +[host]$ python3 scripts/test-all-examples.py --only 38-opentelemetry-to-lgtm --mode local +verifying... + verify 38-opentelemetry-to-lgtm [cpp]: PASS + verify 38-opentelemetry-to-lgtm [go]: PASS + verify 38-opentelemetry-to-lgtm [rust]: PASS + +example cpp go rust +38-opentelemetry-to-lgtm PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: two signals, the same ten requests + +The verification's real strength is that it confirms the export from **two +independent backends**, tagged by each language's own `service.name`. After a +`drive --n 10`, both the counter and the histogram's observation count read +exactly **10** in Prometheus, for all three languages: + +| `service_name` | `requests_total` | `request_duration_milliseconds_count` | +|---|---|---| +| `lsp-otel-cpp` | 10 | 10 | +| `lsp-otel-go` | 10 | 10 | +| `lsp-otel-rust` | 10 | 10 | + +And the *traces* for the same runs are in Tempo: a search by +`service.name=lsp-otel-` returns traces whose `rootServiceName` is that +name and whose `rootTraceName` is `request`, and fetching one back shows its +spans are exactly `work`, `respond`, and the `request` parent — the tree +`handle_request` built. Metrics saying "ten requests happened" and traces saying +"here is the shape of each of those requests" are two different subsystems +agreeing about the same work, which is a far stronger signal that the +instrumentation is correct than either backend alone — and neither number came +from the program telling us about itself. + +## What you learned + +- **A service emits observability as three signal types** — traces (a tree of + parent/child spans describing one request's shape and timing), a monotonic + counter (`requests_total`, "how many"), and a histogram (`request_duration`, + "how long", one observation per request) — and `lsp-otel` produces all three + from a single `handle_request`. +- **Parent/child span nesting is carried by context** — Go and Rust thread an + explicit context value into each child `Start`, C++ installs the parent with a + `Scope` guard — and it is what turns three separate spans into the + `request → {work, respond}` tree Tempo shows. +- **Export is batched, and a flush on shutdown is what makes the last data + arrive** — a short (~2s) interval plus a signal-driven flush, not the slow SDK + defaults, is why a start-drive-stop demo's telemetry shows up at all. +- **Prove the export landed by querying the backend, never the program.** The + verify step reads `requests_total` and `request_duration_milliseconds_count` + out of Prometheus and the trace tree out of Tempo, through Grafana's proxy — an + export that 404'd silently (the wrong-endpoint bug both non-Go ports hit) fails + this check even though the program looked healthy. +- **The same OTLP contract needs per-language care**: the full `/v1/traces` and + `/v1/metrics` URLs, silencing each SDK's default logging to keep the output + contract, and — in C++ — installing a real signal handler to override the + `SIG_IGN` a background shell hands a child before `sigwait` can ever see the + signal. + +Chapter 39 turns from *collecting* numbers to *trusting* them: how to benchmark +your own code without the warmup, coordinated-omission, and dead-code traps that +make most microbenchmarks quietly measure the wrong thing. + +--- + +

verified — on the Fedora 44 +reference host (kernel 7.1.3-200.fc44) this session, with the lsp-lgtm +LGTM stack running (OTLP on :4318, Grafana on :3000): +python3 scripts/test-all-examples.py --only 38-opentelemetry-to-lgtm +--mode local printed PASS PASS PASS (3 passed, 0 failed; +verify.lua reported PASS 15 / FAIL 0 for each of +cpp/go/rust). After a serve + drive --n 10 + SIGINT for +each language, querying through Grafana's datasource proxy returned +requests_total{service_name="lsp-otel-<lang>"} = 10 and +request_duration_milliseconds_count{service_name="lsp-otel-<lang>"} += 10 for all three, and Tempo returned traces with +rootServiceName=lsp-otel-<lang>, +rootTraceName=request, and spans work/respond/request. +serve's output was exactly its listening and +shutting down (SIGINT) lines with no SDK log noise; +drive printed drive sent 10/10 ok and exited 0; a drive +to a dead address exited 1 with an app: error: prefix. Not +exercised: any run with the LGTM stack down (verify.lua SKIPs that +case rather than failing) and any lab-VM run — this is a host-only, local-mode +example.

diff --git a/_docs/39-benchmarking-without-lies.md b/_docs/39-benchmarking-without-lies.md new file mode 100644 index 0000000..b984051 --- /dev/null +++ b/_docs/39-benchmarking-without-lies.md @@ -0,0 +1,541 @@ +--- +title: "Benchmarking without lies: warmup, coordinated omission, and the percentile the mean hides" +order: 39 +part: "Performance and Low Latency" +description: "benchlab measures the ch21 chatterd frame codec two ways in the same binary — a real run with warmup discard, per-iteration timing, and a coordinated-omission-corrected p99, and --lie's single unwarmed wall-clock sample — cross-checked with perf stat cycles/instructions and an objdump proof that an unwatched microbenchmark just gets compiled away." +duration: "50 minutes" +--- + +Chapter 37 gave you the USE method's 60-second checklist for a system already +in trouble, and Chapter 38 wired that telemetry into the LGTM stack you can +query back. Both assume the numbers arriving are real. This chapter asks the +harder question: when *you* produce the number — a microbenchmark of your own +function, timed on your own laptop — what makes it trustworthy? The uncomfortable +answer is that most benchmarks people paste into commit messages and PR +descriptions are not measuring what they claim to. `benchlab` is built to show +exactly how, by measuring the same thing two ways in the same binary: once +properly, and once as the anti-pattern this chapter argues against, so you can +watch the story change under your own two hands. + +The code is in `examples/39-benchmarking-without-lies/`. The run script there +builds/sets up and runs it; its `README.md` covers what it does and how to +drive it. + +{% include excalidraw.html + file="39-benchmark-pipeline" + alt="Two bands. Top, benchlab real path: a warmup loop of W untimed calls to run_once feeding a timed loop of N iterations each timed with the monotonic clock into a raw samples array with a running checksum, splitting into a sort-and-percentile path producing min/median/p99/max and a parallel co_correct path producing co_p99_ns, co_n, and checksum, both landing in a printed table. Bottom, benchlab --lie: one untimed call straight to a wall-clock delta straight to a printed elapsed_ns, no percentiles because there is nothing to compute one from. A note reads same codec, same fixed workload — the harness, not the code under test, is what changes the story." + caption="Figure 39.1 — the same codec measured two ways: warmup, per-iteration timing, and coordinated-omission correction versus one unwarmed wall-clock sample" %} + +> **Tools used** — `perf stat`, `cpupower`, `objdump`, `g++` (host, ad hoc +> compile for the dead-code-elimination demonstration). `perf` and `g++`/`gcc` +> are checked by `scripts/check-host.sh`; `cpupower` and `objdump` ship with +> Fedora's `kernel-tools` and `binutils`, which this book already requires. + +## Warmup, and why a cold call is not a measurement + +`benchlab --op roundtrip --iters N --warmup W` runs `W` throwaway calls to +`run_once` first, discarding every one, before it times a single sample. The +reason is that the *first* calls to almost anything are not representative of +the steady state a real caller experiences. The allocator has not yet carved +out the arena it will reuse for the rest of the run; the branch predictor has +not yet learned which way `decode_frame`'s length check usually goes; on a +JIT'd or GC'd runtime the picture is worse still (Go's escape analysis and GC +pacer are both steady-state phenomena — irrelevant here since the codec is +allocation-light, but the discipline generalizes). Skip the warmup and your +"benchmark" is measuring cold-start cost, which is a real and useful number, +just not the one anyone means by "how fast is `encode_frame`." + +Run-to-run variance is the second reason a single sample is worthless. Ten +runs of the same fixed workload on the same host produce ten different +numbers, because the scheduler, the cache state, and (on this laptop) the +CPU's own clock all move between runs. `benchlab`'s answer is to never trust +one number: it collects `N` individually-timed samples and reports the +distribution — min, median, p99, max — not a single average. + +## Fixing the clock: the governor is part of the experiment + +Before any of those samples mean anything, the thing that's supposed to be +constant across the run — the CPU's clock rate — has to actually be close to +constant. Modern Intel chips run under `intel_pstate`, which picks a frequency +per core, per instant, based on load and a policy called the governor. On the +reference host for this chapter: + +```console +[host]$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor +powersave +[host]$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors +performance powersave +[host]$ cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq +800000 +4600000 +``` + +That is an 800 MHz to 4.6 GHz range on one core, entirely at the governor's +discretion under `powersave`. `cpupower frequency-info` reads the same policy +back with more context: + +```console +[host]$ cpupower frequency-info +analyzing CPU 1: + driver: intel_pstate + ... + energy performance preference: balance_performance + hardware limits: 800 MHz - 4.60 GHz + available cpufreq governors: performance powersave + current policy: frequency should be within 800 MHz and 4.60 GHz. + The governor "powersave" may decide which speed to use + within this range. + current CPU frequency: 2.05 GHz (asserted by call to kernel) + boost state support: + Supported: yes + Active: yes +``` + +`powersave` with turbo **Active: yes** means the core running your benchmark +loop can legitimately be executing the same instruction stream at 800 MHz in +one sample and north of 4.6 GHz two samples later, purely because the +governor decided the load justified a boost — or a neighboring core's thermal +budget forced a step down. That is a confound with nothing to do with your +code, and it lands directly in `max_ns` and in the tail percentiles this +chapter cares about. The fix, on a host you control, is to pin the governor +before you benchmark: `echo performance | sudo tee +/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor` (or the equivalent +`sudo cpupower frequency-set -g performance`) locks every core to its highest +non-turbo sustained frequency for the duration of the run. This chapter's own +numbers below were captured *without* making that change — deliberately, so +that the variance the next two sections discuss is the real variance this +host produces under its default policy, not variance edited away before you +could see it. If you reproduce these numbers and they differ, check your +governor first. + +## Coordinated omission: the request that didn't get to start + +`benchlab`'s timed loop is a **closed loop**: iteration `N+1` does not start +its clock until iteration `N` has returned. This is the natural way to write +a benchmark loop, and it is also, in Gil Tene's phrase, the exact shape that +produces **coordinated omission**. Suppose iteration 4 stalls — a page fault, +a context switch onto a different core, a neighbor process winning the +scheduler's attention — for 1000x the typical iteration cost. The closed loop +records exactly *one* sample for that stall: iteration 4's latency. But a real +caller is not a closed loop. A client sending one request every 47 ns (this +run's median) does not pause its request rate while iteration 4's stall plays +out — five, ten, a hundred more requests queue up behind it, each waiting its +turn, each experiencing a delay close to the stall's own length. The closed +loop's benchmark omitted every one of those queued requests from its sample +set. It didn't just under-measure the tail; it *erased* the part of the +distribution the tail exists to describe. + +{% include excalidraw.html + file="39-coordinated-omission" + alt="Two stacked bands. Top, benchlab's raw closed-loop timer: iterations 1 through 3 each a small box around 40 nanoseconds, iteration 4 an accent box labeled STALL page fault slash sched, 1057 nanoseconds, one sample. Bottom, what a fixed-rate caller would have seen: the same three small requests, the same stalling request 4, then two dashed ghost boxes labeled request 5 queued and request 6 queued at roughly 1018 and 979 nanoseconds, backfilled synthetic samples continuing down toward the expected interval floor. A dashed vertical line connects the two request 4 boxes labeled same stall. Captions read the raw p99 over a 1000-sample window buries the single stall sample, while the corrected co_p99 backfills the queued requests so co_p99_ns is always at least p99_ns." + caption="Figure 39.2 — one stall, recorded once by the closed loop (top) versus what a fixed-rate caller actually experiences (bottom); co_correct backfills the difference" %} + +The correction — HdrHistogram calls it `recordValueWithExpectedInterval` — is +mechanical once you see the picture: for every raw sample `v` that exceeds an +*expected interval* `E` (the per-op time a sustained caller would target), +backfill synthetic samples `v - E, v - 2E, ...` down to `E`, standing in for +the requests that would have queued behind the stall. `benchlab` sets `E` to +the run's own raw median, floored at 1 ns, so the correction needs no extra +flag and comes from the same run it corrects — and caps the backfill at +5,000,000 synthetic samples so one pathological stall can't grow the array +without bound. On a run captured this session: + +```console +[host]$ perf stat -e cycles,instructions,task-clock \ + ./cpp/build/release/app --op roundtrip --iters 100000 --warmup 5000 +benchlab: op=roundtrip iters=100000 warmup=5000 +benchlab: n=100000 min_ns=38 median_ns=42 p99_ns=86 max_ns=56595 +benchlab: co_p99_ns=16930 expected_interval_ns=42 co_n=107077 +benchlab: checksum=000000000079c3e0 +``` + +The raw `p99_ns` is **86 nanoseconds** — a single stall out near `max_ns` +(56,595 ns, almost 1350x the median) barely registers, because it is exactly +one sample out of the 1000-widest slice the p99 looks at. The corrected +`co_p99_ns` is **16,930 nanoseconds** — nearly 200x the raw figure — because +that one 56 microsecond stall backfilled thousands of synthetic queued +samples into the tail, and enough of them landed above the 99th-percentile +cutoff to move it by two orders of magnitude. Both numbers came from the +*same* 100,000-iteration run; the only difference is whether the harness +accounts for the requests coordinated omission would otherwise omit. Reporting +the raw number alone — which is what most naive benchmark loops do — is not +wrong arithmetic, it is measuring the wrong thing. + +## HDR-style percentiles, and why the mean lies + +None of the numbers above are an arithmetic mean, and that is deliberate. +Latency distributions are heavy-tailed: the vast majority of calls finish in +tens of nanoseconds, and a small minority — the ones a scheduler, a page +fault, or (in a GC'd runtime) a collection pause touches — finish orders of +magnitude slower. Average those together and the mean is dragged toward the +tail by exactly the outliers a reader most needs to see reported accurately, +while simultaneously hiding *how bad* the tail actually gets: a mean of 200 ns +is consistent with "every call took 200 ns" and with "999 calls took 40 ns and +one took 160,200 ns," and those are utterly different systems to operate. A +**percentile** answers a sharper question — "what latency do 99% of calls beat?" +— and doesn't blend the tail into the body. `benchlab`'s `percentile()` +function is the simplest correct version of this idea, nearest-rank over a +fully sorted sample array (`sorted[floor(p * (n-1))]`); a production HdrHistogram +would instead bucket values logarithmically to report percentiles in +constant memory over unbounded sample counts, which is the one simplification +worth naming here — this harness sorts a real in-memory array, which is fine +at these iteration counts and would need revisiting past tens of millions of +samples. + +## Microbench pitfalls: the compiler deletes what you don't watch + +The final trap has nothing to do with clocks: it's that a sufficiently smart +compiler will happily prove your benchmark's result is never used and delete +the code that produces it, leaving you an accurate measurement of an empty +loop. Every `run_once` call in `benchlab` folds its result into a running +`checksum` for exactly this reason — the accumulation is a real, observable +side effect (it's printed at the end), so nothing upstream of it can be +proven dead. To see the failure mode this guards against, compile a stripped +version of the same idea and read what GCC actually produced: + +```console +[host]$ g++ -std=c++23 -O2 -c dce.cpp -o dce.o && objdump -d --no-show-raw-insn dce.o +0000000000000000 : + 0: ret + 1: nopl 0x0(%rax) + 5: data16 cs nopw 0x0(%rax,%rax,1) + +0000000000000010 : + 10: mov %edi,%r9d + 13: test %edi,%edi + ... +``` + +`bench_discarded` runs a checksum function over the workload bytes `iters` +times and never uses the result — GCC 16.1.1 at `-O2` proved the whole loop +has no observable effect and compiled it to a bare `ret`. Nothing ran. A +benchmark harness with this shape reports a real, reproducible, extremely +fast time for code that never executed even once. `bench_sunk` is the +identical loop with one change — the result feeds a `volatile` global — and +the disassembly shows the real loop, vectorized with SSE, doing actual work +on every iteration. `benchlab`'s checksum plays exactly the role of that +`volatile` sink, and it earns its keep twice: it stops the optimizer from +proving the loop dead, and — because the workload is fixed — it doubles as a +correctness oracle. All three languages produce the identical checksum for +the identical fixed workload (`--op encode --iters 20000 --warmup 1000`, this +session): + +```console +[host]$ ./cpp/build/release/app --op encode --iters 20000 --warmup 1000 | tail -1 +benchlab: checksum=00000000001f6710 +[host]$ ./go/bin/app --op encode --iters 20000 --warmup 1000 | tail -1 +benchlab: checksum=00000000001f6710 +[host]$ ./rust/target/release/app --op encode --iters 20000 --warmup 1000 | tail -1 +benchlab: checksum=00000000001f6710 +``` + +A diverging checksum across languages would mean a codec bug, not +measurement noise — the ns timings are expected to vary run to run, the +checksum is not. + +Real benchmark frameworks solve the same dead-code problem with a dedicated +API rather than a manual accumulator: Google Benchmark's +`benchmark::DoNotOptimize()`, Rust's `criterion`, and Go's `testing.B` (whose +`b.N`-driven loop the compiler cannot fully see through, plus `go test +-bench . -benchmem` and the companion `benchstat` tool for comparing two +runs statistically rather than eyeballing them) all exist to answer exactly +this question at scale, across many benchmarked functions, with proper +statistical comparison between runs. `benchlab` deliberately uses none of +them: it is a from-scratch harness precisely so the mechanism — warmup, +per-iteration timing, coordinated-omission correction, the dead-code trap — +is visible in ~150 lines you can read end to end, identically shaped in all +three languages, rather than hidden inside a framework's own reporting +format. + +## How the code works + +The percentile math and the coordinated-omission correction are the two +functions worth reading closely; everything else in `main` is argument +parsing and the two loops already described. `co_correct` takes the raw, +unsorted sample array, the expected interval, and a hard cap, and returns a +new array with synthetic samples backfilled behind every stall: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +std::vector co_correct(const std::vector& raw, + std::int64_t expected_interval_ns, + std::size_t cap) { + std::vector out; + out.reserve(raw.size()); + for (std::int64_t v : raw) { + out.push_back(v); + if (expected_interval_ns <= 0 || v <= expected_interval_ns) continue; + std::int64_t missing = v - expected_interval_ns; + while (missing >= expected_interval_ns && out.size() < cap) { + out.push_back(missing); + missing -= expected_interval_ns; + } + if (out.size() >= cap) break; + } + return out; +} +``` + +```go +func coCorrect(raw []int64, expectedIntervalNs int64, cap int) []int64 { + out := make([]int64, 0, len(raw)) + for _, v := range raw { + out = append(out, v) + if expectedIntervalNs <= 0 || v <= expectedIntervalNs { + continue + } + missing := v - expectedIntervalNs + for missing >= expectedIntervalNs { + if len(out) >= cap { + return out + } + out = append(out, missing) + missing -= expectedIntervalNs + } + } + return out +} +``` + +```rust +fn co_correct(raw: &[i64], expected_interval_ns: i64, cap: usize) -> Vec { + let mut out = Vec::with_capacity(raw.len()); + 'outer: for &v in raw { + out.push(v); + if expected_interval_ns <= 0 || v <= expected_interval_ns { + continue; + } + let mut missing = v - expected_interval_ns; + while missing >= expected_interval_ns { + if out.len() >= cap { + break 'outer; + } + out.push(missing); + missing -= expected_interval_ns; + } + } + out +} +``` + +Every sample `v` is pushed to `out` unconditionally first — a normal, +non-stalling iteration contributes exactly itself, which is why `co_n >= n` +always and equals `n` on a run with no stalls at all. The `if` guard skips +correction entirely for samples at or below the interval — those are the +*fast* calls, nothing queued behind them. Once `v` exceeds the interval, the +`while`/`for` loop walks backward from `v - E` in steps of `E`, pushing one +synthetic sample per step: this is precisely "how many requests, each +spaced `E` apart, would have piled up during a stall of length `v`?" The cap +check inside the loop (not just at the top) matters because a single +pathological stall can demand millions of synthetic samples in one pass — +without the inner check, that one sample could blow the cap on its own +before the outer loop ever gets to see it. C++ mutates a `std::vector` under +manual `reserve`/`push_back`; Go uses `append` against a capacity-preallocated +slice and a `for`-with-condition in place of `while`; Rust needs a labeled +`'outer` loop because breaking the outer `for` from inside the inner `while` +has no unlabeled spelling — three idioms, the same three-branch algorithm, +provably matching because `verify.lua` runs all three against the identical +fixed workload and checks the identical invariants (`co_p99_ns >= p99_ns`, +`co_n >= n`) rather than the identical source text. + +`percentile()` is the other half: given a *sorted* array and a fraction `p`, +nearest-rank indexing (`sorted[floor(p * (len - 1))]`) picks the sample at +that rank directly — no interpolation, no bucketing, correct as long as the +array really is sorted first, which `main` guarantees by calling `sort` +(C++'s `std::sort`, Go's `sort.Slice`, Rust's `sort_unstable`) immediately +before computing `median_ns`, `p99_ns`, and again after `co_correct` before +computing `co_p99_ns`. The warmup loop and the timed loop share the same +`run_once` call; the only difference between them is that the warmup loop +discards the per-call scalar into `checksum` without ever starting a clock +around it, and the timed loop wraps each call between two monotonic-clock +reads (`std::chrono::steady_clock`, `time.Now()`'s monotonic reading, +`std::time::Instant`) and appends the delta to `raw` before moving to the +next iteration. + +## Errors, three ways + +`benchlab` has two failure surfaces, and they map to two different exit +codes across all three languages. A **usage error** — no `--op` given for a +real run, an unrecognized `--op` value, or `--warmup >= --iters` (which would +leave zero samples to time) — prints the two-line usage banner to stderr and +exits **2**; C++ funnels every parse path through `usage_and_exit()`, Go +through `usage()` plus `os.Exit(2)`, Rust through a `usage() -> !` that calls +`std::process::exit(2)` and therefore never returns, which the type checker +enforces. A **codec bug** — `decode_frame` rejecting a frame from a workload +this harness controls and never corrupts — is exit **1**, and it is the one +path `verify.lua` never exercises, because it should be unreachable on this +fixed input; it exists so a future change to the workload or the codec fails +loudly rather than silently producing a bogus checksum. C++ checks the +`std::expected` returned by `decode_frame` and calls `std::exit(1)` on +`std::unexpected`; Go's `decodeFrame` wraps the sentinel errors +(`errShortHeader`, `errBadMagic`, `errBadVersion`, `errLengthMismatch`) with +`%w` so `runOnce` can wrap them again with call-site context and the caller +still `errors.Is`-matches the original; Rust's `decode_frame` returns +`Result<(u8, Vec), String>` and `run_once` propagates it with a bare `?`, +letting `main`'s `match` turn any `Err` into `ExitCode::from(1)`. A successful +run of either mode — the real path or `--lie` — exits **0**. + +## Concurrency lens + +`benchlab` itself is deliberately single-threaded — running the timed loop on +one thread is what makes "iteration N+1 starts after N returns" a clean, +analyzable closed loop in the first place; multiply the callers and you'd +need to reason about contention on top of coordinated omission, a strictly +harder problem this chapter isn't taking on. But the *host* underneath a +single-threaded benchmark is never single-threaded, and that's where the +stall in this chapter's own `co_p99_ns=16930` sample almost certainly came +from: the Linux scheduler is free to migrate the benchmark's one thread +between any of this host's 16 logical CPUs between iterations, and a +migration that lands it on a core the kernel had to first bring back up +from a low C-state, or evicts its L1/L2 working set entirely, produces +exactly the multi-microsecond single-iteration stall this chapter measures +and corrects for. Chapter 26's `taskset -c 0,2` pinning is the mitigation: +running `taskset -c 4 ./cpp/build/release/app ...` confines the benchmark +to one core for the whole run, trading the ability to migrate away from a +noisy neighbor for the elimination of migration-induced stalls as a source +of tail latency — worth doing whenever you want the tail to reflect the code +under test rather than the scheduler's decisions about it. + +## Build, run, observe + +```bash +[host]$ cd examples/39-benchmarking-without-lies && ./demo.sh build +``` + +```console +[host]$ ./cpp/build/release/app --op roundtrip --iters 100000 --warmup 5000 +benchlab: op=roundtrip iters=100000 warmup=5000 +benchlab: n=100000 min_ns=39 median_ns=47 p99_ns=73 max_ns=5480 +benchlab: co_p99_ns=75 expected_interval_ns=47 co_n=100428 +benchlab: checksum=000000000079c3e0 +[host]$ ./cpp/build/release/app --lie --op roundtrip +benchlab: lie op=roundtrip (no warmup, single wall-clock sample, ignores variance) +benchlab: lie elapsed_ns=1077 sink=76 +``` + +The real run and the lie run measure the identical codec against the +identical fixed workload — the same `alice\0the quick brown fox jumps over +the lazy dog, three times, for benchlab` `DELIVER` frame — and produce +answers that look nothing alike: a five-line distribution with a correction +term, versus a single number with no way to tell whether it's typical or a +fluke. Neither `p99_ns=` nor `median_ns=` ever appears in `--lie`'s output; +`verify.lua` asserts that absence directly, because it's the entire point of +running `--lie` at all. + +The runner drives all three languages and asserts the shared contract: + +```console +[host]$ python3 scripts/test-all-examples.py --only 39-benchmarking-without-lies +building 3 example-lang combinations (jobs=1)... + build 39-benchmarking-without-lies [cpp]: ok + build 39-benchmarking-without-lies [go]: ok + build 39-benchmarking-without-lies [rust]: ok + +verifying... + verify 39-benchmarking-without-lies [cpp]: PASS + verify 39-benchmarking-without-lies [go]: PASS + verify 39-benchmarking-without-lies [rust]: PASS + +example cpp go rust +39-benchmarking-without-lies PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: perf counts the work, the corrected percentiles count the stalls + +Two independent signals confirm two different claims. First, `perf stat` +counts CPU work directly, which a wall-clock number alone cannot separate +from scheduling noise: + +```console +[host]$ perf stat -e cycles,instructions,task-clock \ + ./cpp/build/release/app --op roundtrip --iters 100000 --warmup 5000 + 41,029,644 cycles:u + 95,724,894 instructions:u + 12.03 msec task-clock:u + 0.012523557 seconds time elapsed +``` + +95,724,894 instructions over 41,029,644 cycles is an IPC of **~2.33** for +105,000 total roundtrip calls (5,000 warmup plus 100,000 timed) — a plausible, +stable figure for a branch-light encode/decode over a ~78-byte frame, and a +number that doesn't move with scheduler noise the way `max_ns` does, because +`cycles`/`instructions` count retired work, not wall time. If a change to the +codec doubled `cycles:u` for the same `instructions:u`, that would be cache or +branch-misprediction cost growing, not timing variance — a distinction the ns +percentiles alone can't make. + +Second, the corrected-versus-naive percentiles from that same invocation are +the other half of the cross-check, and they're worth setting side by side +explicitly: + +| | raw (naive) | coordinated-omission-corrected | +|---|---|---| +| p99 | 86 ns | 16,930 ns | +| sample count | 100,000 | 107,077 | + +Same run, same 100,000 timed iterations, same single 56,595 ns stall visible +in `max_ns` — one number treats that stall as a footnote, the other treats it +as the ~197x-worse tail a real fixed-rate caller would have actually felt. +Neither number is "wrong" in isolation; reporting only the left column while +implying it describes what users experience is the lie this chapter's title +refers to. + +## What you learned + +- A closed-loop benchmark (iteration N+1 waits for N) records a stall as one + sample when a real fixed-rate caller would have queued several requests + behind it — Gil Tene's coordinated omission — and the fix is to backfill + synthetic samples from the raw value down to the expected interval, which + is exactly what turned this chapter's `86 ns` raw p99 into a `16,930 ns` + corrected one from the identical run. +- Percentiles, not the mean, are how you report a heavy-tailed latency + distribution: nearest-rank indexing over a sorted sample array is the + simplest correct version, and it's what real HdrHistogram bucketing + approximates at scale. +- A benchmark whose result nothing uses is a benchmark the compiler is free + to delete — GCC turned an unwatched loop calling a real function `iters` + times into a bare `ret`, confirmed by reading the actual `objdump` output; + folding every call's result into a printed checksum (`benchlab`'s own + correctness oracle, identical across all three languages) is what keeps + the loop alive, playing the same role as `benchmark::DoNotOptimize()`, + criterion's black-boxing, or a `testing.B` sink. +- The CPU frequency governor is part of the experiment, not incidental to + it: `/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` reporting + `powersave` with turbo boost active means the clock itself can move + between samples, and pinning to `performance` before benchmarking removes + that confound from the tail you're trying to measure. + +Chapter 40 takes the discipline this chapter earned — real percentiles, a +pinned clock, a tail you trust — and spends it: stripping a hot path down to +pinned cores, zero allocation, and the low-latency techniques that make the +corrected p99 small in the first place. + +--- + +

verified — on the Fedora 44 +reference host (kernel 7.1.3-200.fc44, 11th-gen Intel i7-11800H, 16 CPUs, +intel_pstate/powersave governor, turbo active) this +session: python3 scripts/test-all-examples.py --only +39-benchmarking-without-lies printed PASS PASS PASS (3 +passed, 0 failed, 0 skipped; verify.lua 42 checks total across +cpp/go/rust). The cited real run printed +min_ns=39 median_ns=47 p99_ns=73 max_ns=5480 co_p99_ns=75; the +perf stat run printed 41,029,644 cycles:u, +95,724,894 instructions:u, and (that specific run) +p99_ns=86 co_p99_ns=16930 max_ns=56595 — the side-by-side +percentile table above is that exact run's output, not a constructed example. +--lie printed elapsed_ns=1077 sink=76 with no +p99_ns=/median_ns= anywhere in its output. The +dead-code-elimination excerpt (bench_discarded compiling to a +bare ret, bench_sunk compiling to a real vectorized +loop) came from a throwaway file compiled ad hoc with g++ -std=c++23 +-O2 (GCC 16.1.1) this session, not from the shipped example. The +checksum 00000000001f6710 for --op encode --iters 20000 +--warmup 1000 was confirmed identical across all three built binaries +this session. Not exercised: pinning the governor to performance +(this host's numbers are all under the default powersave policy, +deliberately, as the CPU-frequency section explains) and any run against a +lab VM — this is a host-only, local-mode example with no VM/LGTM +requirement.

diff --git a/_docs/40-low-latency-fastpath.md b/_docs/40-low-latency-fastpath.md new file mode 100644 index 0000000..7576abe --- /dev/null +++ b/_docs/40-low-latency-fastpath.md @@ -0,0 +1,559 @@ +--- +title: "The low-latency fast path: pin a core, reuse one buffer, and stop asking the scheduler for permission" +order: 40 +part: "Performance and Low Latency" +description: "chatterd-fastpath strips the ch21 echo hot path down to a pinned core, one reused 64-byte buffer, and an optional busy-poll spin, then proves each move at the kernel level — Cpus_allowed_list and voluntary_ctxt_switches read from /proc//status, not the program's own claim. The three languages reach identical behavior by three different shutdown mechanisms, and Go's CPU count lies about its container through a taskset mask rather than a cgroup." +duration: "50 minutes" +--- + +Chapter 39 taught you to trust a latency number: warm the code, time each +iteration, correct for coordinated omission, and report the tail instead of the +mean. This chapter spends that discipline. If you now believe a `p99` you +measured, the next question is how to make it *small* — and the answer, on Linux, +is a short list of moves that each remove one source of the multi-microsecond +stalls Chapter 39's `co_p99` was busy accounting for. `chatterd-fastpath` is the +same echo hot path from Chapter 21's `chatterd`, stripped to exactly those moves: +pin the process to one CPU, reuse a single stack buffer for every message so the +loop never allocates, and — optionally — spin on a non-blocking `recv` instead of +blocking, trading a whole core for the elimination of the sleep/wake round trip +through the scheduler. A `naive` server in the same binary does the opposite of +each, so you can race them under identical placement and watch the gap. + +The code is in `examples/40-low-latency-fastpath/`. It is a **VM example**: the +measurements only mean something on a quiet, pinned machine, so `verify.lua` +stages the binary and `run-latency-bench.sh` to the lab guest and drives them +there. + +{% include excalidraw.html + file="40-fastpath-datapath" + alt="Two bands showing one echo round trip each. Top, the naive server: a client frame arrives at a server that may run on any of the online CPUs; per message it calls make_unique to allocate a fresh 64-byte heap buffer, then a blocking read(2) that puts the thread to sleep — an arrow labeled voluntary context switch loops out through a box labeled kernel scheduler and back, waking the thread when data arrives — then write(2) echoes and the heap buffer is freed. A counter reads voluntary_ctxt_switches roughly one per message. Bottom, the fastpath server: pinned by sched_setaffinity to a single CPU shown boxed and shaded, one reused 64-byte stack buffer for the whole connection, and a busy-poll loop that spins on non-blocking recv(2) over EAGAIN, never entering the scheduler — no context-switch arrow, the thread stays on its core. A counter reads voluntary_ctxt_switches roughly zero. A note reads same 64-byte frame, same round trip — the difference is who the thread has to ask for its CPU back." + caption="Figure 40.1 — one echo round trip, naive versus fastpath: a per-message heap buffer and a blocking read that sleeps through the scheduler, against one reused buffer on a pinned core that never gives the CPU up" %} + +> **Tools used** — `taskset` (systems-target VM), `ssh`/`scp` (host, the lab's +> own transport). The pinning and busy-poll proofs read `/proc//status` on +> the guest directly — procfs, no extra tool. `taskset` ships in +> `util-linux-core` on the Fedora 44 guest; the lab transport is the same +> `scripts/lab/` tooling every VM example uses. + +## Three moves that shrink the tail + +Every technique in `fastpath` targets a specific line item in a latency budget, +and each is worthless — or actively harmful — applied blindly. Naming them up +front: + +- **Pin to one CPU** (`sched_setaffinity(2)`). A thread the scheduler is free to + migrate can be moved, between one message and the next, onto a core that first + has to be brought back up from a low C-state and whose caches hold none of your + working set. That migration *is* a tail-latency event. Pinning trades the + ability to flee a noisy neighbor for the removal of migration as a source of + jitter. It is the right trade for a dedicated latency-critical process and the + wrong one for a general-purpose worker pool. +- **Reuse one buffer** (zero heap allocation in the loop). `fastpath` reads into a + single 64-byte stack array for the entire life of a connection; `naive` + `make_unique`s a fresh heap buffer for every message. At 64 bytes the allocator + cost is small, but it is *variable* — most calls hit the thread-local cache and + return in nanoseconds, and the occasional one that has to grab the arena lock or + fault in a fresh page does not, and that variance lands in the tail. +- **Busy-poll** (spin on non-blocking `recv`). A blocking `read(2)` on an idle + socket puts the thread to sleep; when the next frame arrives, the kernel has to + wake it, which is a scheduler round trip — a **voluntary context switch** — on + every single message. Busy-poll never sleeps: it spins on a non-blocking `recv` + that returns `EAGAIN` until data is there, burning 100% of the pinned core to + keep the thread on-CPU and ready. This is the most expensive move (a whole core, + gone) and the one with the largest single effect on median latency, as the + numbers below show. + +The rest of the program is deliberately ordinary: a fixed 64-byte wire frame +(magic `CF`, a version and type byte, a big-endian `seq`, and zero padding), a +`measure` subcommand that runs `N` synchronous ping-pong round trips against +either server and prints the same percentile table Chapter 39 built, and a +`run-latency-bench.sh` driver that races the two under identical `taskset` +placement so the *only* variable is the in-server discipline. + +## Pinning, and the count that lies about its container + +`pin_to_cpu` is four lines of real work — validate the CPU index, build a +one-CPU affinity mask, install it — wrapped around one decision that turns out to +diverge across the three languages more sharply than anything else in the +program: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +Result pin_to_cpu(int cpu) { + long nproc = ::sysconf(_SC_NPROCESSORS_ONLN); + if (cpu < 0 || (nproc > 0 && cpu >= nproc)) { + return std::unexpected( + std::format("cpu {} out of range (0..{})", cpu, nproc > 0 ? nproc - 1 : 0)); + } + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (::sched_setaffinity(0, sizeof(set), &set) != 0) { + return std::unexpected(std::format("sched_setaffinity({}): {}", cpu, errno_text())); + } + return {}; +} +``` + +```go +func pinToCPU(cpu int) error { + nproc := onlineCPUs() + if cpu < 0 || (nproc > 0 && cpu >= nproc) { + last := 0 + if nproc > 0 { + last = nproc - 1 + } + return fmt.Errorf("cpu %d out of range (0..%d)", cpu, last) + } + + // One P, one hot thread: without this the runtime is free to schedule the + // loop's goroutine onto another M running on another CPU. + runtime.GOMAXPROCS(1) + runtime.LockOSThread() + + var set unix.CPUSet + set.Zero() + set.Set(cpu) + if err := unix.SchedSetaffinity(0, &set); err != nil { + return fmt.Errorf("sched_setaffinity(%d): %v", cpu, err) + } + if err := unix.SchedSetaffinity(os.Getpid(), &set); err != nil { + return fmt.Errorf("sched_setaffinity(%d): %v", cpu, err) + } + return nil +} +``` + +```rust +fn pin_to_cpu(cpu: i32) -> StepResult<()> { + // sysconf(_SC_NPROCESSORS_ONLN) reports CPUs online SYSTEM-WIDE and ignores + // this process's affinity mask — the same call the C++ port uses. That + // matters under the bench driver's `taskset -c 1`: a mask-aware count + // (like Go's runtime.NumCPU) would report 1 and wrongly reject a pin to + // CPU 1. sysconf sidesteps that. + let nproc = unsafe { libc::sysconf(libc::_SC_NPROCESSORS_ONLN) }; + if cpu < 0 || (nproc > 0 && i64::from(cpu) >= nproc) { + let last = if nproc > 0 { nproc - 1 } else { 0 }; + return Err(format!("cpu {cpu} out of range (0..{last})")); + } + let mut set = CpuSet::new(); + set.set(cpu as usize) + .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; + // Pid 0 == the calling thread, which is the main (and only) thread, whose + // tid == pid — so this is also what /proc//status reports. + sched_setaffinity(Pid::from_raw(0), &set) + .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; + Ok(()) +} +``` + +The range check needs a CPU count, and the correct count is the number of CPUs +online **system-wide**, because the argument is an absolute CPU index. C++ and +Rust both ask `sysconf(_SC_NPROCESSORS_ONLN)`, which returns exactly that (2 on +the lab guest) and is blind to the caller's affinity mask. The obvious Go +equivalent, `runtime.NumCPU()`, is *not* the same thing: it returns the size of +the process's **CPU affinity mask**. Those two numbers agree on an idle machine +and diverge the instant something narrows the mask — which is precisely what the +benchmark driver does. Phase B of `run-latency-bench.sh` launches each server +under `taskset -c 1`, so the fastpath process starts life with a one-CPU mask, +`runtime.NumCPU()` returns **1**, and the Go server's own range check rejects the +`--pin 1` it was told to use: + +``` +app: error: cpu 1 out of range (0..0) +``` + +This is the same failure mode as `hardware_concurrency()` reporting a container's +cgroup quota instead of the host's core count — the running theme of this part of +the book — reached through an unexpected door. Here the count lies not because a +cgroup capped it but because a `taskset` affinity mask narrowed it, and the fix +is the same in spirit: ask the kernel the question you actually mean. +`onlineCPUs()` in the Go port reads `/sys/devices/system/cpu/online` and counts +the ranges there, matching the `sysconf` semantics the other two languages get +for free: + +```go +func onlineCPUs() int { + data, err := os.ReadFile("/sys/devices/system/cpu/online") + if err != nil { + return runtime.NumCPU() + } + // ... parse the "0-1" / "0,2-3" range list, falling back to NumCPU on any + // parse failure ... +} +``` + +The Go port also carries two lines the others don't need. `sched_setaffinity(2)` +acts on a **thread**, and a goroutine is not a thread — the runtime multiplexes +goroutines over OS threads (Ms) and will migrate the one running your loop at +will. `runtime.LockOSThread()` nails the hot loop's goroutine to one M and +`GOMAXPROCS(1)` stops the runtime from parking worker threads on other cores; only +then does pinning that thread mean anything. And the affinity call is made +*twice* — once with `0` (the calling thread, which is what actually restricts the +loop) and once with `os.Getpid()` (the thread-group leader, which is the mask +`/proc//status` reports and the benchmark reads back). C++ and Rust are +single-threaded here, so one `sched_setaffinity(0, …)` both pins the loop and +shows up in procfs; there is no leader-versus-worker gap to close. + +## Busy-poll: trading a whole core for the scheduler round trip + +The second codetab is the fastpath's defining loop. In busy-poll mode the +connection socket is set `O_NONBLOCK`, and this is the read: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +bool read_full_busy(int fd, void* buf, std::size_t n) { + auto* p = static_cast(buf); + std::size_t off = 0; + while (off < n) { + ssize_t r = ::recv(fd, p + off, n - off, 0); + if (r < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + if (g_stop_for_busy_read && g_stop_for_busy_read->load(std::memory_order_relaxed)) { + return false; + } + continue; // spin + } + return false; + } + if (r == 0) { + return false; // peer closed + } + off += static_cast(r); + } + return true; +} +``` + +```go +func readFullBusy(fd int, buf []byte, s *stopper) bool { + off := 0 + for off < len(buf) { + n, err := unix.Read(fd, buf[off:]) + if err != nil { + if errors.Is(err, unix.EAGAIN) || errors.Is(err, unix.EWOULDBLOCK) { + if s.stop.Load() { + return false + } + continue // spin + } + if errors.Is(err, unix.EINTR) { + continue + } + return false + } + if n == 0 { + return false // peer closed + } + off += n + } + return true +} +``` + +```rust +fn read_full_busy(fd: RawFd, buf: &mut [u8]) -> bool { + let mut off = 0; + while off < buf.len() { + match recv(fd, &mut buf[off..], MsgFlags::empty()) { + Ok(0) => return false, // peer closed + Ok(n) => off += n, + Err(Errno::EAGAIN) => { + if stop_requested() { + return false; + } + // spin + } + Err(_) => return false, + } + } + true +} +``` + +There is nothing in the `EAGAIN` branch but a stop-flag check and `continue`. No +`sched_yield`, no `nanosleep`, no `poll` — those would all, in different ways, +hand the CPU back to the kernel, which is the exact cost busy-poll exists to +avoid. The thread stays runnable and on its core, re-issuing `recv` as fast as the +CPU will let it, so the instant a frame lands the very next `recv` returns it with +no wakeup latency at all. The blocking path (`read_full_blocking`, not shown) is +the ordinary alternative: a plain blocking `read` that the kernel parks on until +data arrives. + +The difference is not subtle, and it does not require a profiler to see — it is +counted for you in `/proc//status`. Every time a blocking `read` sleeps and +is woken, the kernel records a **voluntary context switch** against the process. +Running a 20,000-message `measure` against each server, both `taskset`-pinned to +CPU 1, and reading `voluntary_ctxt_switches` before and after: + +```console +[vm]$ # naive (blocking read) +before: voluntary_ctxt_switches: 2 +after: voluntary_ctxt_switches: 20500 +[vm]$ # fastpath (busy-poll) +before: voluntary_ctxt_switches: 2 +after: voluntary_ctxt_switches: 3 +``` + +The blocking server took **one voluntary context switch per message** — it slept +and was woken 20,498 times to serve 20,000 requests plus warmup. The busy-poll +server slept **once** for the whole run. That single number is the mechanism the +median latency reflects: on this run the naive `p50` was 11,426 ns and the +fastpath `p50` was 8,148 ns, and the ~3,300 ns the fastpath saved is, to a first +approximation, the sleep/wake round trip through the scheduler that it declined to +take 20,000 times. + +## How the code works + +Both servers share one accept skeleton: a `poll(2)` on the listener with a 200 ms +timeout so the stop flag is checked a few times a second even when idle, then +`accept4`, `TCP_NODELAY`, and a per-connection echo loop that serves one client to +completion before returning to `accept`. The whole program runs on **one thread** +by design — the entire point is that a latency-critical hot loop should not be +handing work between threads, so there is no thread pool and no per-connection +goroutine or task. The fastpath loop reads into the one 64-byte buffer and writes +it straight back; the naive loop `make_unique`s (C++), `make([]byte, …)`s (Go), or +`vec![0u8; …]`s (Rust) a fresh buffer every iteration and lets it fall out of +scope, the deliberate pessimization the benchmark is built to expose. + +`measure` is the client half, and its percentile math is exactly Chapter 39's: +sort the samples, then nearest-rank index with `idx = ceil(p/100 · n)` clamped to +`[1, n]`. It builds each frame with `seq = i`, times the round trip between two +`CLOCK_MONOTONIC` reads, and — a correctness check, not a timing one — verifies +the echo came back with the same `seq` before recording the sample, so a +cross-wired or corrupted connection fails loudly rather than reporting fast +garbage. Warmup round trips (500 by default in the benchmark) are discarded before +recording starts, for the reasons Chapter 39 spent a section on. + +## Errors, three ways + +`fastpath` has two exit codes for two kinds of failure, and they mean different +things. A **usage error** — an unknown subcommand, a missing `--pin`, a `measure` +with no `--n`, a port outside 1–65535 — prints the three-line usage banner to +stderr and exits **2**. Argument parsing is strict full-string unsigned, so +`--n 5x` is a usage error, not a silent truncation to 5. + +A **runtime error** is exit **1**, and it carries an `app: error: …` message +naming what went wrong: `cpu 99 out of range (0..1)` from `pin_to_cpu`, +`connect 127.0.0.1:1: Connection refused` when `measure` can't reach a server, +`notanip: not an IPv4 address` when the target host isn't a dotted quad (the +servers use `inet_pton`, not a DNS lookup, so a hostname is an error by design), +and `measure: connection closed at iteration N` if a server dies mid-run. The +three languages carry these three ways — C++ returns `std::expected` and prints the `error()` string; Go returns `error` values with the +`strerror` text unwrapped so the message reads like C's rather than Go's layered +`dial tcp4 …: connect: …`; Rust returns `Result` with the message +pre-formatted at the failure site — but the bytes on stderr are identical across +all three on the same guest, because `verify.lua` asserts exactly that. + +## Concurrency lens + +The interesting concurrency in this chapter is not in the hot loop — that is +single-threaded on purpose — it is in **shutdown**, and it is the one place where +identical observable behavior demanded three genuinely different mechanisms. + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +extern "C" void on_signal(int) { g_stop.store(true, std::memory_order_relaxed); } + +void install_signal_handlers() { + struct sigaction sa{}; + sa.sa_handler = on_signal; + sa.sa_flags = 0; // deliberately NOT SA_RESTART + ::sigemptyset(&sa.sa_mask); + ::sigaction(SIGINT, &sa, nullptr); + ::sigaction(SIGTERM, &sa, nullptr); +} +``` + +```go +func (s *stopper) trigger() { + s.stop.Store(true) + s.mu.Lock() + defer s.mu.Unlock() + if s.ln != nil { + _ = s.ln.Close() + } + if s.conn != nil { + _ = s.conn.Close() + } + if s.rawFD >= 0 { + // SHUT_RDWR makes an in-flight recv(2) return 0 immediately; the fd + // itself is closed by the loop that owns it. + _ = unix.Shutdown(s.rawFD, unix.SHUT_RDWR) + } +} +``` + +```rust +fn install_signal_handlers() { + let sa = SigAction::new( + SigHandler::Handler(on_signal), + SaFlags::empty(), // deliberately NOT SA_RESTART + SigSet::empty(), + ); + // SAFETY: on_signal only performs a relaxed atomic store, which is + // async-signal-safe. + unsafe { + let _ = sigaction(Signal::SIGINT, &sa); + let _ = sigaction(Signal::SIGTERM, &sa); + } +} +``` + +C++ and Rust are showing the *same* function and the same idea: install a signal +handler that does nothing but set an atomic stop flag, with `sa_flags` explicitly +**not** `SA_RESTART`. That flag is the whole trick. Without `SA_RESTART`, a +blocking `read(2)` or `poll(2)` already in progress when `SIGINT` arrives does not +resume — it returns `EINTR`, the loop unwinds, sees the stop flag, and exits. The +signal handler and the blocked syscall cooperate through the kernel's default +`EINTR` behavior, and no second thread is involved. Rust can copy the C++ design +line for line because Rust's standard library makes real blocking syscalls; the +one wrinkle is that std's higher-level wrappers (`TcpListener::accept`, +`Read::read_exact`) retry `EINTR` internally, so the servers are built on raw file +descriptors via `nix` where every `EINTR` is handled explicitly. + +Go cannot use this technique at all, which is why its tab shows a completely +different function. The Go runtime installs its own signal handlers with +`SA_RESTART`, and — more fundamentally — a "blocking" read on a `net.Conn` is not +sitting in a blocking syscall to begin with. The runtime's network poller parks +the goroutine on an `epoll` registration and lets the OS thread go do other work; +no signal will make that parked read return `EINTR`, because there is no syscall +in flight to interrupt. So Go reaches identical behavior — `SIGINT`, then a clean +`app: … shutting down` and exit 0 — by a different route entirely: the +`signal.Notify` goroutine sets the stop flag and then **closes the listener and +the live connection**. Closing an fd is what wakes a poller-parked read; it is the +`shutdown(2)`-driven wakeup Chapter 21's `chatterd` used, and it is the idiomatic +Go answer to "interrupt a blocked I/O wait." The busy-poll path needs one extra +touch — its socket is a dup'd raw fd outside the poller — so `trigger` also calls +`shutdown(SHUT_RDWR)` on it to break the spin. Three mechanisms — non-`SA_RESTART` +`EINTR`, non-`SA_RESTART` `EINTR` again, and close-the-fd — for one observable +outcome, which `verify.lua` pins down with a bounded 10-second wait so a server +that *fails* to shut down surfaces as a clear failure rather than a hung run. + +## Build, run, observe + +```bash +[host]$ cd examples/40-low-latency-fastpath && ./demo.sh build +``` + +The `run-latency-bench.sh` driver does the real work. Phase A brings up a naive +and a fastpath server with no external pinning and reads each one's +`Cpus_allowed_list` from `/proc//status` — this is the pinning proof, taken +from the kernel rather than from anything the program prints about itself. Phase B +then races the two under identical `taskset` placement (both servers pinned to the +same CPU, the client to the other), running three interleaved trials of each so a +lone scheduling stall on this shared nested-KVM guest can't decide the outcome: + +```console +[vm]$ bash run-latency-bench.sh ./app 20000 500 +app: naive-cpus-allowed 0-1 +app: fastpath-cpus-allowed 1 +app: percentiles_ns tag=naive-1 p50=11957 p90=15325 p99=28509 p99.9=80747 min=10075 max=442827 mean=12975.38 n=20000 +app: percentiles_ns tag=fastpath-1 p50=9282 p90=11225 p99=16626 p99.9=47039 min=6900 max=87199 mean=9607.65 n=20000 +app: percentiles_ns tag=naive-2 p50=11135 ... n=20000 +app: percentiles_ns tag=fastpath-2 p50=7936 ... n=20000 +app: percentiles_ns tag=naive-3 p50=11320 ... n=20000 +app: percentiles_ns tag=fastpath-3 p50=8184 ... n=20000 +``` + +`naive-cpus-allowed` is `0-1`, the full online set — the naive server never +touches its affinity mask. `fastpath-cpus-allowed` is `1`, a single CPU, because +`sched_setaffinity` narrowed it. Median of the three naive `p50`s is 11,320 ns; +median of the three fastpath `p50`s is 8,184 ns — the fast path is about 1.38× +faster on this run, and the margin held across every language and every capture +this session (1.38×–1.62×), never inverted, never marginal. The runner drives all +three languages against the same `verify.lua`: + +```console +[host]$ python3 scripts/test-all-examples.py --only 40-low-latency-fastpath --mode vm +verifying... + verify 40-low-latency-fastpath [cpp]: PASS + verify 40-low-latency-fastpath [go]: PASS + verify 40-low-latency-fastpath [rust]: PASS + +example cpp go rust +40-low-latency-fastpath PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: two kernel-level signals, neither the program's own word + +The chapter's two central claims are checked by two independent numbers, both read +from `/proc//status` on the guest — not from anything the server prints about +itself, which is the discipline that makes them proofs rather than assertions: + +| claim | procfs field | naive | fastpath | +|---|---|---|---| +| the fast path is pinned | `Cpus_allowed_list` | `0-1` (full set) | `1` (one CPU) | +| busy-poll never sleeps | `voluntary_ctxt_switches`, 20k-message run | 2 → 20500 | 2 → 3 | + +The first row proves the pinning move happened at the kernel level: if +`sched_setaffinity` had silently failed, `fastpath-cpus-allowed` would read `0-1` +like the naive server, and the benchmark would be measuring two unpinned processes +fighting over both cores. The second row proves the busy-poll move: a server that +claimed to busy-poll but was secretly blocking would show ~20,000 voluntary +context switches like the naive one. Both numbers come from the kernel's own +accounting, and both are exactly what the mechanism predicts — which is why +`verify.lua` asserts the `Cpus_allowed_list` shape directly and why the +context-switch contrast, though not gated, is the single clearest picture of what +busy-poll buys and what it costs. + +## What you learned + +- **Pinning, zero allocation, and busy-poll each remove one specific source of + tail latency** — migration jitter, allocator variance, and the scheduler + sleep/wake round trip respectively — and each has a real cost (a core you can't + reclaim, most of all for busy-poll), so they are tools for a dedicated + latency-critical path, not defaults. +- **Busy-poll's whole effect is visible as a context-switch count**: a blocking + server takes one voluntary context switch per message (20,500 over this run), a + busy-poll server takes essentially none (3), and the median-latency gap is that + scheduler round trip, declined 20,000 times. +- **Prove a low-latency claim from the kernel, not the program.** + `Cpus_allowed_list` and `voluntary_ctxt_switches` in `/proc//status` are + the ground truth for "is it pinned" and "does it sleep"; a server's own printed + line is a claim, and the two are allowed to disagree when something is broken. +- **A CPU count lies about its container through more than one door.** Go's + `runtime.NumCPU()` reports the affinity-mask size, so under `taskset -c 1` it + returns 1 where `sysconf(_SC_NPROCESSORS_ONLN)` returns the host's 2 — the same + class of bug as `hardware_concurrency()` reading a cgroup quota, reached via an + affinity mask instead. For an absolute-CPU-index range check, the system-wide + online count is the one you mean. +- **Identical observable behavior can require different mechanisms per runtime.** + C++ and Rust unblock a shutdown with a non-`SA_RESTART` handler and `EINTR`; Go's + network poller means no syscall is in flight to interrupt, so it closes the fd + instead — same `SIGINT`, same exit 0, three different routes there. + +Chapter 41 takes everything from this part and the last — the USE checklist, the +LGTM pipeline, benchmarks you can trust, and this chapter's pinned, allocation-free +hot path — and assembles it into the capstone: a supervised, sandboxed, fully +observed two-VM fleet. + +--- + +

verified — on the +systems-target lab guest (Fedora 44, kernel +6.19.10-300.fc44, 2 vCPU nested KVM) this session: +python3 scripts/test-all-examples.py --only 40-low-latency-fastpath --mode +vm printed PASS PASS PASS (3 passed, 0 failed; +verify.lua reported PASS 46 / FAIL 0 for each of +cpp/go/rust). The pinning proof read naive-cpus-allowed 0-1 and +fastpath-cpus-allowed 1 from /proc/<pid>/status. +A 20,000-message measure against each server, both +taskset-pinned to CPU 1, moved voluntary_ctxt_switches +from 2 to 20500 for the blocking naive server and from 2 to 3 for the busy-poll +fastpath server, with p50 11426 ns (naive) versus 8148 ns +(fastpath) on that run. A full run-latency-bench.sh ./app 20000 500 +gave median p50 11320 ns (naive) versus 8184 ns (fastpath), a 1.38× +margin; per-language runner medians this session were 1.47× (cpp), 1.62× (go), +and 1.51× (rust), all clearing the verify.lua +median(fastpath p50) < median(naive p50) · 0.90 gate. Go's +runtime.NumCPU()-under-taskset range-check failure +(cpu 1 out of range (0..0)) was reproduced directly before the +onlineCPUs() fix and is gone after it. Not exercised: any run +outside the lab guest — this is a VM example whose numbers are only meaningful on +a quiet, pinned machine.

diff --git a/_plans/r09-batch1-golden-capture.md b/_plans/r09-batch1-golden-capture.md new file mode 100644 index 0000000..2547157 --- /dev/null +++ b/_plans/r09-batch1-golden-capture.md @@ -0,0 +1,782 @@ +--- +title: "r09 batch 1 — golden capture from the C++ reference" +layout: plan +render_with_liquid: false +published: false +--- + +# r09 batch 1 — golden capture from the C++ reference + +Captured on `systems-target` (the KVM lab guest) against the already-built +C++ reference binaries for `37-use-method-60s-analysis` and +`40-low-latency-fastpath`, per `_plans/r09-batch1-stub-completion.md` steps +S1 and S2. Staging used plain `scp`/`ssh` to `/tmp/lsp37-cpp/app` and +`/tmp/lsp40-cpp/` — never `demo.sh run` under `TARGET`, per the plan's +warning about `deploy-to-vm.sh`'s `ssh -t` CRLF and fixed `/home/fedora/app` +path. (That stray `/home/fedora/app` from a prior session was in fact found +on the guest during this capture — see the pitfall note below.) + +**Read this file's "Calibration findings" section first** — it contains a +critical divergence from the plan's behavioral contract that affects how +`verify.lua` must be written for example 37. + +**UPDATE (re-capture against commit `f2b2dba`):** the example-37 saturator +bug described below (all four `sat_*` functions exiting in milliseconds +regardless of `--seconds`) is **fixed** as of commit `f2b2dba` +(`fix(demo-37): saturators now honor --seconds instead of exiting +instantly`), which adds `std::this_thread::sleep_until(deadline)` inside each +`sat_*` pool scope so `~jthread`'s `request_stop()`-before-`join()` ordering +no longer kills workers on their first iteration. **The entire "S1 — +example 37" section below has been re-captured from scratch against the +rebuilt, fixed binary** (verified staged via `md5sum` match between the +host build and the guest copy, and via a live timing check — +`--resource cpu --seconds 4` took 4.002s wall on the guest, not +milliseconds — before any capture began). **All example-37 numbers in this +version supersede the pre-fix numbers; the pre-fix numbers were invalid and +must not be used to write `verify.lua`.** Example-40 sections (S2 golden +capture and Calibration finding 3) are untouched by this update — they were +never affected by the example-37 bug. + +## Guest facts + +``` +nproc: 2 +uname -r: 6.19.10-300.fc44.x86_64 +uname -a: Linux systems-target.lab 6.19.10-300.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 25 18:23:49 UTC 2026 x86_64 GNU/Linux +os-release: Fedora Linux 44 (Cloud Edition) +MemTotal: 3997020 kB (~3.9 GiB) +swap: ~3902 MiB +``` + +Guest was healthy before, during, and after all captures (`free -m`/`uptime` +checked repeatedly; no OOM-killer messages in `dmesg` at any point). + +--- + +## S1 — example 37 (`loadmix`) golden capture — **RE-CAPTURED post-`f2b2dba`** + +**This entire section is a fresh capture against the fixed binary, replacing +the pre-fix capture in full.** Host build directory +`examples/37-use-method-60s-analysis/cpp/build/release/app` was confirmed to +already reflect commit `f2b2dba` (working tree clean, `ninja` reported no +outstanding work); staged to `/tmp/lsp37-cpp/app` on the guest via `scp`, +overwriting a stale pre-fix copy left over from the earlier capture session, +`chmod 755`. `md5sum` matched byte-for-byte between host and guest +(`48ec7371b960552a39efc2fdbf658d72`) before any capture began. A pre-capture +sanity check confirmed the fix is live on the guest: `time app --resource +cpu --seconds 4` → `real 0m4.002s` (was ~0.007s pre-fix). + +### 1. No args — exit 2 + +``` +$ /tmp/lsp37-cpp/app +EXIT=2 +--STDOUT-- (empty) +--STDERR-- +usage: app --resource cpu|mem|io|net --seconds N + app analyze [--seconds N] +``` + +### 2. `analyze --seconds` (missing value) — exit 2 + +Identical usage text and exit code to #1. + +### 3. `--resource bogus --seconds 5` — exit 2 + +Identical usage text and exit code to #1. + +### 4. `--seconds 0` — exit 2 + +Identical usage text and exit code to #1. + +### 5. `analyze --bogus` — exit 2 + +Identical usage text and exit code to #1. + +All five exit-2 shapes are byte-identical on stderr; stdout is empty in every +case. Unchanged from the pre-fix capture, as expected — the fix only touches +the `sat_*` functions' internal duration handling, not argument parsing. + +### 6. `--resource cpu --seconds 5` — exit 0, timed — **now ~5s** + +``` +loadmix: start resource=cpu seconds=5 workers=6 +loadmix: done resource=cpu seconds=5 workers=6 ops=8801461550 bytes=11315899737308079569 + +real 0m5.002s +user 0m9.219s +sys 0m0.001s +``` + +`workers=6` matches the contract: `max(4, cpus*3)` with `cpus=2` → +`max(4,6)=6`. Wall time **5.002s** — matches `--seconds 5` almost exactly +(pre-fix: 0.007s regardless of `--seconds`). `ops` is now ~9 orders of +magnitude larger than the pre-fix capture (16896201 → 8801461550) since the +workers genuinely spin for the full duration instead of exiting on their +first loop check. + +### 7. `--resource io --seconds 5` — exit 0, timed — **now ~5s** + +``` +loadmix: start resource=io seconds=5 workers=8 +loadmix: done resource=io seconds=5 workers=8 ops=4131 bytes=67682304 + +real 0m5.012s +user 0m0.021s +sys 0m0.361s +``` + +`workers=8` matches `max(4, cpus*4)` = `max(4,8)=8`. `bytes = ops * 16384` +(4131 × 16384 = 67682304), consistent with the fixed 16 KiB file contract. +Wall time **5.012s** (pre-fix: 0.058s). + +### 8. `--resource net --seconds 5` — exit 0, timed — **now ~5s** + +``` +loadmix: start resource=net seconds=5 workers=8 +loadmix: done resource=net seconds=5 workers=8 ops=6587351 bytes=421590464 + +real 0m5.207s +user 0m0.915s +sys 0m8.315s +``` + +`workers=8` matches `max(4, cpus*4)=8`. `bytes = ops * 64` +(6587351 × 64 = 421590464). Wall time **5.207s** (pre-fix: 0.220s) — the +small overshoot past 5.0s (vs. cpu/io's tighter ~5.00–5.01s) is consistent +with `sat_net`'s two-pool structure (sender pool joins inside the `sleep_until` +scope, then a receiver thread is stopped and joined afterward, noticing +`stop` within ~200ms per the source comment). + +### 9. `analyze --seconds 10` (idle), wall-clock timed — exit 0 + +Wall time: **10.417s** (`time` builtin, `%R`) for `--seconds 10` — confirms +the five interval samplers (`vmstat`, `mpstat`, `iostat`, `sar`, `pidstat`, +each `1 10`) still run **concurrently**, not serially (serial would be +~50s). This capture ran immediately after captures 6–8 (back-to-back +saturator runs), so the baseline is slightly elevated versus a cold-guest +idle reading — noted here because capture 10 below is compared against +this baseline, not a cold one. + +``` +analyze: start seconds=10 cpus=2 +analyze: tool uptime raw="05:51:20 up 3:39, 1 user, load average: 1.78, 0.45, 0.15" +analyze: tool dmesg status=denied +analyze: tool top cpu="%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st" tasks="Tasks: 149 total, 2 running, 147 sleep, 0 d-sleep, 0 stopped, 0 zombie" +analyze: tool ss raw="Total: 167" +analyze: tool pidstat active_processes=4 +analyze: metric resource=cpu name=busy_pct value=1.34 unit=pct +analyze: metric resource=cpu name=run_queue value=3.00 unit=procs +analyze: metric resource=mem name=used_pct value=12.30 unit=pct +analyze: metric resource=mem name=swap_io value=0.00 unit=kbps +analyze: metric resource=io name=util_pct value=8.20 unit=pct +analyze: metric resource=io name=iowait_pct value=0.59 unit=pct +analyze: metric resource=io name=await_ms value=5.33 unit=ms +analyze: metric resource=net name=pkts value=0.00 unit=per_s +analyze: signal resource=cpu type=Utilization fired=false value=1.34 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=false value=3.00 threshold=4.00 +analyze: signal resource=cpu type=Errors fired=false +analyze: signal resource=mem type=Utilization fired=false value=12.30 threshold=75.00 +analyze: signal resource=mem type=Saturation fired=false value=0.00 threshold=300.00 +analyze: signal resource=mem type=Errors fired=false +analyze: signal resource=io type=Utilization fired=false value=8.20 threshold=40.00 +analyze: signal resource=io type=Saturation fired=false value=0.59 threshold=8.00 +analyze: signal resource=io type=Errors fired=false +analyze: signal resource=net type=Utilization fired=false value=0.00 threshold=2000.00 +analyze: signal resource=net type=Saturation fired=false value=0.00 threshold=8000.00 +analyze: signal resource=net type=Errors fired=false +analyze: verdict resource=cpu ratio=0.75 +analyze: done +``` + +Note `dmesg status=denied` — the `fedora` user is unprivileged and `dmesg` +returns `Operation not permitted` on this guest; that is the expected +`denied` path, not a capture error. + +### 10. Compound load-attribution run — **now WORKS, run 3× for stability** + +`nohup /tmp/lsp37-cpp/app --resource cpu --seconds 25 >/tmp/bgN.out 2>/tmp/bgN.err & sleep 2; /tmp/lsp37-cpp/app analyze --seconds 12` + +This is the run S3's load-attribution assertion depends on, and it was +previously impossible to satisfy (see the superseded pre-fix Calibration +finding). Run 3 times back-to-back; all three attempts show strong, +unambiguous attribution: + +| attempt | busy_pct | run_queue | Utilization fired | Saturation fired | verdict ratio | +|---|---|---|---|---|---| +| 1 | 100.00 | 8.00 | true | true | 2.00 | +| 2 | 100.00 | 8.00 | true | true | 2.00 | +| 3 | 100.00 | 7.00 | true | true | 1.75 | + +Full output, attempt 1: + +``` +analyze: start seconds=12 cpus=2 +analyze: tool uptime raw="05:51:41 up 3:39, 1 user, load average: 1.82, 0.56, 0.20" +analyze: tool dmesg status=denied +analyze: tool top cpu="%Cpu(s): 97.8 us, 2.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st" tasks="Tasks: 150 total, 1 running, 149 sleep, 0 d-sleep, 0 stopped, 0 zombie" +analyze: tool ss raw="Total: 167" +analyze: tool pidstat active_processes=3 +analyze: metric resource=cpu name=busy_pct value=100.00 unit=pct +analyze: metric resource=cpu name=run_queue value=8.00 unit=procs +analyze: metric resource=mem name=used_pct value=12.35 unit=pct +analyze: metric resource=mem name=swap_io value=0.00 unit=kbps +analyze: metric resource=io name=util_pct value=0.00 unit=pct +analyze: metric resource=io name=iowait_pct value=0.00 unit=pct +analyze: metric resource=io name=await_ms value=0.00 unit=ms +analyze: metric resource=net name=pkts value=0.00 unit=per_s +analyze: signal resource=cpu type=Utilization fired=true value=100.00 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=true value=8.00 threshold=4.00 +analyze: signal resource=cpu type=Errors fired=false +analyze: signal resource=mem type=Utilization fired=false value=12.35 threshold=75.00 +analyze: signal resource=mem type=Saturation fired=false value=0.00 threshold=300.00 +analyze: signal resource=mem type=Errors fired=false +analyze: signal resource=io type=Utilization fired=false value=0.00 threshold=40.00 +analyze: signal resource=io type=Saturation fired=false value=0.00 threshold=8.00 +analyze: signal resource=io type=Errors fired=false +analyze: signal resource=net type=Utilization fired=false value=0.00 threshold=2000.00 +analyze: signal resource=net type=Saturation fired=false value=0.00 threshold=8000.00 +analyze: signal resource=net type=Errors fired=false +analyze: verdict resource=cpu ratio=2.00 +analyze: done +ANALYZE_EXIT=0 +BG_EXIT=0 + +--BG-STDOUT-- +loadmix: start resource=cpu seconds=25 workers=6 +loadmix: done resource=cpu seconds=25 workers=6 ops=44164896060 bytes=5514586990755454969 +``` + +Attempts 2 and 3 (metric/signal/verdict lines only — the rest of the output +shape was identical to attempt 1 in both): + +``` +# attempt 2 +analyze: metric resource=cpu name=busy_pct value=100.00 unit=pct +analyze: metric resource=cpu name=run_queue value=8.00 unit=procs +analyze: signal resource=cpu type=Utilization fired=true value=100.00 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=true value=8.00 threshold=4.00 +analyze: verdict resource=cpu ratio=2.00 +ANALYZE_EXIT=0 +BG_EXIT=0 +loadmix: start resource=cpu seconds=25 workers=6 +loadmix: done resource=cpu seconds=25 workers=6 ops=44886125692 bytes=17660584183443754701 + +# attempt 3 +analyze: metric resource=cpu name=busy_pct value=100.00 unit=pct +analyze: metric resource=cpu name=run_queue value=7.00 unit=procs +analyze: signal resource=cpu type=Utilization fired=true value=100.00 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=true value=7.00 threshold=4.00 +analyze: verdict resource=cpu ratio=1.75 +ANALYZE_EXIT=0 +BG_EXIT=0 +loadmix: start resource=cpu seconds=25 workers=6 +loadmix: done resource=cpu seconds=25 workers=6 ops=44575950456 bytes=8335093140041756969 +``` + +`busy_pct` rose from the idle baseline (1.34, capture 9) to **100.00** in +all three attempts — nowhere near the 60.00 threshold's margin. `run_queue` +rose from the idle baseline (3.00) to **7.00–8.00**, comfortably clearing +both proposed gate conditions from the plan (`busy_pct >= 60` and +`run_queue >= nproc+2 = 4`). **`Utilization fired=true` and `verdict +resource=cpu` fired identically in all three attempts — STABLE, no +flakiness observed.** This directly reverses the pre-fix finding (which +showed the background job invisible to `analyze` because it had already +exited in milliseconds); the background `loadmix` process is now still +running and genuinely saturating both vCPUs for the entire 12-second +`analyze` window (and beyond — it was launched for `--seconds 25`). + +### 11. `--resource mem` — re-tested under real sustained load (run last) + +**This is the highest-risk item in this re-capture.** Pre-fix, `mem` only +ran for ~2.3–2.8s regardless of `--seconds` (the same missing-wait bug, see +superseded finding below) and so never actually sustained 1.35×MemTotal of +touched memory for long — it was never a real OOM test. Post-fix, `mem` now +holds `workers=2` threads each touching a `target_bytes/2 ≈ 2635 MiB` chunk +for the genuine requested duration, so this needed to be re-verified as an +actual sustained-memory-pressure test, not just an argument-parsing check. + +Guest health immediately before (after a fresh `free -m`/`uptime` check, +following the cpu/io/net/analyze captures above): + +``` +Mem: total=3903MiB used=485MiB free=3399MiB available=3418MiB +Swap: total=3902MiB used=15MiB +load average: 3.96, 1.63, 0.61 (elevated from the immediately preceding cpu saturation runs) +``` + +**Test A** — `time /tmp/lsp37-cpp/app --resource mem --seconds 2`, with a +0.5s-interval `free -m` monitor running concurrently: + +``` +loadmix: start resource=mem seconds=2 workers=2 +loadmix: done resource=mem seconds=2 workers=2 ops=0 bytes=5525480448 + +real 0m2.415s +user 0m0.546s +sys 0m2.268s +``` + +Memory trace during the run (used / total, swap): + +``` +t+0.0s: 3058MiB used / 3903MiB total, avail=844MiB, swap=15MiB +t+0.6s: 3764MiB used / 3903MiB total, avail=138MiB, swap=490MiB +t+1.1s: 3765MiB used / 3903MiB total, avail=137MiB, swap=1236MiB +t+1.7s: 3764MiB used / 3903MiB total, avail=138MiB, swap=1933MiB +t+2.2s (post-exit): 463MiB used / 3903MiB total, avail=3439MiB, swap=19MiB +``` + +Elapsed **2.415s** — now honors `--seconds 2` closely (pre-fix: 2.342s, but +that was coincidental — see below). Memory pressure genuinely climbed +(swap 15→1933 MiB, available RAM dropped to 137 MiB) then fully released +within ~0.2s of process exit, back to 19 MiB swap. Guest survived cleanly; +`ops=0` (no full touch pass completed in 2s once the process's own +allocation overhead is accounted for) — this remains a legitimate value per +the pre-fix note, unchanged. + +**Test B** — `time /tmp/lsp37-cpp/app --resource mem --seconds 8`, same +monitoring: + +``` +loadmix: start resource=mem seconds=8 workers=2 +loadmix: done resource=mem seconds=8 workers=2 ops=2697992 bytes=5525480448 + +real 0m12.624s +user 0m1.243s +sys 0m11.252s +``` + +Memory trace (20 samples over the run, condensed): used climbed to +~3700–3790 MiB within the first second and *stayed there* for the entire +run (unlike Test A's brief spike, this is now a **sustained** plateau, as +expected for a genuinely duration-honoring saturator); swap stabilized +around 1920–2010 MiB for the last ~9 seconds of the run, available RAM +stayed pinned around 110–190 MiB the whole time. Post-exit: memory and swap +returned to baseline (`used=434MiB`, `swap=27MiB`) within ~1s. + +**New finding — duration overrun under swap pressure:** elapsed was +**12.624s for a requested `--seconds 8`**, a ~4.6s overrun, not the ~5.0–5.2s +tight match seen for cpu/io/net. Root cause (inferred from source): the +outer `sleep_until(deadline)` in the parent thread still fires at the +correct wall-clock time and `request_stop()` propagates immediately after, +but each worker's inner `for (size_t off = 0; off < buf.size(); off += +4096)` touch loop has **no stop-check inside it** — only the outer `while +(!stop_requested() && now < deadline)` is checked between passes. Once +pages have been pushed to swap, a single inner pass over a ~2635 MiB chunk +at 4 KiB stride means hundreds of thousands of potential page-fault/swap-in +round trips, so the *in-flight* pass at the moment `request_stop()` fires +can take several extra seconds to finish before `join()` returns and the +process can exit. This is a distinct, more precise finding than the pre-fix +report (which saw no duration-honoring behavior at all for `mem`); post-fix, +`mem` **does** honor `--seconds` as a floor but can overrun it by several +seconds under real swap pressure, scaling with how deep into swap the run +gets. + +Guest health immediately after Test B — checked thoroughly, including an +explicit OOM search across `dmesg` and `journalctl -k`: + +``` +Mem: total=3903MiB used=434MiB free=3513MiB available=3469MiB +Swap: total=3902MiB used=27MiB +load average: 2.55, 1.57, 0.64 (settling; a follow-up check ~90s later showed 0.86) + +dmesg -T | grep -iE "oom-kill|out of memory|killed process" → NONE FOUND +journalctl -k --since "-5 min" | grep -iE "oom|killed process" → NONE FOUND +``` + +No leftover `app` processes; no leftover `/var/tmp/loadmix-io-*` directories +(the `io` saturator's own cleanup ran correctly in captures 7/6). SSH +remained responsive throughout every test. + +**Verdict: `mem` remains SAFE on this guest, now confirmed under real, +sustained 1.35×MemTotal memory pressure for the full requested duration — +not just the brief allocation spike the pre-fix binary produced.** No +OOM-kill at any point across two tests. `workers=2` = `max(2, cpus)`. +`bytes=5525480448` = `target = MemTotal_kB(3997020) * 1024 * 1.35` exactly, +unchanged from pre-fix (this value was never affected by the bug). `ops` is +now genuinely non-zero at `--seconds 8` (2697992, an order of magnitude +above the pre-fix capture's 674498, consistent with real sustained +touching rather than a bug-truncated partial pass) and legitimately zero at +`--seconds 2` (allocation alone consumes most of the budget). **`verify.lua` +may assert exit 0, the `workers=2` value, and the `bytes=` target for `mem` +as before, but must still not assert a tight wall-clock bound** — the +overrun under swap pressure is real and can vary with guest load; a +generous upper bound (e.g. requested seconds × 3, or a fixed ceiling like +30s) would be safer than assuming near-exact duration matching for `mem` +specifically, unlike cpu/io/net which now track `--seconds` tightly (within +~0.2s). + +One incidental discovery while investigating clock behavior on this guest, +carried forward from the pre-fix capture (still relevant, unrelated to +either bug): `dmesg` (captured via `sudo`, since the app's own unprivileged +capture is `denied`) shows the kernel marking the guest's TSC unstable and +falling back to `kvm-clock`: + +``` +[ 222.788514] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable... +[ 222.793981] tsc: Marking TSC unstable due to clocksource watchdog +``` + +This is a nested-KVM artifact and does not affect `steady_clock`-based +deadline computation — confirmed again by this recapture's timings all +tracking their requested `--seconds` correctly (cpu/io/net within ~0.2s; +`analyze --seconds 10` at 10.417s). + +--- + +## S2 — example 40 (`chatterd-fastpath`) golden capture + +`app` and `run-latency-bench.sh` staged to `/tmp/lsp40-cpp/`, both +`chmod 755`. + +### 1. Usage / exit-2 shapes — all identical, all exit 2 + +``` +usage: app fastpath --port P --pin CPU [--busy-poll] + app naive --port P + app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] +``` + +Confirmed byte-identical for: no args; `frobnicate`; `fastpath --port 9999` +(no `--pin`); `naive` (no port); `measure --target 127.0.0.1:9999` (no +`--n`). Stdout empty in every case. + +### 2. `measure --target 127.0.0.1:1 --n 10` — exit 1 + +``` +app: error: connect 127.0.0.1:1: Connection refused +``` + +### 3. `measure --target notanip:9999 --n 10` — exit 1 + +``` +app: error: notanip: not an IPv4 address +``` + +### 4. `fastpath --port 19510 --pin 99` — exit 1 + +``` +app: error: cpu 99 out of range (0..1) +``` + +(`0..1` because `nproc=2` on this guest → `sysconf(_SC_NPROCESSORS_ONLN)-1=1`.) + +### 5. Full `bash run-latency-bench.sh /tmp/lsp40-cpp/app 20000 500` — 3 runs, all exit 0 + +**Run 1:** + +``` +app: nproc=2 pin_cpu=1 client_cpu=0 +app: naive-cpus-allowed 0-1 +app: fastpath-cpus-allowed 1 +app: === naive measure trial 1/3 === +app: percentiles_ns tag=naive-1 p50=13319 p90=19388 p99=30516 p99.9=80617 min=9714 max=117205 mean=14639.83 n=20000 +app: === fastpath measure trial 1/3 === +app: percentiles_ns tag=fastpath-1 p50=7495 p90=9464 p99=17882 p99.9=56822 min=6190 max=187663 mean=8218.93 n=20000 +app: === naive measure trial 2/3 === +app: percentiles_ns tag=naive-2 p50=11005 p90=14279 p99=27274 p99.9=68718 min=9450 max=358138 mean=12009.51 n=20000 +app: === fastpath measure trial 2/3 === +app: percentiles_ns tag=fastpath-2 p50=7745 p90=10172 p99=19281 p99.9=59792 min=5654 max=317138 mean=8450.89 n=20000 +app: === naive measure trial 3/3 === +app: percentiles_ns tag=naive-3 p50=11373 p90=15496 p99=30172 p99.9=75192 min=9405 max=270060 mean=12534.06 n=20000 +app: === fastpath measure trial 3/3 === +app: percentiles_ns tag=fastpath-3 p50=7922 p90=10403 p99=20461 p99.9=55899 min=6496 max=85982 mean=8606.73 n=20000 +EXIT=0 +``` + +median(naive p50) = 11373, median(fastpath p50) = 7745, ratio = **1.468** + +**Run 2:** + +``` +app: nproc=2 pin_cpu=1 client_cpu=0 +app: naive-cpus-allowed 0-1 +app: fastpath-cpus-allowed 1 +naive p50s: 11236, 10684, 11375 → median 11236 +fastpath p50s: 7553, 8172, 7240 → median 7553 +EXIT=0 +``` + +ratio = **1.488** + +**Run 3:** + +``` +app: nproc=2 pin_cpu=1 client_cpu=0 +app: naive-cpus-allowed 0-1 +app: fastpath-cpus-allowed 1 +naive p50s: 10666, 10631, 10655 → median 10655 +fastpath p50s: 7694, 7475, 7390 → median 7475 +EXIT=0 +``` + +ratio = **1.425** + +In every run: `naive-cpus-allowed` = `0-1` (the full online set — matches +contract, naive's affinity mask is left untouched); `fastpath-cpus-allowed` += `1` (exactly `PIN_CPU`, a single value — matches contract). Every +`percentiles_ns` line has `n=20000` and satisfies +`min <= p50 <= p90 <= p99 <= p99.9 <= max`. + +### 6. SIGINT shutdown of `naive --port 19601` — exit 0 + +``` +app: naive listening on 0.0.0.0:19601 +app: naive shutting down +``` + +### 7. SIGINT shutdown of `fastpath --port 19602 --pin 0 --busy-poll` — exit 0 + +``` +app: fastpath listening on 0.0.0.0:19602 pinned-cpu=0 busy-poll=on +app: fastpath shutting down +``` + +### Bonus (not explicitly requested, captured for completeness — the full plan file also mentions SIGTERM) + +SIGTERM shutdown of `naive --port 19603` — exit 0, same shutdown line. +SIGTERM shutdown of `fastpath --port 19604 --pin 1` (no `--busy-poll`) — +exit 0: + +``` +app: fastpath listening on 0.0.0.0:19604 pinned-cpu=1 busy-poll=off +app: fastpath shutting down +``` + +--- + +## Calibration findings + +### 1. RESOLVED (was CRITICAL) — example 37 saturate mode now honors `--seconds` + +**Status as of the `f2b2dba` re-capture: FIXED.** cpu/io/net saturators now +run for very close to the requested `--seconds` (within ~0.2s, see S1 +captures 6–8 above), and the load-attribution compound run (S1 capture 10) +now succeeds — `busy_pct` rises from an idle baseline of ~1.3% to 100.00%, +`run_queue` from ~3 to 7–8, `Utilization fired=true`, and `verdict +resource=cpu` all fire, **stably across 3 independent attempts**. `mem` now +genuinely sustains 1.35×MemTotal memory pressure for the full duration +rather than exiting during allocation, though it can *overrun* its +requested duration by several seconds under real swap pressure (see finding +2, updated below) — a new, narrower nuance replacing the old "duration not +honored at all" finding. + +**Practical effect on S3:** the plan's load-attribution assertion (S3 step +6 / acceptance criterion 5) **can now be written against the actual +`--resource cpu` saturator as originally intended** — no need for the +`yes`-style external-load workaround or for skipping/weakening the +assertion for `cpu`. The cross-language-parity concern raised below (Go/Rust +honoring the contract while C++ didn't) no longer applies: all three +languages can now be held to the same "saturate for N seconds, and +`analyze` sees it" standard, since the C++ reference itself finally does. + +**Recommended gate, per the plan's own hedge:** retry up to 3 attempts, +gating on `busy_pct >= 60` **or** `run_queue >= nproc+2` (`>= 4` on this +guest) — both conditions were cleared by an enormous margin in every +attempt (100.00 vs. 60 threshold; 7–8 vs. 4 threshold), so this gate has +comfortable headroom against a noisier CI run, not a tautology. + +The original (superseded) root-cause writeup is preserved below for +context — it explains *why* the bug existed and is exactly what commit +`f2b2dba` fixed (adding `std::this_thread::sleep_until(deadline)` inside +each `sat_*` pool scope, so `~jthread`'s `request_stop()`-before-`join()` +ordering becomes the intended post-deadline signal instead of an immediate +kill). + +--- + +**SUPERSEDED — pre-fix analysis, kept for context:** + +**This was a real bug in the pre-fix C++ reference binary, reproducible +independent of this guest's environment, confirmed by reading +`examples/37-use-method-60s-analysis/cpp/src/main.cpp`.** It is not a clock +or nested-KVM artifact. + +Empirical evidence — foreground, wall-clock timed: + +| invocation | requested seconds | actual elapsed | +|---|---|---| +| `--resource cpu --seconds 1` | 1 | 0.007s | +| `--resource cpu --seconds 3` | 3 | 0.008s | +| `--resource cpu --seconds 6` | 6 | 0.008s | +| `--resource cpu --seconds 20` | 20 | 0.006s | +| `--resource cpu --seconds 60` | 60 | 0.006s | +| `--resource io --seconds 6` | 6 | 0.058s | +| `--resource net --seconds 6` | 6 | 0.220s | +| `--resource mem --seconds 2` | 2 | 2.342s (dominated by allocation, see below) | +| `--resource mem --seconds 8` | 8 | 2.775s | + +`cpu`, `io`, and `net` all exit in well under a second regardless of the +requested duration — never anywhere near N seconds. + +**Root cause** (read from `sat_cpu`/`sat_mem`/`sat_io`/`sat_net`, lines +~297–457 of `main.cpp`): each function computes a `deadline`, spawns +`workers` `std::jthread`s into a local `std::vector pool`, and +the enclosing `{ }` block ends **immediately after the spawn loop** — there +is no `std::this_thread::sleep_until(deadline)` (or any wait) in the parent +thread before the block closes. When the block ends, `pool`'s destructor +runs, and `std::jthread`'s destructor calls `request_stop()` then `join()` +for each element **in sequence**: element 0 is stopped and joined to +completion before element 1 even receives its `request_stop()`. Since +`request_stop()` for the first worker fires within microseconds of thread +creation, and each worker's own loop condition is `!stop_requested() && now +< deadline` (stop-requested checked first, short-circuiting), the workers +exit almost immediately — bounded only by how long the *first* full loop +body takes (a handful of `2^20`-iteration batches for `cpu`, one syscall for +`io`/`net`), never by the actual `--seconds` value. The `deadline` variable +is effectively dead: it can only matter if the deadline is *shorter* than +the natural stop-propagation delay, which never happens with real durations. + +`mem`'s longer, roughly duration-independent ~2.3–2.8s wall time is a +*different* symptom of the exact same missing-wait bug: each worker's +`std::vector buf(chunk, 0)` (a ~2.6 GiB per-worker zero-fill, given +`workers=2` here) dominates the elapsed time, and the deadline/stop-request +race resolves *during or immediately after* that allocation rather than +after any real "touch until deadline" loop. At `--seconds=2` the whole +budget is consumed by allocation and `ops=0` (zero touches); at +`--seconds=8` one worker gets a partial pass in before being stopped +(`ops=674498` ≈ one full pass over a 2.6 GiB buffer at 4096-byte strides), +still nowhere near 8 seconds of sustained touching. + +**Isolation proof that this is a saturate-mode-only bug, not a parsing bug:** +a genuine sustained load (`yes >/dev/null &` ×4, which really does peg both +vCPUs) produces exactly the behavior the contract describes: + +``` +analyze: metric resource=cpu name=busy_pct value=100.00 unit=pct +analyze: metric resource=cpu name=run_queue value=6.00 unit=procs +analyze: signal resource=cpu type=Utilization fired=true value=100.00 threshold=60.00 +analyze: signal resource=cpu type=Saturation fired=true value=6.00 threshold=4.00 +analyze: verdict resource=cpu ratio=1.50 +``` + +So `analyze`'s subprocess orchestration, column-name parsing, and +threshold/verdict logic are all correct as specified. The bug is entirely +contained in `sat_cpu`/`sat_mem`/`sat_io`/`sat_net`'s missing wait. + +**Why this matters for S3 and beyond:** + +- The plan's own load-attribution assertion (S3 step 6 / acceptance + criterion 5: "under concurrent `--resource cpu` load, `busy_pct` and + `run_queue` rise materially above idle, `Utilization fired=true`, + `verdict resource=cpu`") **will fail against the current C++ binary** if + driven exactly as the plan describes (`nohup app --resource cpu --seconds + N &`, then `analyze`), because the background load is gone in + milliseconds — this capture's own compound run (#10 above) demonstrates + the failure directly. +- If Go and Rust are written to literally honor the documented contract + ("saturate that resource" for the given duration — the plausible reading + of the prose in the plan), they will *actually* run for N seconds. That + would make the load-attribution test **pass for Go/Rust but fail for + C++**, inverting the cross-language-parity goal the whole golden-capture + approach is built on, and breaking "S3 must pass with `LSP_LANG=cpp` + before a single line of Go or Rust exists." +- Recommendation: **this is very likely a bug that should be fixed in the + C++ source** (add `std::this_thread::sleep_until(deadline);` in the + parent thread of each `sat_*` function, before the `pool` block closes) + before S3 is written, so that the golden-capture oracle reflects intended, + correct, sustained-duration behavior. Absent a fix, `verify.lua`'s + load-attribution check cannot be written against this binary's actual + behavior without either (a) skipping/weakening it for `cpu` specifically, + contradicting acceptance criterion 5, or (b) using the `yes`-style + external-load workaround demonstrated above instead of the app's own + `--resource cpu` saturator — which changes the test's meaning from "prove + our own load generator is visible to our own analyzer" to "prove the + analyzer sees load, full stop," a materially weaker claim than the plan + intends. +- This does **not** affect the `worker=` count or `ops>0` assertions + (acceptance criterion 3): those still pass, since `ops` is always > 0 (a + partial batch still counts), and `workers` is computed and printed before + any of this executes. +- Wall-clock concurrency of the five `analyze` samplers (criterion 7 / plan + step 7) is unaffected and confirmed correct (10.37s for `--seconds 10`). + +### 2. `--resource mem` survivability — **RE-TESTED under real sustained load, still SAFE** + +**Updated as of the `f2b2dba` re-capture.** The pre-fix testing below never +actually exercised sustained memory pressure (the process exited during +allocation, in ~2.3–2.8s regardless of `--seconds`), so it was not a real +OOM test. Post-fix, `mem` genuinely holds ~2635 MiB/worker × 2 workers +(≈5271 MiB total, exceeding this guest's 3903 MiB RAM) touched for the +requested duration, pushing 1.9–2.0 GiB into swap and driving available RAM +down to ~110–190 MiB for the full run. **Re-tested at `--seconds 2` and +`--seconds 8` with continuous `free -m` monitoring and an explicit +post-run `dmesg`/`journalctl` OOM search: no OOM-kill at either duration, +guest remained fully SSH-responsive throughout, and memory/swap fully +recovered to baseline within ~1s of process exit.** `workers=2` and +`bytes=5525480448` still match the formula exactly (unaffected by the bug +or the fix). `ops` is now legitimately non-zero even at `--seconds 8` +(2697992, up from a bug-truncated 674498 pre-fix) and remains legitimately +0 at `--seconds 2` (allocation alone consumes the short budget). + +**New nuance the pre-fix capture could not see: `mem` can *overrun* its +requested `--seconds` by several seconds under real swap pressure** — Test B +requested 8s and took 12.624s wall, because the per-worker touch loop has no +stop-check *inside* its inner `for` loop, only between passes, and a single +pass over a swapped-out ~2635 MiB chunk at 4 KiB stride can take many +seconds once `request_stop()` fires mid-pass. **`verify.lua` should assert +exit 0, the `workers=2` value, and the `bytes=` target for `mem` as before, +but should use a generous upper bound on wall-clock time (e.g. requested +seconds × 3, or a fixed ceiling) rather than assuming near-exact duration +tracking** — unlike cpu/io/net, which now track `--seconds` tightly (within +~0.2s) and can reasonably use a tighter bound. Per the plan's own risk +mitigation, `mem` was again run last in S1, after everything else, and the +guest was confirmed stable both immediately after and on a delayed +follow-up check. + +**Superseded pre-fix testing (kept for context — these numbers predate the +fix and must not be used for `verify.lua` bounds):** exit 0 both times +tested, guest remained healthy (no OOM, swap barely touched: 10→17 MiB), +`workers=2` and `bytes=5525480448` matched the formula exactly. The +duration was not honored at all pre-fix (2.3–2.8s regardless of `--seconds +2` vs. `8`) for the different reason described in finding 1 above (the +missing wait resolved during/immediately after the initial buffer +allocation, before any real touch loop ran). + +### 3. Example 40 fastpath-vs-naive margin — stable, not marginal + +| run | naive median p50 (ns) | fastpath median p50 (ns) | ratio | +|---|---|---|---| +| 1 | 11373 | 7745 | 1.468 | +| 2 | 11236 | 7553 | 1.488 | +| 3 | 10655 | 7475 | 1.425 | + +Fastpath is consistently **~30–33% faster** (never inverted, never close) +across three independent full bench runs on this 2-vCPU nested-KVM guest. +This is a comfortable margin, not one requiring hedging. Recommend +calibrating the `verify.lua` threshold conservatively below the observed +floor — e.g. assert `median(fastpath p50) < median(naive p50) * 0.90` +(a 10% requirement against an observed ~30%+ margin) — to tolerate a noisier +CI run without becoming a tautology. Both `naive-cpus-allowed` (`0-1`, the +full online set) and `fastpath-cpus-allowed` (`1`, the single pinned CPU) +were exact and identical across all three runs. + +### 4. Sampler concurrency (example 37 `analyze`) — confirmed correct, re-confirmed post-fix + +`analyze --seconds 10` completed in **10.417s** wall-clock in the `f2b2dba` +re-capture (was 10.370s pre-fix — both consistent with the five interval +samplers, `vmstat`/`mpstat`/`iostat`/`sar`/`pidstat`, each run as ` +... 1 10`, running concurrently rather than serially; serial would be +~50s). `analyze` itself was never touched by the `sat_*` bug or its fix — +this timing is unaffected and re-confirmed rather than changed, since the +re-capture re-ran capture 9 fresh against the same guest for completeness. +No divergence here in either capture. + +### 5. Pitfall encountered and worked around (documented for S3/S4 authors) + +A stale `/home/fedora/app` binary (an old build of example 40, from a prior +`deploy-to-vm.sh`-driven session) was found already present on the guest +during this capture. It was never touched or removed — captures in this +file all used explicit `/tmp/lsp37-cpp/app` / `/tmp/lsp40-cpp/app` absolute +paths — but its presence is exactly the hazard the plan's staging design +(scp to `/tmp/lsp-/app` with plain `ssh`, never `demo.sh run` +under `TARGET`) is meant to avoid: a compound remote command that +constructs a `cd dir && cmd1 & sleep N; cmd2` (background one command with a +`cd` prefix, then a second command without repeating that `cd` or an +absolute path) will silently execute the second command from the SSH +session's default directory (`/home/fedora`) and can pick up this stale +binary instead of the one under test. `verify.lua` must always use full +absolute paths for every invocation in a compound remote command, never rely +on a leading `cd` to apply to later `;`-separated commands in the same +string. diff --git a/_plans/r09-batch1-stub-completion.md b/_plans/r09-batch1-stub-completion.md new file mode 100644 index 0000000..4b3b0b5 --- /dev/null +++ b/_plans/r09-batch1-stub-completion.md @@ -0,0 +1,682 @@ +--- +title: "r09 batch 1 — stub completion for examples 37 and 40" +layout: plan +render_with_liquid: false +published: false +--- + +# r09 batch 1 — stub completion for examples 37 and 40 + +Working plan for completing the Go, Rust, and `verify.lua` stubs in +`37-use-method-60s-analysis` and `40-low-latency-fastpath`. The C++ +implementation in each is the finished reference; Go and Rust conform to the +behavioral contract derived from it. + +Recorded here so the work survives an interrupted session. Checkpoint commit +after every step. + +## Approach + +Golden-capture first, `verify.lua` second, ports last. Run the already-built +C++ binaries on `systems-target`, capture exact stdout/stderr as the oracle, +write `verify.lua` against that oracle and prove it passes with `LSP_LANG=cpp` +before a single line of Go or Rust exists. Then port one language at a time, +gating each on the same unchanged `verify.lua`. Dependencies are chosen to be +satisfiable from the existing `~/.cargo` and `~/go/pkg/mod` caches so no build +needs the network. + +Rejected alternatives: + +- **Write all three languages then verify** — with `mode: vm` and one shared + guest, a failing assertion gives three suspects instead of zero. The + C++-passing gate makes `verify.lua` itself trustworthy first. +- **A new driver script for 37** mirroring 40's `run-latency-bench.sh` — 37 + needs one thing that script provides (a background load generator running + concurrently with `analyze`), and that fits in a single compound `ssh` + command, the shape example 33 already uses. +- **Driving 37/40 through `demo.sh run` under `TARGET`** — + `deploy-to-vm.sh` scp's to a fixed `/home/fedora/app` and uses `ssh -t`. Two + concurrent invocations race on that path, and the pty injects CRLF that + breaks Lua patterns. `verify.lua` stages to `/tmp/lsp37-/app` with + plain `ssh` instead. +- **The `core_affinity` crate** already declared in 40's `Cargo.toml` — not in + the local cargo cache, so it needs network. `nix 0.31.3` is cached and + provides `sched::sched_setaffinity` and `CpuSet`. + +## Behavioral contract — example 37 (`loadmix`) + +Program name in usage strings is literally `app`. All normal output on stdout; +usage on stderr. + +### CLI + +| Invocation | Behavior | +|---|---| +| `app --resource cpu\|mem\|io\|net --seconds N` | saturate; exit 0 | +| `app analyze [--seconds N]` | checklist; exit 0; `N` default 60 | +| anything else, unknown flag, missing value, non-integer or non-positive `--seconds`, bad or missing `--resource` | usage on stderr, exit 2 | + +Usage is exactly two lines on stderr: + +``` +usage: app --resource cpu|mem|io|net --seconds N + app analyze [--seconds N] +``` + +`analyze` is recognized only as `args[0]`. In saturate mode any token that is +not `--resource` or `--seconds` followed by a value is a usage error. + +### Saturate mode + +`cpus = hardware_concurrency()` (0 becomes 1). Worker count: + +- `mem` — `max(2, cpus)` +- `cpu` — `max(4, cpus * 3)` +- `io`, `net` — `max(4, cpus * 4)` + +Two stdout lines, in order: + +``` +loadmix: start resource={res} seconds={n} workers={w} +loadmix: done resource={res} seconds={n} workers={w} ops={u64} bytes={u64} +``` + +Per-resource semantics, with the meaning of `ops` and `bytes`: + +- **cpu** — `w` threads run xorshift64 until a `steady_clock` deadline, + checking the clock every 2^20 iterations. `ops` is total iterations; `bytes` + is the sum of each thread's final state word (a liveness checksum, not a byte + count). +- **mem** — reads `MemTotal:` kB from `/proc/meminfo`; + `target = MemTotal_kB * 1024 * 1.35` (fallback 2 GiB if unreadable); + `chunk = max(target/w, 65536)`; each worker allocates a `chunk`-byte buffer + and increments one byte every 4096 until the deadline. `ops` is total page + touches; `bytes` is `target`. +- **io** — `mkdir /var/tmp/loadmix-io-` (0700); each worker opens + `w.dat` with `O_WRONLY|O_CREAT|O_TRUNC|O_DSYNC` (0600) and `pwrite`s a + 16384-byte xorshift64\*-filled buffer at offset 0 repeatedly (the file stays + 16 KiB; the random fill defeats btrfs zstd compression); then `close`, + `unlink`, and `rmdir`. `ops` is write count; `bytes` is bytes written. +- **net** — one `AF_INET/SOCK_DGRAM` receiver bound to `127.0.0.1:0` with + `SO_RCVTIMEO` 200 ms; `w` sender threads `connect` to that port and `send` a + 64-byte zero payload until the deadline; the receiver drains until stopped. + `ops` is packets sent; `bytes` is bytes sent. + +### Analyze mode + +Exact line sequence on stdout: + +``` +analyze: start seconds={n} cpus={cpus} +analyze: tool uptime raw="{trimmed uptime output, or "unavailable"}" +analyze: tool dmesg status={ok|denied} +analyze: tool top cpu="{trimmed line starting %Cpu, else empty}" tasks="{trimmed line starting Tasks:, else empty}" +analyze: tool ss raw="{trimmed first line of `ss -s`}" +analyze: tool pidstat active_processes={count} +analyze: metric resource=cpu name=busy_pct value={:.2f} unit=pct +analyze: metric resource=cpu name=run_queue value={:.2f} unit=procs +analyze: metric resource=mem name=used_pct value={:.2f} unit=pct +analyze: metric resource=mem name=swap_io value={:.2f} unit=kbps +analyze: metric resource=io name=util_pct value={:.2f} unit=pct +analyze: metric resource=io name=iowait_pct value={:.2f} unit=pct +analyze: metric resource=io name=await_ms value={:.2f} unit=ms +analyze: metric resource=net name=pkts value={:.2f} unit=per_s +``` + +then, for each resource in the fixed order cpu, mem, io, net, three lines: + +``` +analyze: signal resource={r} type=Utilization fired={true|false} value={:.2f} threshold={:.2f} +analyze: signal resource={r} type=Saturation fired={true|false} value={:.2f} threshold={:.2f} +analyze: signal resource={r} type=Errors fired=false +``` + +then: + +``` +analyze: verdict resource={r} ratio={:.2f} +analyze: done +``` + +Timing and subprocess rules: + +- Every subprocess runs under `env LC_ALL=C LANG=C`, wrapped so `2>&1` + captures the whole pipeline: `( env LC_ALL=C LANG=C dmesg | tail -n 5 ) 2>&1`. +- `uptime`, `dmesg | tail -n 5`, `top -b -n1`, and `ss -s` are captured + **before** the interval samplers. +- The five interval samplers run **concurrently**, each redirected to its own + file under `/tmp/loadmix-analyze-/`, then all are waited on and the + directory removed. Wall clock is about N seconds, not 5N. Commands: + `vmstat 1 N`, `mpstat -P ALL 1 N`, `iostat -xz 1 N`, `sar -n DEV 1 N`, + `pidstat 1 N`. +- `free -m` is captured **after** the samplers. +- `dmesg` status is `denied` if the capture failed or the output contains + `Operation not permitted`. + +Parsing rules — all by column *name*, read from each tool's own header row, +never a fixed index: + +- **vmstat** — header is the token row containing `swpd`; data rows are those + with `len(tokens)+2 >= len(header)`; **skip data row 0** (the since-boot + average); `run_queue_max = max(r)`, `swap_io_max = max(si+so)`. +- **mpstat** — use the `Average:` / `CPU` header row and the `Average:` / `all` + data row, both with the first 2 tokens dropped; `busy_pct = 100 - %idle` + (`%idle` defaults to 100), `iowait_pct = %iowait` (defaults to 0). +- **free -m** — header row is the one containing `available`; the data row + starts with `Mem:` with 1 token dropped (header drops 0); + `used_pct = (total-available)/total*100`, or 0 if `total <= 0`. +- **iostat -xz** — block index increments on each `avg-cpu:` line; header is the + `Device` row; **skip block 0**; skip devices whose name starts with `zram`, + `sr`, `loop`, or `dm-`; drop 1 token from header and row; + `util_max = max(%util)`, `await_max = max(w_await)`. +- **sar -n DEV** — `Average:`/`IFACE` header and `Average:`/`lo` data row, 2 + tokens dropped each; value is `rxpck/s + txpck/s`. +- **pidstat** — `active_processes` is the count of lines whose first token is + `Average:` and whose second token is not `UID`. +- Any missing header or data row yields the documented default (0, or 100 for + `%idle`); a non-numeric field is treated as absent and falls back to the + default. + +Thresholds and verdict, all fixed: + +| resource | Utilization value / threshold | Saturation value / threshold | +|---|---|---| +| cpu | `busy_pct` / 60.0 | `run_queue_max` / `cpus + 2.0` | +| mem | `mem_used_pct` / 75.0 | `swap_io_max` / 300.0 | +| io | iostat `%util` max / 40.0 | mpstat `%iowait` / 8.0 | +| net | sar `lo` pkts/s / 2000.0 | same pkts/s / 8000.0 | + +`fired = value >= threshold`. The verdict is the resource with the greatest +`saturation_value / saturation_threshold`, scanned cpu → mem → io → net with a +strict `>` so ties keep the earlier resource; `ratio` is that maximum. `Errors` +is always `fired=false` by design — this example induces load, not +kernel-logged faults. + +> The C++ header comment on line 15 uses a word banned by `CLAUDE.md`. Do not +> carry it into the Go or Rust headers; write "accurate" or "a real checklist +> run" instead. + +## Behavioral contract — example 40 (`chatterd-fastpath`) + +Usage is three lines on stderr; exit 2 on any parse failure, unknown +subcommand, or empty argv: + +``` +usage: app fastpath --port P --pin CPU [--busy-poll] + app naive --port P + app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] +``` + +### Wire frame — fixed 64 bytes, nothing variable-length + +| offset | size | field | +|---|---|---| +| 0 | 1 | magic0 `0x43` (`'C'`) | +| 1 | 1 | magic1 `0x46` (`'F'`) | +| 2 | 1 | version `0x01` | +| 3 | 1 | type `0x01` (echo) | +| 4 | 8 | `seq`, big-endian u64 | +| 12 | 8 | `send_ns`, big-endian u64 — the client always writes 0 | +| 20 | 44 | zero padding | + +Servers never inspect `seq` or `send_ns`; they read exactly 64 bytes and write +the identical 64 bytes back. + +### `fastpath --port P --pin CPU [--busy-poll]` + +The order of operations is observable and matters: + +1. **Pin first**, before binding. Range check: + `cpu < 0 || cpu >= sysconf(_SC_NPROCESSORS_ONLN)` gives stderr + `app: error: cpu {c} out of range (0..{nproc-1})`, exit 1. + `sched_setaffinity` failure gives + `app: error: sched_setaffinity({c}): {strerror}`, exit 1. +2. Listen: `SO_REUSEADDR`, bind `INADDR_ANY:P`, backlog 16. Failures give + `app: error: socket: …`, `app: error: bind 0.0.0.0:{P}: …`, or + `app: error: listen: …`, exit 1. +3. Install non-restarting SIGINT and SIGTERM handlers — deliberately not + `SA_RESTART`. +4. stderr: + `app: fastpath listening on 0.0.0.0:{P} pinned-cpu={C} busy-poll={on|off}` +5. Accept loop while not stopped: `poll(listener, POLLIN, 200ms)`; `EINTR` + continues; a poll error gives `app: error: poll: {strerror}`, exit 1; a + timeout or no `POLLIN` continues. On accept, set `TCP_NODELAY`; with + `--busy-poll`, set the connection `O_NONBLOCK`. Accept `EINTR` or `EAGAIN` + continues; any other accept error gives `app: error: accept: {strerror}`, + exit 1. +6. Echo loop over **one** 64-byte stack buffer reused for the connection's whole + life — zero heap allocation in the loop. Blocking mode reads until 64 bytes; + any error, EOF, or EINTR breaks. Busy-poll mode spins on non-blocking `recv` + over `EAGAIN`/`EWOULDBLOCK` with no `sched_yield` or `nanosleep`, + re-checking the stop flag on each `EAGAIN`. Then write all 64 bytes back, + retrying on `EINTR`; on any other write error, give up on the connection. +7. On stop: stderr `app: fastpath shutting down`, exit 0. One connection is + served at a time, to completion. + +### `naive --port P` + +Identical to fastpath minus pinning and busy-poll, plus the deliberate +pessimization: **a fresh heap buffer allocated and freed per message**. +Announce line `app: naive listening on 0.0.0.0:{P}`; shutdown line +`app: naive shutting down`; exit 0. Its affinity mask is left untouched (all +online CPUs) — `run-latency-bench.sh` asserts this at the kernel level. + +### `measure --target HOST:PORT --n N [--warmup W] [--tag TAG]` + +- `HOST` is split at the **last** `:` and must be an IPv4 dotted quad; + otherwise stderr `app: error: {host}: not an IPv4 address`, exit 1. A + `connect` failure gives `app: error: connect {host}:{port}: {strerror}`, + exit 1. +- `--warmup` defaults to 200. `--n` is required and must be greater than 0; the + port must be 1..65535; all parsing is strict full-string unsigned, so + `--n 5x` is a usage error with exit 2. +- `TCP_NODELAY` on the client socket. +- stdout, first line: + `app: measure target={host}:{port} n={N} warmup={W}` +- `N + W` synchronous round trips, never more than one frame in flight. Per + iteration: build the frame with `seq = i` and `send_ns = 0`; take + `t_send` from `CLOCK_MONOTONIC`; write 64; blocking-read exactly 64; take + `t_recv`. A read failure gives stderr + `app: error: measure: connection closed at iteration {i}`, exit 1. A header + mismatch or an echoed `seq != i` gives + `app: error: measure: malformed/mismatched echo at iteration {i}`, exit 1. + Samples are recorded only when `i >= W`. +- Percentile over the ascending-sorted samples: `idx = ceil(p/100 * n)`, clamped + to `[1, n]`, returning `sorted[idx-1]` — 1-based nearest-rank, to be + reproduced exactly. +- Output tail, exact: + +``` +app: percentiles_ns tag={TAG or -} p50={} p90={} p99={} p99.9={} min={} max={} mean={:.2f} n={} +app: table + p50 {} ns + p90 {} ns + p99 {} ns + p99.9 {} ns + min {} ns + max {} ns + mean {:.2f} ns +``` + +Indentation is literal: two leading spaces, then the label padded so values +start at column 10 (`p50` + 4 spaces, `p90` + 4, `p99` + 4, `p99.9` + 2, +`min` + 4, `max` + 4, `mean` + 3). Exit 0. + +### `run-latency-bench.sh` + +Complete and language-agnostic — **do not rewrite it**. It takes +`/path/to/app [N=20000] [WARMUP=500]` and is staged onto the guest by +`verify.lua`, exactly as example 33 stages `run-sandbox-checks.sh`. It emits: + +``` +app: nproc={} pin_cpu={1|0} client_cpu={0} +app: naive-cpus-allowed {Cpus_allowed_list} # phase A, expect the full set +app: fastpath-cpus-allowed {Cpus_allowed_list} # phase A, expect exactly PIN_CPU +app: === naive measure trial k/3 === # phase B, k = 1..3, interleaved +app: === fastpath measure trial k/3 === +``` + +Ports 19501/19502 in phase A, 19503/19504 in phase B. Phase B externally +`taskset`s both servers to `PIN_CPU` and the client to `CLIENT_CPU`, so the +only variable is in-server discipline. Exit 0 on success; if a server never +comes up it prints `app: error: phase A/B servers did not come up` and exits 1. +Trials are interleaved and tagged `naive-k` / `fastpath-k` precisely so +`verify.lua` can group by tag and compare medians of three, never a single +sample. + +### Cross-language mechanism notes + +Behavior identical, mechanism idiomatic. + +- **Go 40** — raw fds via `golang.org/x/sys/unix` (already at v0.47.0 in + `go.mod`/`go.sum`) for socket, bind, listen, accept4, read, recv, write, and + setsockopt, so busy-poll bypasses the netpoller. Shutdown uses + `signal.Notify` plus a goroutine that sets an atomic and calls `shutdown(2)` + on the listener and the live connection fd — the ch21 pattern the C++ header + itself cites — because Go's runtime installs `SA_RESTART` handlers and you + cannot rely on EINTR. Pinning needs `runtime.LockOSThread()` in `main` **and** + `unix.SchedSetaffinity(tid, &set)` for every tid in `/proc/self/task`, since + new threads inherit the mask and phase A reads `/proc//status`, which + reports the thread-group leader. +- **Rust 40** — single-threaded; `nix 0.31` with features + `["signal", "sched", "poll"]` (cached locally) for `sigaction` without + `SA_RESTART`, `sched_setaffinity`/`CpuSet`, and `poll`. `std::net` for the + sockets, `set_nodelay` and `set_nonblocking` from std. + `ErrorKind::Interrupted` is the EINTR path. **Remove `core_affinity = "0.8"` + from `Cargo.toml`** — not in the local cargo cache, so it would force a + network fetch — and regenerate `Cargo.lock`, which is currently the stale + template lock naming `template-hello` and `rustix`. +- **Go 37** — `os/exec` for captures, `runtime.NumCPU()`, `unix.Open` plus + `unix.Pwrite` with `unix.O_DSYNC`, a `net.UDPConn` with `SetReadDeadline` for + the receiver, and goroutines with `sync.WaitGroup` and atomic counters. +- **Rust 37** — `std::process::Command`, + `std::thread::available_parallelism()`, `std::thread::scope` (the `jthread` + analogue), `AtomicU64`, `std::net::UdpSocket::set_read_timeout`, and + `OpenOptions::custom_flags(libc::O_DSYNC)` with `FileExt::write_at` for the + pwrite path. Only new dependency is `libc = "0.2"` (cached). Rename the + package from `template-hello` to `loadmix`; keep `[[bin]] name = "app"`. + +## Steps + +Concurrency legend: **[VM]** touches `systems-target` and is strictly serial +against every other [VM] step; **[host]** is local. Never two +`cargo build --release` at once, and never a cargo build concurrent with a +cmake build. + +**S1 [VM] — golden capture, example 37.** Stage the already-built +`examples/37-use-method-60s-analysis/cpp/build/release/app` to +`/tmp/lsp37-cpp/app` via `scp`. Record verbatim: no-args, `analyze --seconds`, +`--resource bogus`, `--seconds 0` (all exit-2 shapes); `--resource cpu +--seconds 5`; `--resource io --seconds 5`; `--resource net --seconds 5`; +`analyze --seconds 10` idle; and a compound +`nohup app --resource cpu --seconds 25 & sleep 2; app analyze --seconds 12`. +Record the guest's `nproc`. **Determine empirically whether `--resource mem` +survives** on this guest — it deliberately targets 1.35 × MemTotal and may be +OOM-killed; the answer decides whether `verify.lua` asserts `mem` exit 0 or only +its argument acceptance. No dependencies; read-only. + +**S2 [VM] — golden capture, example 40.** Stage +`examples/40-low-latency-fastpath/cpp/build/release/app` and +`run-latency-bench.sh` to `/tmp/lsp40-cpp/`. Record: usage and exit-2 shapes; +`measure --target 127.0.0.1:1 --n 10` (connect failure, exit 1); +`measure --target notanip:9999 --n 10`; `fastpath --port P --pin 99` +(out of range, exit 1); a full +`bash run-latency-bench.sh /tmp/lsp40-cpp/app 20000 500` run; and a SIGINT and +SIGTERM shutdown of both servers checking exit 0 and the shutdown lines. Record +`naive-cpus-allowed`, `fastpath-cpus-allowed`, and the three-trial p50 medians +for both variants — the observed margin sets the verify threshold. Depends on +S1 (single guest). + +**S3 [host+VM] — write `examples/37-use-method-60s-analysis/verify.lua`.** +Shape it on `examples/33-seccomp-and-landlock/verify.lua` (skip unless +`LSP_LANG` is valid; skip unless `TARGET` is set; resolve the IP via +`scripts/lab/vm-ip.sh`; skip if unresolvable) plus the `expect_true` helper +from `examples/39-benchmarking-without-lies/verify.lua`. Assertions: + +1. `./demo.sh build` exits 0. +2. Stage the binary to `/tmp/lsp37-/app` — named `app` so `/proc` comm is + `app`, the book-wide rule — and assert staging exits 0. +3. Error shapes: no args, `--resource bogus --seconds 5`, + `--resource cpu --seconds 0`, `analyze --bogus` all exit 2 with + `usage: app %-%-resource`. +4. Saturate `cpu`, `io`, and `net` with `--seconds 4`: exit 0, + `loadmix: start resource=%s seconds=4 workers=%d+`, + `loadmix: done … ops=%d+ bytes=%d+`, the worker count equals the documented + formula against the guest's `nproc`, and `ops > 0`. +5. Idle `analyze --seconds 8`: exit 0; all 8 metric lines present with a + `%d+%.%d%d` value; all 12 signal lines present in cpu/mem/io/net × U/S/E + order; `analyze: verdict resource=%a+ ratio=%d+%.%d%d`; `analyze: done` last; + `analyze: tool pidstat active_processes=%d+`; + `analyze: tool dmesg status=%a+`; `analyze: start seconds=8 cpus=%d+`. +6. **Load attribution** — one compound `ssh` with a background saturator and a + foreground analyze: under `--resource cpu`, both `cpu busy_pct` and + `cpu run_queue` rise materially above the idle values, + `signal resource=cpu type=Utilization fired=true`, and + `verdict resource=cpu`. Retry up to 3 attempts like example 36's cpu-busy + check, gating on `busy_pct >= 60` **or** `run_queue >= nproc+2` so a noisy + guest does not produce a false red. +7. Wall-clock sanity: `analyze --seconds 8` completes well under 5×8 s, proving + the samplers really run concurrently. +8. Cleanup `rm -rf /tmp/lsp37-` on the guest; `checks.finish()`. + +Then run `LSP_LANG=cpp TARGET=systems-target REPO_ROOT=… lua verify.lua` and +require a clean pass. Depends on S1. + +**S4 [host+VM] — write `examples/40-low-latency-fastpath/verify.lua`.** Same +skeleton. Stage `app` **and** `run-latency-bench.sh` to `/tmp/lsp40-/`, +`chmod 755` both. Assertions: + +1. Build and stage exit 0. +2. Error shapes: no args, `frobnicate`, `fastpath --port 9999` (no `--pin`), + `naive` (no port), `measure --target 127.0.0.1:9999` (no `--n`) all exit 2 + with `usage: app fastpath`. +3. `fastpath --port P --pin ` exits 1 with + `app: error: cpu %d+ out of range`. +4. `measure --target 127%.0%.0%.1:1 --n 10` exits 1 with `app: error: connect`; + `measure --target nothost:9999 --n 5` exits 1 with `not an IPv4 address`. +5. `bash run-latency-bench.sh /tmp/lsp40-/app 20000 500` exits 0, and from + its output `fastpath-cpus-allowed` is exactly the pinned CPU (a single value, + no range or comma) while `naive-cpus-allowed` is the full set — the + kernel-level pinning proof, not the program's own claim. +6. All six `app: percentiles_ns tag=(naive|fastpath)-%d …` lines parse; each has + `n` equal to the requested 20000; `min <= p50 <= p90 <= p99 <= p99.9 <= max`; + `mean` parses as a float; every value is greater than 0. +7. Median-of-three per variant: `median(fastpath p50) < median(naive p50)` with + the margin calibrated from S2, printing both medians unconditionally so a + marginal run is diagnosable from the log. +8. Separately: start `naive --port P` backgrounded over ssh, wait for the port, + send SIGINT, assert `app: naive shutting down` and exit 0; repeat for + `fastpath --port P --pin 0 --busy-poll` asserting + `app: fastpath listening on 0%.0%.0%.0:%d+ pinned%-cpu=0 busy%-poll=on`, + `app: fastpath shutting down`, and exit 0. This is the assertion Go is most + likely to fail, so it must be explicit. +9. Cleanup; `checks.finish()`. Run with `LSP_LANG=cpp` and require a clean pass. + +Depends on S2, and S3's VM runs must be finished (serial guest). + +**S5 [host, then VM] — Go port of 37.** Rewrite +`examples/37-use-method-60s-analysis/go/main.go` against the contract above. +`go.mod` already pins `go 1.26` and `toolchain go1.26.5` and requires +`golang.org/x/sys v0.47.0` with a matching `go.sum` — do not change them. The +build is host-local and cheap; the verify run is [VM] and serial. Gate: +`LSP_LANG=go` passes the unchanged S3 `verify.lua`. Depends on S3. + +**S6 [host, then VM] — Rust port of 37.** Rewrite +`examples/37-use-method-60s-analysis/rust/src/main.rs`; edit `Cargo.toml` +(`name = "loadmix"`, add `libc = "0.2"`, keep `edition = "2024"` and +`[[bin]] name = "app"`); regenerate `Cargo.lock`. Leave `rust-toolchain.toml` at +1.97.1. **This cargo build runs alone** — no concurrent cmake or cargo. Gate: +`LSP_LANG=rust` passes S3's `verify.lua`. Depends on S5. + +**S7 [host, then VM] — Go port of 40.** Rewrite +`examples/40-low-latency-fastpath/go/main.go` per the raw-fd, `shutdown(2)`, +pin-all-tids notes. `go.mod` and `go.sum` unchanged. Gate: `LSP_LANG=go` passes +S4's `verify.lua`, including the phase-A single-CPU assertion and the SIGINT +assertions. Depends on S4 and S6. + +**S8 [host, then VM] — Rust port of 40.** Rewrite +`examples/40-low-latency-fastpath/rust/src/main.rs`; fix `Cargo.toml` (drop +`core_affinity`, set +`nix = { version = "0.31", features = ["signal","sched","poll"] }`, keep +`anyhow`); regenerate the stale `Cargo.lock`. **Alone on the machine.** Gate: +`LSP_LANG=rust` passes S4's `verify.lua`. Depends on S7. + +**S9 [VM] — whole-runner gate.** +`python3 scripts/test-all-examples.py --mode vm --only 37-use-method-60s-analysis --only 40-low-latency-fastpath --jobs 1`. +VM examples are already strictly serial in the runner. Record the result here +per the verification discipline. Depends on S8. + +Safe to run concurrently: editing and reading files at any time; a Go build +alongside prose edits. Must be sequenced: every VM step against every other VM +step (one guest, and 40's bench pins vCPUs and would corrupt 37's USE +measurements); S6 and S8 against each other and against any cmake build. + +Out of scope but flagged: both `README.md` files are still verbatim `_template` +hello-syscall text. + +## Acceptance criteria + +1. `python3 scripts/test-all-examples.py --mode vm --only 37-use-method-60s-analysis --only 40-low-latency-fastpath` + reports PASS for all six example × language pairs, with the lab up. +2. Neither `verify.lua` is under 60 lines, and each has at least 15 + `checks.expect_*` assertions; neither can pass against a binary that merely + exits 0, provable by pointing `demo.sh` at `/bin/true` and observing failure. +3. For 37, for each of `cpu`, `io`, `net` at `--seconds 4`, all three languages + print `loadmix: start` and `loadmix: done` with an identical `workers=` value + for the same guest and `ops > 0`. +4. For 37 `analyze`, all three languages emit the identical 25-line skeleton in + the identical order — 1 start, 5 tool, 8 metric, 12 signal, verdict, done — + with every metric value formatted to exactly 2 decimal places. +5. For 37 under concurrent `--resource cpu` load, all three languages report + `signal resource=cpu type=Utilization fired=true` and `verdict resource=cpu` + within 3 attempts. +6. For 40, `run-latency-bench.sh` exits 0 for all three languages, phase A + reports a single CPU in `fastpath-cpus-allowed` and the full online set in + `naive-cpus-allowed`, and all six `percentiles_ns` lines have `n=20000` with + monotonic `min <= p50 <= p90 <= p99 <= p99.9 <= max`. +7. For 40, `median(fastpath p50) < median(naive p50)` over three interleaved + trials, for each language. +8. For 40, SIGINT to a running `naive` and to a running `fastpath --busy-poll` + produces the documented shutdown line and exit status 0, in all three + languages. +9. For both examples, all exit-2 usage paths and all exit-1 error paths produce + byte-identical message text across the three languages, modulo `strerror` + wording, which is identical on the same guest. +10. Every build succeeds with no network access: `go build` resolves from + `~/go/pkg/mod` and `cargo build --release` from `~/.cargo/registry`, + verifiable with `--offline`; `go.mod` still says `go 1.26` and + `toolchain go1.26.5`; both `rust-toolchain.toml` still say + `channel = "1.97.1"`; both `Cargo.toml` still say `edition = "2024"` with + `[[bin]] name = "app"`. +11. `python3 scripts/validate.py` passes — it `bash -n`s every `.sh` under + `examples/`. +12. No new files are created. The only files changed are the four language + sources, four Rust manifest and lock files, and the two `verify.lua`. +13. No Go or Rust source contains any word banned by `CLAUDE.md`. + +## Risks + +- **37 `--resource mem` OOM-kills the guest.** It deliberately targets 1.35 × + MemTotal and touches every page. Detected in S1 by a non-zero or signal exit + from the C++ reference. If it does, `verify.lua` must not assert exit 0 for + `mem` — assert only the shared start/done line shape at a very short + duration, or omit `mem` from the saturate loop and document why in the + `verify.lua` header. The silent-failure mode is the guest dying mid-suite and + later examples failing for unrelated reasons; mitigate by running `mem` last, + if at all. +- **Go's `/proc//status` affinity mask is the thread-group leader's**, not + the hot loop's. If the Go port sets affinity only on the `LockOSThread`'d + goroutine, phase A can still print the full CPU set — or worse, print the + right answer by luck on one run and the wrong one on another. Detected by + criterion 6; mitigated by setting affinity for every tid in + `/proc/self/task`. +- **Go's blocking reads do not return EINTR** because the runtime installs + `SA_RESTART`. A Go server copying the C++ signal design will hang forever on + SIGINT with an idle connection open, and the verify assertion will time out + rather than fail cleanly. Mitigated by the explicit SIGINT assertion in S4 + step 8 and the `shutdown(2)` design; the failure mode is a runner timeout, so + read a `VERIFY_TIMEOUT` as this bug first. +- **The fastpath-beats-naive margin is not guaranteed** on a 2-vCPU nested-KVM + guest; `run-latency-bench.sh`'s own comments note a lone trial can hit a + multi-hundred-microsecond stall. Calibrate the threshold from S2's real + numbers, compare medians of three, and always print both medians. If Go's GC + or netpoller makes the margin vanish, that is a genuine finding about the + port, not a flake — distinguish by running the bench three times manually. +- **Cargo re-resolution needs the network.** Both `Cargo.lock` files are stale + template locks, and changing `[dependencies]` forces re-resolution. + `nix 0.31.3`, `libc`, `anyhow`, `bitflags`, `cfg-if`, and `memoffset` are + cached; `core_affinity` is not. Detect early with `cargo fetch --offline` + before writing any Rust; the symptom otherwise is a build failure deep in S6 + or S8 after the expensive work is done. +- **`deploy-to-vm.sh` uses `ssh -t` and a fixed `/home/fedora/app` path.** If + `verify.lua` takes the `demo.sh run` path instead of staging directly, CRLF + from the pty silently breaks Lua patterns that look correct, and concurrent + invocations race on the binary. Mitigated by the staging design; detected as + "pattern did not match" on output that looks right in the log. +- **Locale and sysstat column drift.** The C++ forces `LC_ALL=C LANG=C` around + every capture and parses by column name. A port that hardcodes indices or + omits the locale forcing will pass on this guest and break on any other. Not + detectable by `verify.lua` on a single guest — enforce by code review against + the parsing rules above. +- **Concurrency and OOM on the host.** A previous wide fan-out was already + OOM-killed and two lab VMs are running. Two `cargo build --release` at once, + or a cargo build alongside a cmake build, can repeat that. Sequence S6 and S8 + strictly; the symptom is a build killed by signal 9 with no compiler + diagnostic. +- **Rounding divergence in `{:.2f}`.** C++, Go, and Rust do not agree on + half-way cases. Verify must match `%d+%.%d%d` shapes and numeric ranges, never + exact decimal strings compared across languages. + +## Port findings — example 40 (S7/S8) + +Recorded from the Go and Rust ports of `chatterd-fastpath`. Two of the three +were pre-empted by the Risks section above; the third was not, and is the one +worth carrying into chapter 40's prose. + +### 1. Go shutdown — the `SA_RESTART`/netpoller divergence (predicted, Risk 3) + +Confirmed exactly as Risk 3 warned. The C++ reference unblocks an in-flight +`read(2)`/`poll(2)` by installing a **non-`SA_RESTART`** handler so the syscall +returns `EINTR` and the loop notices the stop flag. That mechanism is +unavailable to Go: the Go runtime installs its own `SA_RESTART` handlers, and a +"blocking" read on a `net.Conn` never sits in a syscall to begin with — the +netpoller parks the goroutine on an epoll registration. No signal makes that +read return `EINTR`. + +The Go port therefore reaches identical observable behavior by a different +mechanism: a `signal.Notify` goroutine sets the stop flag and then **closes the +listener and the live connection**. Closing is what wakes a parked netpoller +read — the `shutdown(2)`-driven wakeup the C++ header itself cites from ch21. +The accept loop additionally carries a 200ms deadline mirroring the C++ `poll` +timeout, so shutdown stays bounded even if a close races an in-flight accept. +The busy-poll path takes a dup'd fd out of the netpoller (`TCPConn.File()` + +`O_NONBLOCK`) and is woken by `shutdown(SHUT_RDWR)` on that raw fd. Verified: the +SIGINT assertion (the one Risk 3 flagged Go was most likely to fail) passes, +`PASS 28 / FAIL 0`. + +### 2. Go pinning proof — the `/proc` thread-group leader (predicted, Risk 2) + +As Risk 2 warned, `sched_setaffinity` on a goroutine is meaningless (goroutines +migrate between Ms). The port pins with `runtime.LockOSThread` + `GOMAXPROCS(1)` +to nail the hot loop to one M, then calls `SchedSetaffinity` **twice** — once +with pid 0 (the calling thread, which is what actually restricts the loop) and +once with the process pid (the thread-group leader, which is what +`/proc//status` reports and phase A reads). This is a lighter touch than +the plan's "every tid in `/proc/self/task`" note and was sufficient: phase A +prints `fastpath-cpus-allowed 1` (single CPU) and `naive-cpus-allowed 0-1` (full +set), the kernel-level proof, on every run. + +### 3. Go range check — `runtime.NumCPU()` masked by `taskset` (NOT predicted) + +**This is the novel finding — nothing in the Risks section anticipated it.** The +bench driver's phase B launches the server under `taskset -c 1`. The C++ range +check uses `sysconf(_SC_NPROCESSORS_ONLN)`, which reports CPUs online +**system-wide** (2 on the guest) and is blind to the affinity mask, so a pin to +CPU 1 is in range. Go's `runtime.NumCPU()` returns the size of the process's +**affinity mask** — under `taskset -c 1` that is 1 — so the port's own range +check rejected the pin to CPU 1 with `app: error: cpu 1 out of range (0..0)`, +and the whole phase-B timing comparison failed to start. + +This is the same class of bug as `hardware_concurrency()` lying about its +container (the running theme of this part), but reached through an unexpected +door: a `taskset` affinity mask rather than a cgroup quota. The count that +matters for a *range check on absolute CPU indices* is the system-wide online +count, not the caller's affinity-masked view. The fix is `onlineCPUs()`, which +parses `/sys/devices/system/cpu/online` to match the C++ `sysconf` semantics. +C++ and Rust never hit this because both use `sysconf(_SC_NPROCESSORS_ONLN)` +directly. + +### 4. Rust — the port that follows C++ verbatim + +Unlike Go, Rust makes real blocking syscalls (no M:N runtime parking work off +the syscall), so the C++ non-`SA_RESTART` → `EINTR` → notice-the-stop-flag +technique works **verbatim**. The catch is that std's higher-level wrappers hide +`EINTR` — `TcpListener::accept` retries it internally (`cvt_r`) and +`Read::read_exact` treats `ErrorKind::Interrupted` as "try again" — so the +servers are built on raw fds via `nix` (`OwnedFd` as the RAII `Socket`), where +every `EINTR` is handled explicitly. Pinning is a single `sched_setaffinity(0, +…)`: the process is single-threaded (tid == pid), so one call both pins the hot +loop and shows up in `/proc//status` — no `LockOSThread`/`GOMAXPROCS` dance. +`Cargo.toml` dropped `core_affinity`/`anyhow`/`rustix`; `nix` now carries +`socket`/`net`/`poll`/`sched`/`signal`/`fs`. Verified `PASS`, median(naive +p50)=12063 ns vs median(fastpath p50)=7747 ns. + +### Whole-example gate + +`test-all-examples.py --only 40-low-latency-fastpath --mode vm` — all three +languages PASS in one runner pass; per-run margins cpp 1.47x, go 1.62x, rust +1.51x, all clearing the `verify.lua` 0.90-factor gate. + +## Progress + +| Step | Status | +|---|---| +| S1 golden capture 37 | done — see `r09-batch1-golden-capture.md` | +| S2 golden capture 40 | done — see `r09-batch1-golden-capture.md` | +| S3 verify.lua 37 | done — 50 assertions, clean pass with `LSP_LANG=cpp` (`PASS 50 / FAIL 0`); `/bin/true` negative control confirmed FAIL (`PASS 9 / FAIL 41`) | +| S4 verify.lua 40 | done — clean pass with `LSP_LANG=cpp` (`PASS 46 / FAIL 0`), commit `4ccb916` | +| S5 Go port 37 | pending | +| S6 Rust port 37 | pending | +| S7 Go port 40 | done — `PASS 28 / FAIL 0`, commit `ba1387a` (see Port findings 1–3) | +| S8 Rust port 40 | done — `PASS`, commit `127c049` (see Port finding 4) | +| S9 runner gate | 40 done (all three languages PASS in one `--mode vm` pass); 37 pending | diff --git a/assets/diagrams/36-agent-architecture.excalidraw b/assets/diagrams/36-agent-architecture.excalidraw new file mode 100644 index 0000000..1718165 --- /dev/null +++ b/assets/diagrams/36-agent-architecture.excalidraw @@ -0,0 +1,2063 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1408971376", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 1180, + "height": 160, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 175451974, + "versionNonce": 1561542465, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t560555799", + "type": "text", + "x": 32, + "y": 26, + "width": 250.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1791443454, + "versionNonce": 831328990, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent (one binary, cpp | go | rust)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent (one binary, cpp | go | rust)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r85805679", + "type": "rectangle", + "x": 20, + "y": 200, + "width": 1180, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1790748190, + "versionNonce": 1379508898, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1059376803", + "type": "text", + "x": 32, + "y": 206, + "width": 277.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1635971247, + "versionNonce": 443328149, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cross-check tools (same host, same moment)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cross-check tools (same host, same moment)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1149462620", + "type": "rectangle", + "x": 20, + "y": 370, + "width": 1180, + "height": 110, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 172902104, + "versionNonce": 1153886491, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1186737698", + "type": "text", + "x": 32, + "y": 376, + "width": 118.80000000000001, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 937125678, + "versionNonce": 1312651915, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "forward references", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "forward references", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r623934927", + "type": "rectangle", + "x": 60, + "y": 60, + "width": 260, + "height": 90, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1488770635, + "versionNonce": 540740013, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t908133552", + "type": "text", + "x": 70, + "y": 68, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1494198366, + "versionNonce": 679293075, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sample [--json]", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sample [--json]", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t991030870", + "type": "text", + "x": 70, + "y": 88, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 103655480, + "versionNonce": 252189396, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one snapshot to stdout", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one snapshot to stdout", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t424710115", + "type": "text", + "x": 70, + "y": 103, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1825550589, + "versionNonce": 1532092194, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "text or JSON", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "text or JSON", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1224269688", + "type": "rectangle", + "x": 380, + "y": 60, + "width": 260, + "height": 90, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 789997444, + "versionNonce": 569955240, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1035203881", + "type": "text", + "x": 390, + "y": 68, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1447544309, + "versionNonce": 1456869100, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "serve --port P", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "serve --port P", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t906663573", + "type": "text", + "x": 390, + "y": 88, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 473494409, + "versionNonce": 788347419, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "GET /metrics per request", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "GET /metrics per request", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1086680477", + "type": "text", + "x": 390, + "y": 103, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1416834983, + "versionNonce": 1947744740, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "hand-rolled HTTP/1.1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "hand-rolled HTTP/1.1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1270026810", + "type": "rectangle", + "x": 700, + "y": 60, + "width": 460, + "height": 90, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 19713347, + "versionNonce": 1424695723, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t414013293", + "type": "text", + "x": 710, + "y": 68, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 156631273, + "versionNonce": 436138437, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "take_snapshot() / procfs readers", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "take_snapshot() / procfs readers", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t525343333", + "type": "text", + "x": 710, + "y": 88, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1540244014, + "versionNonce": 1939007576, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shared by both subcommands", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shared by both subcommands", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t880339119", + "type": "text", + "x": 710, + "y": 103, + "width": 235.95000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 862207975, + "versionNonce": 1839741853, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "identical field schema, all 3 languages", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "identical field schema, all 3 languages", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1652806534", + "type": "rectangle", + "x": 50, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1318712184, + "versionNonce": 583351667, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t823007343", + "type": "text", + "x": 60, + "y": 248, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 80740814, + "versionNonce": 1303043119, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "vmstat 1 2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "vmstat 1 2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1909027055", + "type": "text", + "x": 60, + "y": 268, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1902127795, + "versionNonce": 1060671537, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "us+sy vs cpu_util_pct", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "us+sy vs cpu_util_pct", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1140269658", + "type": "rectangle", + "x": 240, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1185162214, + "versionNonce": 287949792, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1197506908", + "type": "text", + "x": 250, + "y": 248, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1548498705, + "versionNonce": 1640951154, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "mpstat -P ALL 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "mpstat -P ALL 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1934533405", + "type": "text", + "x": 250, + "y": 268, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 498042753, + "versionNonce": 702731533, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "not installed here", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "not installed here", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t590804793", + "type": "text", + "x": 250, + "y": 283, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 700107201, + "versionNonce": 1252904855, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ch37 lab VM", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ch37 lab VM", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1070242780", + "type": "rectangle", + "x": 430, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1350989243, + "versionNonce": 925370510, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1024625685", + "type": "text", + "x": 440, + "y": 248, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1834940360, + "versionNonce": 1347186100, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "free -m", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "free -m", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t69749757", + "type": "text", + "x": 440, + "y": 268, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1210891841, + "versionNonce": 1769203072, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "MemAvailable vs available", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "MemAvailable vs available", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r639081000", + "type": "rectangle", + "x": 620, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 852035380, + "versionNonce": 177363354, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1936399169", + "type": "text", + "x": 630, + "y": 248, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1522520593, + "versionNonce": 243125817, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ss -s", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ss -s", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t75438015", + "type": "text", + "x": 630, + "y": 268, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 314238875, + "versionNonce": 699165630, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "socket census", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "socket census", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1876216466", + "type": "rectangle", + "x": 810, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1738533183, + "versionNonce": 1391312006, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t510864948", + "type": "text", + "x": 820, + "y": 248, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1808743559, + "versionNonce": 665481561, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "awk /proc/meminfo", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "awk /proc/meminfo", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t454795003", + "type": "text", + "x": 820, + "y": 268, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1533854942, + "versionNonce": 1723377714, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "exact MemTotal match", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "exact MemTotal match", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r653716738", + "type": "rectangle", + "x": 1000, + "y": 240, + "width": 170, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1709344877, + "versionNonce": 1106210058, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1358277950", + "type": "text", + "x": 1010, + "y": 248, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 791421968, + "versionNonce": 206708710, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kill -INT", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kill -INT", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t627084449", + "type": "text", + "x": 1010, + "y": 268, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1764373489, + "versionNonce": 589366713, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "clean serve shutdown", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "clean serve shutdown", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1965094371", + "type": "rectangle", + "x": 60, + "y": 405, + "width": 340, + "height": 55, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 461765643, + "versionNonce": 1480114670, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1191583550", + "type": "text", + "x": 70, + "y": 413, + "width": 338.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 524252373, + "versionNonce": 1556169944, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Chapter 37: USE method + 60-second checklist", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Chapter 37: USE method + 60-second checklist", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1828840349", + "type": "text", + "x": 70, + "y": 433, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 292756916, + "versionNonce": 247096908, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent's fields, read systematically", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent's fields, read systematically", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1579707110", + "type": "rectangle", + "x": 430, + "y": 405, + "width": 340, + "height": 55, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1069759040, + "versionNonce": 1039170478, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t720714378", + "type": "text", + "x": 440, + "y": 413, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1423825931, + "versionNonce": 1835496390, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Chapter 38: OTLP export", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Chapter 38: OTLP export", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1647419568", + "type": "text", + "x": 440, + "y": 433, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1315725064, + "versionNonce": 1817026641, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent's numbers into the LGTM stack", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent's numbers into the LGTM stack", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r95708083", + "type": "rectangle", + "x": 800, + "y": 405, + "width": 360, + "height": 55, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1442022809, + "versionNonce": 1171431337, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1132413218", + "type": "text", + "x": 810, + "y": 413, + "width": 161.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 708102942, + "versionNonce": 297297792, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Chapter 45: PCP pmrep", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Chapter 45: PCP pmrep", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t716045537", + "type": "text", + "x": 810, + "y": 433, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 40629652, + "versionNonce": 719497315, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "an alternate collector over the same /proc", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "an alternate collector over the same /proc", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1417296650", + "type": "arrow", + "x": 320, + "y": 105, + "width": 60, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 538864892, + "versionNonce": 637906557, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 60, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1129996080", + "type": "text", + "x": 350.0, + "y": 91.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 346883638, + "versionNonce": 964886518, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shares", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shares", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a678997579", + "type": "arrow", + "x": 640, + "y": 105, + "width": 60, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 604867801, + "versionNonce": 441691488, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 60, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1208182444", + "type": "text", + "x": 670.0, + "y": 91.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1919522219, + "versionNonce": 1673706845, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "calls", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "calls", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a563870337", + "type": "arrow", + "x": 190, + "y": 150, + "width": 55, + "height": 90, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 274129387, + "versionNonce": 609705373, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -55, + 90 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t886637083", + "type": "text", + "x": 162.5, + "y": 181.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1628620958, + "versionNonce": 1760687619, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "compared", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "compared", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1609425599", + "type": "arrow", + "x": 210, + "y": 150, + "width": 115, + "height": 90, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1666249338, + "versionNonce": 1866172928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 115, + 90 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1458536597", + "type": "arrow", + "x": 470, + "y": 150, + "width": 45, + "height": 90, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1013893928, + "versionNonce": 1105397356, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 45, + 90 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1157727703", + "type": "text", + "x": 492.5, + "y": 181.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1937808698, + "versionNonce": 84170294, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "compared", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "compared", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a779698707", + "type": "arrow", + "x": 490, + "y": 150, + "width": 405, + "height": 90, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 380166965, + "versionNonce": 1995552681, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 405, + 90 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t499656212", + "type": "text", + "x": 692.5, + "y": 181.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1421787490, + "versionNonce": 1156229643, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "same host", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "same host", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a634589409", + "type": "arrow", + "x": 610, + "y": 150, + "width": 475, + "height": 90, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 929046319, + "versionNonce": 1947683796, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 475, + 90 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1756481011", + "type": "text", + "x": 847.5, + "y": 181.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1241731163, + "versionNonce": 433503743, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SIGINT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SIGINT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1419466643", + "type": "text", + "x": 40, + "y": 185, + "width": 534.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 205425328, + "versionNonce": 763126351, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Every arrow below is a number quoted from both sides, not just a tool invocation.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Every arrow below is a number quoted from both sides, not just a tool invocation.", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/36-agent-architecture.svg b/assets/diagrams/36-agent-architecture.svg new file mode 100644 index 0000000..7dfb15a --- /dev/null +++ b/assets/diagrams/36-agent-architecture.svg @@ -0,0 +1,69 @@ + + + +sysagent (one binary, cpp | go | rust) + +cross-check tools (same host, same moment) + +forward references + +sample [--json] +one snapshot to stdout +text or JSON + +serve --port P +GET /metrics per request +hand-rolled HTTP/1.1 + +take_snapshot() / procfs readers +shared by both subcommands +identical field schema, all 3 languages + +vmstat 1 2 +us+sy vs cpu_util_pct + +mpstat -P ALL 1 +not installed here +ch37 lab VM + +free -m +MemAvailable vs available + +ss -s +socket census + +awk /proc/meminfo +exact MemTotal match + +kill -INT +clean serve shutdown + +Chapter 37: USE method + 60-second checklist +sysagent's fields, read systematically + +Chapter 38: OTLP export +sysagent's numbers into the LGTM stack + +Chapter 45: PCP pmrep +an alternate collector over the same /proc + +shares +shares + +calls +calls + +compared +compared + + +compared +compared + +same host +same host + +SIGINT +SIGINT +Every arrow below is a number quoted from both sides, not just a tool invocation. + \ No newline at end of file diff --git a/assets/diagrams/36-proc-datasource-map.excalidraw b/assets/diagrams/36-proc-datasource-map.excalidraw new file mode 100644 index 0000000..fb07451 --- /dev/null +++ b/assets/diagrams/36-proc-datasource-map.excalidraw @@ -0,0 +1,2527 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r582954106", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 1180, + "height": 190, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 881008215, + "versionNonce": 1107163254, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1642727415", + "type": "text", + "x": 32, + "y": 26, + "width": 323.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 709170272, + "versionNonce": 360679167, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel-maintained files (procfs / sysfs, no root)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel-maintained files (procfs / sysfs, no root)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r264461921", + "type": "rectangle", + "x": 20, + "y": 230, + "width": 1180, + "height": 260, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1092239654, + "versionNonce": 66166987, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t661903285", + "type": "text", + "x": 32, + "y": 236, + "width": 231.00000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1654480562, + "versionNonce": 1997805573, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sysagent take_snapshot(interval_ms)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sysagent take_snapshot(interval_ms)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1420626845", + "type": "rectangle", + "x": 50, + "y": 60, + "width": 190, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 443428486, + "versionNonce": 1530249177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1430706411", + "type": "text", + "x": 60, + "y": 68, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1297372127, + "versionNonce": 1897498214, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/stat", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/stat", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t552176536", + "type": "text", + "x": 60, + "y": 88, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1504124575, + "versionNonce": 172249956, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cpu jiffies", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cpu jiffies", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1562606920", + "type": "text", + "x": 60, + "y": 103, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 291203962, + "versionNonce": 1976050593, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "counter, since boot", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "counter, since boot", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r919745058", + "type": "rectangle", + "x": 260, + "y": 60, + "width": 190, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1505719214, + "versionNonce": 1830973985, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1476184124", + "type": "text", + "x": 270, + "y": 68, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1683417667, + "versionNonce": 1605556404, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/loadavg", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/loadavg", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1145228361", + "type": "text", + "x": 270, + "y": 88, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 918006600, + "versionNonce": 840908565, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "load1/5/15, runnable/total", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "load1/5/15, runnable/total", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t422660269", + "type": "text", + "x": 270, + "y": 103, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1204710733, + "versionNonce": 1704714938, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gauge", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gauge", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1941386602", + "type": "rectangle", + "x": 470, + "y": 60, + "width": 190, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 163487257, + "versionNonce": 1220495630, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t937007672", + "type": "text", + "x": 480, + "y": 68, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1186615972, + "versionNonce": 172650057, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/meminfo", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/meminfo", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t799130928", + "type": "text", + "x": 480, + "y": 88, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1497276949, + "versionNonce": 608716653, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "MemTotal, MemAvailable", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "MemTotal, MemAvailable", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1938648012", + "type": "text", + "x": 480, + "y": 103, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 723915278, + "versionNonce": 297532465, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gauge", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gauge", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1271505938", + "type": "rectangle", + "x": 680, + "y": 60, + "width": 190, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1877821599, + "versionNonce": 298393125, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t95403573", + "type": "text", + "x": 690, + "y": 68, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 314128928, + "versionNonce": 405487442, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/diskstats", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/diskstats", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1533312022", + "type": "text", + "x": 690, + "y": 88, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1143344147, + "versionNonce": 796228345, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reads/writes/sectors", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reads/writes/sectors", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t53933938", + "type": "text", + "x": 690, + "y": 103, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 124474962, + "versionNonce": 455850626, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "counter", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "counter", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1490013097", + "type": "rectangle", + "x": 890, + "y": 60, + "width": 150, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1670216047, + "versionNonce": 923775090, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1646705131", + "type": "text", + "x": 900, + "y": 68, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1498336496, + "versionNonce": 1316548591, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/net/dev", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/net/dev", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1636043901", + "type": "text", + "x": 900, + "y": 88, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 440749574, + "versionNonce": 606976839, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "rx/tx bytes", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "rx/tx bytes", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1240568345", + "type": "text", + "x": 900, + "y": 103, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 798883801, + "versionNonce": 563487971, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "counter", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "counter", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r956982056", + "type": "rectangle", + "x": 1060, + "y": 60, + "width": 130, + "height": 70, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1645707991, + "versionNonce": 1213256934, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t237846414", + "type": "text", + "x": 1070, + "y": 68, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1578716149, + "versionNonce": 1687444078, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cgroup PSI", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cgroup PSI", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t750408835", + "type": "text", + "x": 1070, + "y": 88, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 199895702, + "versionNonce": 341082034, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cpu/mem/io.pressure", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cpu/mem/io.pressure", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1060376742", + "type": "text", + "x": 1070, + "y": 103, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1905942501, + "versionNonce": 8488753, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gauge (avg10)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gauge (avg10)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1245750523", + "type": "rectangle", + "x": 50, + "y": 155, + "width": 190, + "height": 45, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1451183121, + "versionNonce": 105737307, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t146893609", + "type": "text", + "x": 60, + "y": 163, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1039210267, + "versionNonce": 1560861384, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "t0 read, t1 read", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "t0 read, t1 read", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1531612119", + "type": "text", + "x": 60, + "y": 183, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 617441146, + "versionNonce": 1359829711, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "interval_ms apart", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "interval_ms apart", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r668174511", + "type": "rectangle", + "x": 1060, + "y": 155, + "width": 130, + "height": 45, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1081675615, + "versionNonce": 1829291249, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t191676075", + "type": "text", + "x": 1070, + "y": 163, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1691485808, + "versionNonce": 1379190537, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/proc/self/cgroup", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/proc/self/cgroup", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t153567840", + "type": "text", + "x": 1070, + "y": 183, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1346310740, + "versionNonce": 187596383, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0:: lookup", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0:: lookup", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1541657366", + "type": "rectangle", + "x": 60, + "y": 260, + "width": 230, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 436043283, + "versionNonce": 993722734, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t486879342", + "type": "text", + "x": 70, + "y": 268, + "width": 161.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 873019736, + "versionNonce": 1712902918, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "t0 = read_cpu_ticks()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "t0 = read_cpu_ticks()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1494336765", + "type": "text", + "x": 70, + "y": 288, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1550000380, + "versionNonce": 1720390844, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one read, no sleep yet", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one read, no sleep yet", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r710538971", + "type": "rectangle", + "x": 60, + "y": 340, + "width": 230, + "height": 100, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1420478656, + "versionNonce": 604495348, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1579143657", + "type": "text", + "x": 70, + "y": 348, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1842487917, + "versionNonce": 430468431, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "5 concurrent readers", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "5 concurrent readers", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1859601584", + "type": "text", + "x": 70, + "y": 368, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 273862888, + "versionNonce": 741845747, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "loadavg \u00b7 meminfo", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "loadavg \u00b7 meminfo", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1123498189", + "type": "text", + "x": 70, + "y": 383, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 178117342, + "versionNonce": 243278523, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "diskstats \u00b7 netdev \u00b7 psi", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "diskstats \u00b7 netdev \u00b7 psi", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1795362122", + "type": "text", + "x": 70, + "y": 398, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 919279696, + "versionNonce": 1041544759, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "jthread / goroutine / scoped thread", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "jthread / goroutine / scoped thread", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r192656191", + "type": "rectangle", + "x": 340, + "y": 340, + "width": 200, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1287151491, + "versionNonce": 1161508496, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1574348933", + "type": "text", + "x": 350, + "y": 348, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 714579930, + "versionNonce": 445293958, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sleep(interval_ms)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sleep(interval_ms)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t471381859", + "type": "text", + "x": 350, + "y": 368, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 210370648, + "versionNonce": 1856084044, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "default 200ms", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "default 200ms", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1597569630", + "type": "rectangle", + "x": 340, + "y": 260, + "width": 200, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 971193476, + "versionNonce": 182165555, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1568643442", + "type": "text", + "x": 350, + "y": 268, + "width": 161.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 538570455, + "versionNonce": 1409193992, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "t1 = read_cpu_ticks()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "t1 = read_cpu_ticks()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t473983340", + "type": "text", + "x": 350, + "y": 288, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1391716021, + "versionNonce": 1385861964, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "second read, after sleep", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "second read, after sleep", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r632907618", + "type": "rectangle", + "x": 600, + "y": 300, + "width": 260, + "height": 100, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 273208832, + "versionNonce": 587879424, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1126866390", + "type": "text", + "x": 610, + "y": 308, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 584886868, + "versionNonce": 443020024, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "delta = t1 - t0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "delta = t1 - t0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t119522152", + "type": "text", + "x": 610, + "y": 328, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 283811180, + "versionNonce": 42954499, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "busy = user+nice+sys+irq+softirq+steal", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "busy = user+nice+sys+irq+softirq+steal", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1243915029", + "type": "text", + "x": 610, + "y": 343, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1386095840, + "versionNonce": 1841967542, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cpu_util_pct = 100 * d_busy / d_total", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cpu_util_pct = 100 * d_busy / d_total", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1716074460", + "type": "rectangle", + "x": 920, + "y": 300, + "width": 240, + "height": 100, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 433943966, + "versionNonce": 420060362, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t555011093", + "type": "text", + "x": 930, + "y": 308, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 20302329, + "versionNonce": 1496746012, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Snapshot{}", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Snapshot{}", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t706810953", + "type": "text", + "x": 930, + "y": 328, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 314094729, + "versionNonce": 1462806112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one struct, byte-identical", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one struct, byte-identical", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t135443612", + "type": "text", + "x": 930, + "y": 343, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 728695311, + "versionNonce": 1785726321, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "field names, all 3 languages", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "field names, all 3 languages", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1036942951", + "type": "arrow", + "x": 145, + "y": 130, + "width": 0, + "height": 25, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1004443888, + "versionNonce": 1129423361, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 25 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1443545448", + "type": "arrow", + "x": 1125, + "y": 130, + "width": 0, + "height": 25, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1202691266, + "versionNonce": 337141926, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 25 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1554274264", + "type": "arrow", + "x": 175, + "y": 320, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 748816569, + "versionNonce": 1837353207, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t313911380", + "type": "text", + "x": 175.0, + "y": 316.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1263003475, + "versionNonce": 1667484608, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "t0 read", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "t0 read", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a364559441", + "type": "arrow", + "x": 290, + "y": 370, + "width": 50, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1113368229, + "versionNonce": 373887238, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 50, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1586568690", + "type": "text", + "x": 315.0, + "y": 356.0, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 213898185, + "versionNonce": 471339690, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "in parallel with", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "in parallel with", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1263885598", + "type": "arrow", + "x": 440, + "y": 340, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1902271775, + "versionNonce": 573732112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t413678691", + "type": "text", + "x": 440.0, + "y": 316.0, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 943266053, + "versionNonce": 1650449051, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sleep ends \u2192 t1 read", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sleep ends \u2192 t1 read", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1168116778", + "type": "arrow", + "x": 290, + "y": 260, + "width": 50, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1903545295, + "versionNonce": 725075634, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 50, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t474899805", + "type": "text", + "x": 315.0, + "y": 261.0, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1498311683, + "versionNonce": 653495773, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "joins before delta", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "joins before delta", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a488690370", + "type": "arrow", + "x": 540, + "y": 290, + "width": 0, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1595761353, + "versionNonce": 29220276, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1356527173", + "type": "arrow", + "x": 860, + "y": 350, + "width": 60, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 982701484, + "versionNonce": 325750495, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 60, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t655769952", + "type": "text", + "x": 890.0, + "y": 336.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 834596886, + "versionNonce": 1666204454, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fills", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fills", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1308408212", + "type": "arrow", + "x": 540, + "y": 400, + "width": 60, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 988245849, + "versionNonce": 1406478308, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 60, + -20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t127343140", + "type": "text", + "x": 570.0, + "y": 376.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 182098615, + "versionNonce": 145969596, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "join()", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "join()", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t901811539", + "type": "text", + "x": 40, + "y": 195, + "width": 699.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 614803342, + "versionNonce": 1004517693, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Counters (stat, diskstats, net/dev) need TWO samples \u2014 a single read is cumulative since boot, not a rate.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Counters (stat, diskstats, net/dev) need TWO samples \u2014 a single read is cumulative since boot, not a rate.", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t1536041611", + "type": "text", + "x": 40, + "y": 490, + "width": 640.2, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 161140158, + "versionNonce": 1552536823, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Gauges (loadavg, meminfo, PSI avg10) are valid from one read \u2014 only the CPU line needs the delta.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Gauges (loadavg, meminfo, PSI avg10) are valid from one read \u2014 only the CPU line needs the delta.", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/36-proc-datasource-map.svg b/assets/diagrams/36-proc-datasource-map.svg new file mode 100644 index 0000000..3a143a7 --- /dev/null +++ b/assets/diagrams/36-proc-datasource-map.svg @@ -0,0 +1,82 @@ + + + +kernel-maintained files (procfs / sysfs, no root) + +sysagent take_snapshot(interval_ms) + +/proc/stat +cpu jiffies +counter, since boot + +/proc/loadavg +load1/5/15, runnable/total +gauge + +/proc/meminfo +MemTotal, MemAvailable +gauge + +/proc/diskstats +reads/writes/sectors +counter + +/proc/net/dev +rx/tx bytes +counter + +cgroup PSI +cpu/mem/io.pressure +gauge (avg10) + +t0 read, t1 read +interval_ms apart + +/proc/self/cgroup +0::<path> lookup + +t0 = read_cpu_ticks() +one read, no sleep yet + +5 concurrent readers +loadavg · meminfo +diskstats · netdev · psi +jthread / goroutine / scoped thread + +sleep(interval_ms) +default 200ms + +t1 = read_cpu_ticks() +second read, after sleep + +delta = t1 - t0 +busy = user+nice+sys+irq+softirq+steal +cpu_util_pct = 100 * d_busy / d_total + +Snapshot{} +one struct, byte-identical +field names, all 3 languages + + + +t0 read +t0 read + +in parallel with +in parallel with + +sleep ends → t1 read +sleep ends → t1 read + +joins before delta +joins before delta + + +fills +fills + +join() +join() +Counters (stat, diskstats, net/dev) need TWO samples — a single read is cumulative since boot, not a rate. +Gauges (loadavg, meminfo, PSI avg10) are valid from one read — only the CPU line needs the delta. + \ No newline at end of file diff --git a/assets/diagrams/37-use-method-checklist.excalidraw b/assets/diagrams/37-use-method-checklist.excalidraw new file mode 100644 index 0000000..5a920c5 --- /dev/null +++ b/assets/diagrams/37-use-method-checklist.excalidraw @@ -0,0 +1,1298 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1828535121", + "type": "rectangle", + "x": 20, + "y": 44, + "width": 860, + "height": 132, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1911549681, + "versionNonce": 1194771286, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t971061884", + "type": "text", + "x": 32, + "y": 50, + "width": 409.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1115114775, + "versionNonce": 1851210046, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "loadmix analyze \u2014 run the real tools, read them by column name", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "loadmix analyze \u2014 run the real tools, read them by column name", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1525746211", + "type": "rectangle", + "x": 34, + "y": 64, + "width": 150, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 608658760, + "versionNonce": 442425429, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1757959926", + "type": "text", + "x": 44, + "y": 72, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1388559319, + "versionNonce": 635918750, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "4 pre-samples", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "4 pre-samples", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t576100555", + "type": "text", + "x": 44, + "y": 92, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 719885960, + "versionNonce": 989344109, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uptime \u00b7 dmesg", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uptime \u00b7 dmesg", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1318418002", + "type": "text", + "x": 44, + "y": 107, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 310157728, + "versionNonce": 1921348340, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "top \u00b7 ss", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "top \u00b7 ss", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t26242216", + "type": "text", + "x": 44, + "y": 122, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1323343183, + "versionNonce": 682085440, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\u2192 5 tool lines", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\u2192 5 tool lines", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r760599206", + "type": "rectangle", + "x": 214, + "y": 62, + "width": 184, + "height": 94, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 79190750, + "versionNonce": 559270, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t906552650", + "type": "text", + "x": 224, + "y": 70, + "width": 146.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1228098907, + "versionNonce": 1157472760, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "5 interval samplers", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "5 interval samplers", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t579391197", + "type": "text", + "x": 224, + "y": 90, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1245197840, + "versionNonce": 168562054, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "vmstat mpstat iostat", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "vmstat mpstat iostat", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t146586772", + "type": "text", + "x": 224, + "y": 105, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1887151586, + "versionNonce": 1560082573, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sar pidstat", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sar pidstat", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t929766671", + "type": "text", + "x": 224, + "y": 120, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1701005750, + "versionNonce": 517697944, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "concurrent: N sec, not 5N", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "concurrent: N sec, not 5N", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r298351363", + "type": "rectangle", + "x": 428, + "y": 64, + "width": 158, + "height": 92, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1489284604, + "versionNonce": 1613495238, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t742069179", + "type": "text", + "x": 438, + "y": 72, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 722341116, + "versionNonce": 378585060, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "parse by column name", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "parse by column name", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t587477851", + "type": "text", + "x": 438, + "y": 92, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 846687873, + "versionNonce": 232052212, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Row::get against each", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Row::get against each", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t29240485", + "type": "text", + "x": 438, + "y": 107, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1133292340, + "versionNonce": 61606307, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tool's own header row", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tool's own header row", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r22240711", + "type": "rectangle", + "x": 616, + "y": 62, + "width": 248, + "height": 94, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1839240933, + "versionNonce": 927190976, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1359887031", + "type": "text", + "x": 626, + "y": 70, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 718963117, + "versionNonce": 793117589, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "USE signal matrix", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "USE signal matrix", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1111700547", + "type": "text", + "x": 626, + "y": 90, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1996263799, + "versionNonce": 226458195, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cpu \u00b7 mem \u00b7 io \u00b7 net", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cpu \u00b7 mem \u00b7 io \u00b7 net", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t438232710", + "type": "text", + "x": 626, + "y": 105, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 914374086, + "versionNonce": 1714164059, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\u00d7 Utilization /", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\u00d7 Utilization /", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t321041093", + "type": "text", + "x": 626, + "y": 120, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 225980524, + "versionNonce": 1596231743, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Saturation / Errors", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Saturation / Errors", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1274167609", + "type": "rectangle", + "x": 616, + "y": 200, + "width": 248, + "height": 54, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1253359362, + "versionNonce": 1864780215, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t743220873", + "type": "text", + "x": 626, + "y": 208, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 999833076, + "versionNonce": 184892065, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "verdict", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "verdict", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t435278314", + "type": "text", + "x": 626, + "y": 228, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1456009902, + "versionNonce": 113299583, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "max Saturation value / threshold", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "max Saturation value / threshold", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r212130615", + "type": "rectangle", + "x": 214, + "y": 200, + "width": 372, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1418114415, + "versionNonce": 1518941700, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1118720304", + "type": "text", + "x": 224, + "y": 208, + "width": 415.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1732299435, + "versionNonce": 1224135946, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "saturate \u2014 app --resource cpu|mem|io|net --seconds N", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "saturate \u2014 app --resource cpu|mem|io|net --seconds N", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t237716139", + "type": "text", + "x": 224, + "y": 228, + "width": 302.5, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1624268924, + "versionNonce": 1128535487, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "a worker pool the concurrent analyze can attribute", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "a worker pool the concurrent analyze can attribute", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a486941093", + "type": "arrow", + "x": 184, + "y": 110, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 379850910, + "versionNonce": 251985106, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t908208615", + "type": "text", + "x": 199.0, + "y": 96.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 232273768, + "versionNonce": 1001272374, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "then", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "then", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1852020804", + "type": "arrow", + "x": 398, + "y": 110, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1988260647, + "versionNonce": 1387412134, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t277453272", + "type": "text", + "x": 413.0, + "y": 96.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1344651401, + "versionNonce": 413613547, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "5 files", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "5 files", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a972517270", + "type": "arrow", + "x": 586, + "y": 110, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1282128525, + "versionNonce": 1014633055, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t505051760", + "type": "text", + "x": 601.0, + "y": 96.0, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 21377047, + "versionNonce": 258788533, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "8 metrics", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "8 metrics", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1089215156", + "type": "arrow", + "x": 740, + "y": 156, + "width": 0, + "height": 44, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 151607588, + "versionNonce": 1846314035, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 44 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t39023330", + "type": "text", + "x": 740.0, + "y": 164.0, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 112300514, + "versionNonce": 494363052, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "highest ratio", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "highest ratio", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a308267765", + "type": "arrow", + "x": 400, + "y": 200, + "width": 78, + "height": 44, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1216768742, + "versionNonce": 992035902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -78, + -44 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t689725779", + "type": "text", + "x": 361.0, + "y": 164.0, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 358119249, + "versionNonce": 1287721749, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "seen concurrently", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "seen concurrently", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/37-use-method-checklist.svg b/assets/diagrams/37-use-method-checklist.svg new file mode 100644 index 0000000..8a2c3eb --- /dev/null +++ b/assets/diagrams/37-use-method-checklist.svg @@ -0,0 +1,45 @@ + + + +loadmix analyze — run the real tools, read them by column name + +4 pre-samples +uptime · dmesg +top · ss +→ 5 tool lines + +5 interval samplers +vmstat mpstat iostat +sar pidstat +concurrent: N sec, not 5N + +parse by column name +Row::get against each +tool's own header row + +USE signal matrix +cpu · mem · io · net +× Utilization / +Saturation / Errors + +verdict +max Saturation value / threshold + +saturate — app --resource cpu|mem|io|net --seconds N +a worker pool the concurrent analyze can attribute + +then +then + +5 files +5 files + +8 metrics +8 metrics + +highest ratio +highest ratio + +seen concurrently +seen concurrently + \ No newline at end of file diff --git a/assets/diagrams/38-otlp-pipeline.excalidraw b/assets/diagrams/38-otlp-pipeline.excalidraw new file mode 100644 index 0000000..b465118 --- /dev/null +++ b/assets/diagrams/38-otlp-pipeline.excalidraw @@ -0,0 +1,1504 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r741081021", + "type": "rectangle", + "x": 18, + "y": 44, + "width": 270, + "height": 306, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 225816258, + "versionNonce": 643451878, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t899478514", + "type": "text", + "x": 30, + "y": 50, + "width": 270.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 183786185, + "versionNonce": 448631544, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "serve \u2014 one traced request per input line", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "serve \u2014 one traced request per input line", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1531062361", + "type": "rectangle", + "x": 34, + "y": 70, + "width": 238, + "height": 46, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1712904689, + "versionNonce": 180779477, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1507635861", + "type": "text", + "x": 44, + "y": 78, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1829602298, + "versionNonce": 969454680, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "span: request", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "span: request", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t588968590", + "type": "text", + "x": 44, + "y": 98, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1248966371, + "versionNonce": 239949492, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(attr request.seq)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(attr request.seq)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1806374159", + "type": "rectangle", + "x": 34, + "y": 136, + "width": 114, + "height": 44, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 522606369, + "versionNonce": 153033594, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t159151705", + "type": "text", + "x": 44, + "y": 144, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1838569716, + "versionNonce": 1474132535, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "span: work", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "span: work", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1283974936", + "type": "text", + "x": 44, + "y": 164, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 697353329, + "versionNonce": 1660717177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sleep 1+seq%4 ms", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sleep 1+seq%4 ms", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r483390533", + "type": "rectangle", + "x": 158, + "y": 136, + "width": 114, + "height": 44, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1973827437, + "versionNonce": 181619165, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1135396866", + "type": "text", + "x": 168, + "y": 144, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1977733107, + "versionNonce": 265398536, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "span: respond", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "span: respond", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t841334048", + "type": "text", + "x": 168, + "y": 164, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 335037383, + "versionNonce": 747199743, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "write ok ", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "write ok ", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r576808370", + "type": "rectangle", + "x": 34, + "y": 212, + "width": 238, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 856372215, + "versionNonce": 91473368, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t691581774", + "type": "text", + "x": 44, + "y": 220, + "width": 284.90000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 894499201, + "versionNonce": 1709996838, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "metric requests_total (+1 / request)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "metric requests_total (+1 / request)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1947148424", + "type": "rectangle", + "x": 34, + "y": 262, + "width": 238, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 694883693, + "versionNonce": 908438430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1323531031", + "type": "text", + "x": 44, + "y": 270, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1277903430, + "versionNonce": 1426270640, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "metric request_duration (ms)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "metric request_duration (ms)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r992242329", + "type": "rectangle", + "x": 330, + "y": 150, + "width": 150, + "height": 74, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1383846026, + "versionNonce": 674882274, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t408707240", + "type": "text", + "x": 340, + "y": 158, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1135140922, + "versionNonce": 1219324266, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTLP / HTTP exporter", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTLP / HTTP exporter", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1545332052", + "type": "text", + "x": 340, + "y": 178, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 675966129, + "versionNonce": 507584458, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "batch ~2s \u00b7 flush on SIGINT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "batch ~2s \u00b7 flush on SIGINT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1383581084", + "type": "rectangle", + "x": 522, + "y": 150, + "width": 132, + "height": 74, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 769659761, + "versionNonce": 1061586808, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1781582203", + "type": "text", + "x": 532, + "y": 158, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1696685415, + "versionNonce": 899611772, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTel collector", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTel collector", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1994328770", + "type": "text", + "x": 532, + "y": 178, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 470669961, + "versionNonce": 488947992, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": ":4318", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": ":4318", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1165621825", + "type": "rectangle", + "x": 690, + "y": 80, + "width": 150, + "height": 54, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 499738172, + "versionNonce": 1418026005, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t849248538", + "type": "text", + "x": 700, + "y": 88, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1226474770, + "versionNonce": 61262528, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Tempo", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Tempo", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1196048745", + "type": "text", + "x": 700, + "y": 108, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 569175887, + "versionNonce": 1025293689, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "traces", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "traces", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r377490571", + "type": "rectangle", + "x": 690, + "y": 166, + "width": 150, + "height": 54, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1437235094, + "versionNonce": 497791161, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1499103060", + "type": "text", + "x": 700, + "y": 174, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1150204775, + "versionNonce": 1886390932, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Prometheus", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Prometheus", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t453855235", + "type": "text", + "x": 700, + "y": 194, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1260623957, + "versionNonce": 901005141, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "metrics", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "metrics", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1757789271", + "type": "rectangle", + "x": 876, + "y": 120, + "width": 110, + "height": 74, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1460498779, + "versionNonce": 199530857, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t153346616", + "type": "text", + "x": 886, + "y": 128, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 692769196, + "versionNonce": 1396721558, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Grafana", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Grafana", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t940408504", + "type": "text", + "x": 886, + "y": 148, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 198772307, + "versionNonce": 479327575, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "query :3000", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "query :3000", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a703548535", + "type": "arrow", + "x": 91, + "y": 116, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1535664092, + "versionNonce": 1477039880, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1048238415", + "type": "arrow", + "x": 215, + "y": 116, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 696829022, + "versionNonce": 611854202, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1199800566", + "type": "arrow", + "x": 288, + "y": 187, + "width": 42, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 3819185, + "versionNonce": 1955251081, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 42, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1738072933", + "type": "text", + "x": 309.0, + "y": 173.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1610760803, + "versionNonce": 1621063608, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "OTLP", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "OTLP", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a683849961", + "type": "arrow", + "x": 480, + "y": 187, + "width": 42, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1908184900, + "versionNonce": 1553803712, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 42, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1825144796", + "type": "text", + "x": 501.0, + "y": 173.0, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 191580898, + "versionNonce": 269259120, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "POST /v1/{traces,metrics}", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "POST /v1/{traces,metrics}", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a308828967", + "type": "arrow", + "x": 654, + "y": 175, + "width": 36, + "height": 68, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 734696004, + "versionNonce": 1272647777, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 36, + -68 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1509603150", + "type": "text", + "x": 672.0, + "y": 127.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 363703738, + "versionNonce": 349816037, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spans", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spans", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a571200406", + "type": "arrow", + "x": 654, + "y": 193, + "width": 36, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1331465073, + "versionNonce": 354185675, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 36, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1020187141", + "type": "text", + "x": 672.0, + "y": 179.0, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 414961054, + "versionNonce": 1241625339, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "metrics", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "metrics", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a392474281", + "type": "arrow", + "x": 840, + "y": 107, + "width": 36, + "height": 33, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 51937424, + "versionNonce": 891027450, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 36, + 33 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a745386123", + "type": "arrow", + "x": 840, + "y": 193, + "width": 36, + "height": 25, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 179555394, + "versionNonce": 497029143, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 36, + -25 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t511385737", + "type": "text", + "x": 500, + "y": 328, + "width": 699.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1892688870, + "versionNonce": 1730362066, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cpp \u00b7 go \u00b7 rust \u2014 each tags service.name = lsp-otel-, so the three never collide in the shared stack", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cpp \u00b7 go \u00b7 rust \u2014 each tags service.name = lsp-otel-, so the three never collide in the shared stack", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/38-otlp-pipeline.svg b/assets/diagrams/38-otlp-pipeline.svg new file mode 100644 index 0000000..6ff633c --- /dev/null +++ b/assets/diagrams/38-otlp-pipeline.svg @@ -0,0 +1,50 @@ + + + +serve — one traced request per input line + +span: request +(attr request.seq) + +span: work +sleep 1+seq%4 ms + +span: respond +write ok <seq> + +metric requests_total (+1 / request) + +metric request_duration (ms) + +OTLP / HTTP exporter +batch ~2s · flush on SIGINT + +OTel collector +:4318 + +Tempo +traces + +Prometheus +metrics + +Grafana +query :3000 + + + +OTLP +OTLP + +POST /v1/{traces,metrics} +POST /v1/{traces,metrics} + +spans +spans + +metrics +metrics + + +cpp · go · rust — each tags service.name = lsp-otel-<lang>, so the three never collide in the shared stack + \ No newline at end of file diff --git a/assets/diagrams/39-benchmark-pipeline.excalidraw b/assets/diagrams/39-benchmark-pipeline.excalidraw new file mode 100644 index 0000000..ee47710 --- /dev/null +++ b/assets/diagrams/39-benchmark-pipeline.excalidraw @@ -0,0 +1,1742 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r631905563", + "type": "rectangle", + "x": 20, + "y": 30, + "width": 860, + "height": 220, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1613208840, + "versionNonce": 1945892119, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t652591086", + "type": "text", + "x": 32, + "y": 36, + "width": 290.40000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 668092218, + "versionNonce": 975233278, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "benchlab (real): --op X --iters N --warmup W", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "benchlab (real): --op X --iters N --warmup W", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1682003337", + "type": "rectangle", + "x": 20, + "y": 280, + "width": 860, + "height": 120, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 236019125, + "versionNonce": 580488218, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t898266080", + "type": "text", + "x": 32, + "y": 286, + "width": 211.20000000000002, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1351764687, + "versionNonce": 1170420967, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "benchlab --lie: the anti-pattern", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "benchlab --lie: the anti-pattern", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r512038819", + "type": "rectangle", + "x": 45, + "y": 70, + "width": 130, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 749553404, + "versionNonce": 812152028, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t893179244", + "type": "text", + "x": 55, + "y": 78, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1523081354, + "versionNonce": 108381683, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "warmup loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "warmup loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t196329093", + "type": "text", + "x": 55, + "y": 98, + "width": 90.75000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 242306547, + "versionNonce": 1915129555, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "W untimed calls", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "W untimed calls", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t421244280", + "type": "text", + "x": 55, + "y": 113, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 279617325, + "versionNonce": 1448041281, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "run_once(op)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "run_once(op)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1691018794", + "type": "rectangle", + "x": 205, + "y": 70, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 893651280, + "versionNonce": 1162420507, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1908594576", + "type": "text", + "x": 215, + "y": 78, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 173002470, + "versionNonce": 1119357187, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "timed loop", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "timed loop", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t344973805", + "type": "text", + "x": 215, + "y": 98, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1312314506, + "versionNonce": 469339249, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "N iterations", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "N iterations", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t239063658", + "type": "text", + "x": 215, + "y": 113, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 859172319, + "versionNonce": 285775387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "monotonic clock each", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "monotonic clock each", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1717754425", + "type": "rectangle", + "x": 385, + "y": 70, + "width": 120, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 24766058, + "versionNonce": 327687940, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1113529583", + "type": "text", + "x": 395, + "y": 78, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 901356380, + "versionNonce": 1727224128, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "raw[] samples", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "raw[] samples", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t274412321", + "type": "text", + "x": 395, + "y": 98, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 611467154, + "versionNonce": 848152882, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ns per iter", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ns per iter", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1081138361", + "type": "text", + "x": 395, + "y": 113, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1105270940, + "versionNonce": 1190576768, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "checksum +=", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "checksum +=", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r647931504", + "type": "rectangle", + "x": 535, + "y": 40, + "width": 150, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 29568479, + "versionNonce": 1290876617, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t743194493", + "type": "text", + "x": 545, + "y": 48, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1122045060, + "versionNonce": 1119534115, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sort + percentile", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sort + percentile", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1534321409", + "type": "text", + "x": 545, + "y": 68, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1093172440, + "versionNonce": 23780250, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "min / median /", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "min / median /", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1723119174", + "type": "text", + "x": 545, + "y": 83, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1759277880, + "versionNonce": 24517822, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "p99 / max", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "p99 / max", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1397988809", + "type": "rectangle", + "x": 535, + "y": 115, + "width": 150, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1004606339, + "versionNonce": 1228128388, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1032917138", + "type": "text", + "x": 545, + "y": 123, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 352039324, + "versionNonce": 88405525, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "co_correct()", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "co_correct()", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t193171907", + "type": "text", + "x": 545, + "y": 143, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1273234880, + "versionNonce": 1269800514, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "backfill above", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "backfill above", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t339053596", + "type": "text", + "x": 545, + "y": 158, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 338857100, + "versionNonce": 1867999028, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "expected_interval", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "expected_interval", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1390566498", + "type": "rectangle", + "x": 715, + "y": 40, + "width": 140, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1555802314, + "versionNonce": 33842116, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1434279118", + "type": "text", + "x": 725, + "y": 48, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 545554075, + "versionNonce": 1897243955, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "printed table", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "printed table", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t673520953", + "type": "text", + "x": 725, + "y": 68, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 222087907, + "versionNonce": 1440282450, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "min/median/p99/max", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "min/median/p99/max", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1391311678", + "type": "rectangle", + "x": 715, + "y": 115, + "width": 140, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1406921908, + "versionNonce": 702006582, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1210401811", + "type": "text", + "x": 725, + "y": 123, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 632813702, + "versionNonce": 1271460990, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "co_p99_ns", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "co_p99_ns", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t479006361", + "type": "text", + "x": 725, + "y": 143, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1375328433, + "versionNonce": 1109608073, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "co_n, checksum", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "co_n, checksum", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r942683135", + "type": "rectangle", + "x": 45, + "y": 310, + "width": 160, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 590426300, + "versionNonce": 1692317522, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t16314147", + "type": "text", + "x": 55, + "y": 318, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1039636538, + "versionNonce": 547163004, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one call", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one call", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1197927702", + "type": "text", + "x": 55, + "y": 338, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1106959399, + "versionNonce": 190263162, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no warmup", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no warmup", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r774286531", + "type": "rectangle", + "x": 245, + "y": 310, + "width": 170, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1553888465, + "versionNonce": 1675560172, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t870891255", + "type": "text", + "x": 255, + "y": 318, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1120069591, + "versionNonce": 885246810, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "wall-clock delta", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "wall-clock delta", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1507961026", + "type": "text", + "x": 255, + "y": 338, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1122163430, + "versionNonce": 1679030455, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "t1 - t0, single sample", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "t1 - t0, single sample", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r568432647", + "type": "rectangle", + "x": 455, + "y": 310, + "width": 220, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 348339490, + "versionNonce": 1694392658, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t998011999", + "type": "text", + "x": 465, + "y": 318, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 391978614, + "versionNonce": 1594591628, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "elapsed_ns printed", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "elapsed_ns printed", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t913390233", + "type": "text", + "x": 465, + "y": 338, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1998655104, + "versionNonce": 64455582, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no percentiles \u2014", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no percentiles \u2014", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t56280581", + "type": "text", + "x": 465, + "y": 353, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1685496670, + "versionNonce": 1050297248, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "nothing to compute one from", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "nothing to compute one from", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a521562562", + "type": "arrow", + "x": 175, + "y": 100, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1504868817, + "versionNonce": 1591017346, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t72481414", + "type": "text", + "x": 190.0, + "y": 86.0, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 610180293, + "versionNonce": 1285613430, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "steady state", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "steady state", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a607353358", + "type": "arrow", + "x": 355, + "y": 100, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1986379525, + "versionNonce": 801016860, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a779978086", + "type": "arrow", + "x": 505, + "y": 90, + "width": 30, + "height": 22, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 870885002, + "versionNonce": 367798928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + -22 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a268089903", + "type": "arrow", + "x": 505, + "y": 110, + "width": 30, + "height": 35, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 431687863, + "versionNonce": 1194182529, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 35 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1313449064", + "type": "arrow", + "x": 685, + "y": 68, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1874831, + "versionNonce": 671659736, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1204438320", + "type": "arrow", + "x": 685, + "y": 145, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 259250556, + "versionNonce": 1842265492, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1210825085", + "type": "arrow", + "x": 205, + "y": 340, + "width": 40, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1224398566, + "versionNonce": 1719931239, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 40, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a287245786", + "type": "arrow", + "x": 415, + "y": 340, + "width": 40, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1465130152, + "versionNonce": 1440143458, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 40, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1167347618", + "type": "text", + "x": 450, + "y": 240, + "width": 586.85, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 538893215, + "versionNonce": 1669647837, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "same codec, same fixed workload \u2014 the harness, not the code under test, is what changes the story", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "same codec, same fixed workload \u2014 the harness, not the code under test, is what changes the story", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/39-benchmark-pipeline.svg b/assets/diagrams/39-benchmark-pipeline.svg new file mode 100644 index 0000000..2ebdf6d --- /dev/null +++ b/assets/diagrams/39-benchmark-pipeline.svg @@ -0,0 +1,54 @@ + + + +benchlab (real): --op X --iters N --warmup W + +benchlab --lie: the anti-pattern + +warmup loop +W untimed calls +run_once(op) + +timed loop +N iterations +monotonic clock each + +raw[] samples +ns per iter +checksum += + +sort + percentile +min / median / +p99 / max + +co_correct() +backfill above +expected_interval + +printed table +min/median/p99/max + +co_p99_ns +co_n, checksum + +one call +no warmup + +wall-clock delta +t1 - t0, single sample + +elapsed_ns printed +no percentiles — +nothing to compute one from + +steady state +steady state + + + + + + + +same codec, same fixed workload — the harness, not the code under test, is what changes the story + \ No newline at end of file diff --git a/assets/diagrams/39-coordinated-omission.excalidraw b/assets/diagrams/39-coordinated-omission.excalidraw new file mode 100644 index 0000000..29cdf7b --- /dev/null +++ b/assets/diagrams/39-coordinated-omission.excalidraw @@ -0,0 +1,1883 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r955739174", + "type": "rectangle", + "x": 20, + "y": 30, + "width": 860, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 209512506, + "versionNonce": 131430038, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1390102400", + "type": "text", + "x": 32, + "y": 36, + "width": 376.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1257168031, + "versionNonce": 1416543053, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "closed loop: benchlab's raw timer (recorded samples only)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "closed loop: benchlab's raw timer (recorded samples only)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r163766705", + "type": "rectangle", + "x": 20, + "y": 210, + "width": 860, + "height": 150, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 724609965, + "versionNonce": 656297070, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t312585737", + "type": "text", + "x": 32, + "y": 216, + "width": 343.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 863949145, + "versionNonce": 1027807999, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "what a fixed-rate caller would have seen (corrected)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "what a fixed-rate caller would have seen (corrected)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r515198220", + "type": "rectangle", + "x": 40, + "y": 70, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1228597201, + "versionNonce": 1589053102, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t599885200", + "type": "text", + "x": 50, + "y": 78, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1190622344, + "versionNonce": 1203105143, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "iter 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "iter 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1960588820", + "type": "text", + "x": 50, + "y": 98, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1151027178, + "versionNonce": 1322159334, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "44 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "44 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r637333982", + "type": "rectangle", + "x": 150, + "y": 70, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 145267599, + "versionNonce": 1587821102, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1814083370", + "type": "text", + "x": 160, + "y": 78, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 454409138, + "versionNonce": 1329293865, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "iter 2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "iter 2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t579327016", + "type": "text", + "x": 160, + "y": 98, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 317929327, + "versionNonce": 385881493, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "41 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "41 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r428915559", + "type": "rectangle", + "x": 260, + "y": 70, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1940250490, + "versionNonce": 940933377, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1638915800", + "type": "text", + "x": 270, + "y": 78, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 196481105, + "versionNonce": 1515669972, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "iter 3", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "iter 3", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t453261003", + "type": "text", + "x": 270, + "y": 98, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 721948826, + "versionNonce": 1539371738, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "39 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "39 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r329382474", + "type": "rectangle", + "x": 380, + "y": 60, + "width": 160, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1636456575, + "versionNonce": 1242093072, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1189429756", + "type": "text", + "x": 390, + "y": 68, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1045196728, + "versionNonce": 522455724, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "iter 4 (STALL)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "iter 4 (STALL)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t803085962", + "type": "text", + "x": 390, + "y": 88, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 718186957, + "versionNonce": 1044348014, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "page fault / sched", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "page fault / sched", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t677208352", + "type": "text", + "x": 390, + "y": 103, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 251740415, + "versionNonce": 1861209030, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "1057 ns, one sample", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "1057 ns, one sample", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1466292198", + "type": "rectangle", + "x": 40, + "y": 250, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1477154024, + "versionNonce": 646880315, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1012417396", + "type": "text", + "x": 50, + "y": 258, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 558331623, + "versionNonce": 1584524224, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 1", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 1", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1598592528", + "type": "text", + "x": 50, + "y": 278, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1423898307, + "versionNonce": 850973567, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "44 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "44 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r35321425", + "type": "rectangle", + "x": 150, + "y": 250, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1793357816, + "versionNonce": 1297009210, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1353939489", + "type": "text", + "x": 160, + "y": 258, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 332181774, + "versionNonce": 661120520, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1430084551", + "type": "text", + "x": 160, + "y": 278, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1224830100, + "versionNonce": 1012743700, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "41 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "41 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1337039883", + "type": "rectangle", + "x": 260, + "y": 250, + "width": 90, + "height": 46, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 649400825, + "versionNonce": 1190057711, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t145665262", + "type": "text", + "x": 270, + "y": 258, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 583719151, + "versionNonce": 636054609, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 3", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 3", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t654196762", + "type": "text", + "x": 270, + "y": 278, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1594960087, + "versionNonce": 1888018395, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "39 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "39 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1671368320", + "type": "rectangle", + "x": 380, + "y": 240, + "width": 110, + "height": 66, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 605813528, + "versionNonce": 1469021569, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1705467403", + "type": "text", + "x": 390, + "y": 248, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 447886779, + "versionNonce": 359834545, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 4 stalls", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 4 stalls", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t958418913", + "type": "text", + "x": 390, + "y": 268, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 467462560, + "versionNonce": 1879778646, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "1057 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "1057 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1339571663", + "type": "rectangle", + "x": 500, + "y": 240, + "width": 100, + "height": 66, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 703393466, + "versionNonce": 1447497678, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t397130604", + "type": "text", + "x": 510, + "y": 248, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1367990859, + "versionNonce": 167629633, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 5 queued", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 5 queued", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t968481839", + "type": "text", + "x": 510, + "y": 268, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1248710910, + "versionNonce": 1513890511, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "~1018 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "~1018 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t597141808", + "type": "text", + "x": 510, + "y": 283, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1908123590, + "versionNonce": 1125175088, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(synthetic)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(synthetic)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r558761017", + "type": "rectangle", + "x": 610, + "y": 240, + "width": 100, + "height": 66, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1424679774, + "versionNonce": 1796926758, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t281601287", + "type": "text", + "x": 620, + "y": 248, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1618707521, + "versionNonce": 550606208, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "req 6 queued", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "req 6 queued", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t481815506", + "type": "text", + "x": 620, + "y": 268, + "width": 42.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 48884287, + "versionNonce": 13253933, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "~979 ns", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "~979 ns", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t549675361", + "type": "text", + "x": 620, + "y": 283, + "width": 66.55000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 745897082, + "versionNonce": 309322425, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(synthetic)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(synthetic)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r341681614", + "type": "rectangle", + "x": 720, + "y": 240, + "width": 120, + "height": 66, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 444542827, + "versionNonce": 721974391, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1319566380", + "type": "text", + "x": 730, + "y": 248, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 524221502, + "versionNonce": 1783284838, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "... down to", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "... down to", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1559262", + "type": "text", + "x": 730, + "y": 268, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 398266543, + "versionNonce": 178190748, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "expected_interval", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "expected_interval", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1989933817", + "type": "text", + "x": 730, + "y": 283, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 167998857, + "versionNonce": 479483333, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(43 ns floor)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(43 ns floor)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1230269848", + "type": "arrow", + "x": 130, + "y": 93, + "width": 20, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1787371273, + "versionNonce": 1249710153, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 20, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a990678909", + "type": "arrow", + "x": 240, + "y": 93, + "width": 20, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1786254528, + "versionNonce": 772135926, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 20, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1389375274", + "type": "arrow", + "x": 350, + "y": 93, + "width": 30, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1607180286, + "versionNonce": 114051506, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 30, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1383681615", + "type": "arrow", + "x": 130, + "y": 273, + "width": 20, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1348377937, + "versionNonce": 1954110179, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 20, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a746830321", + "type": "arrow", + "x": 240, + "y": 273, + "width": 20, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1752540114, + "versionNonce": 855328253, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 20, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a601193123", + "type": "arrow", + "x": 370, + "y": 273, + "width": 10, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1391036945, + "versionNonce": 1740821472, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 10, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1431109361", + "type": "arrow", + "x": 490, + "y": 273, + "width": 10, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 985989794, + "versionNonce": 1179014543, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 10, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t627965596", + "type": "text", + "x": 495.0, + "y": 259.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 215096113, + "versionNonce": 680534342, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "backfilled", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "backfilled", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1927887357", + "type": "arrow", + "x": 600, + "y": 273, + "width": 10, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 479368571, + "versionNonce": 653519752, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 10, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a834623444", + "type": "arrow", + "x": 460, + "y": 126, + "width": 0, + "height": 114, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1866200109, + "versionNonce": 790194439, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 114 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t152190447", + "type": "text", + "x": 460.0, + "y": 169.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1487924832, + "versionNonce": 1946521606, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "same stall", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "same stall", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t682298224", + "type": "text", + "x": 450, + "y": 185, + "width": 556.6, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 612325107, + "versionNonce": 914553047, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "raw p99 (1000-sample window): only 1 of the 1000 samples is the stall \u2014 buried, not the tail", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "raw p99 (1000-sample window): only 1 of the 1000 samples is the stall \u2014 buried, not the tail", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1496933294", + "type": "text", + "x": 450, + "y": 375, + "width": 659.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1043379712, + "versionNonce": 312644827, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "co_p99 backfills the requests that would have queued behind the stall \u2014 co_n grows, co_p99_ns \u2265 p99_ns always", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "co_p99 backfills the requests that would have queued behind the stall \u2014 co_n grows, co_p99_ns \u2265 p99_ns always", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/39-coordinated-omission.svg b/assets/diagrams/39-coordinated-omission.svg new file mode 100644 index 0000000..5f8f0f5 --- /dev/null +++ b/assets/diagrams/39-coordinated-omission.svg @@ -0,0 +1,59 @@ + + + +closed loop: benchlab's raw timer (recorded samples only) + +what a fixed-rate caller would have seen (corrected) + +iter 1 +44 ns + +iter 2 +41 ns + +iter 3 +39 ns + +iter 4 (STALL) +page fault / sched +1057 ns, one sample + +req 1 +44 ns + +req 2 +41 ns + +req 3 +39 ns + +req 4 stalls +1057 ns + +req 5 queued +~1018 ns +(synthetic) + +req 6 queued +~979 ns +(synthetic) + +... down to +expected_interval +(43 ns floor) + + + + + + + +backfilled +backfilled + + +same stall +same stall +raw p99 (1000-sample window): only 1 of the 1000 samples is the stall — buried, not the tail +co_p99 backfills the requests that would have queued behind the stall — co_n grows, co_p99_ns ≥ p99_ns always + \ No newline at end of file diff --git a/assets/diagrams/40-fastpath-datapath.excalidraw b/assets/diagrams/40-fastpath-datapath.excalidraw new file mode 100644 index 0000000..41b4ed0 --- /dev/null +++ b/assets/diagrams/40-fastpath-datapath.excalidraw @@ -0,0 +1,1651 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r576847913", + "type": "rectangle", + "x": 20, + "y": 36, + "width": 860, + "height": 176, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1318815311, + "versionNonce": 1210219905, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t792476239", + "type": "text", + "x": 32, + "y": 42, + "width": 217.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 815317867, + "versionNonce": 1874991225, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "naive \u2014 blocking read, migratable", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "naive \u2014 blocking read, migratable", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1447065660", + "type": "rectangle", + "x": 20, + "y": 236, + "width": 860, + "height": 182, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 800957353, + "versionNonce": 1933279194, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t781391618", + "type": "text", + "x": 32, + "y": 242, + "width": 217.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 788654274, + "versionNonce": 1481588822, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fastpath \u2014 busy-poll, pinned core", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fastpath \u2014 busy-poll, pinned core", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r742853361", + "type": "rectangle", + "x": 40, + "y": 100, + "width": 120, + "height": 56, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1349359224, + "versionNonce": 242211483, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1671129575", + "type": "text", + "x": 50, + "y": 108, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 445246785, + "versionNonce": 1129296677, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "client", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "client", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t545271663", + "type": "text", + "x": 50, + "y": 128, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1904394044, + "versionNonce": 171625638, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sends 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sends 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r862482686", + "type": "rectangle", + "x": 210, + "y": 100, + "width": 170, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 187833783, + "versionNonce": 934042177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1305082535", + "type": "text", + "x": 220, + "y": 108, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 142055358, + "versionNonce": 1644296585, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "read(2) \u2014 blocks", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "read(2) \u2014 blocks", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t316123681", + "type": "text", + "x": 220, + "y": 128, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1175947410, + "versionNonce": 1416530451, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fresh heap buf / msg", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fresh heap buf / msg", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r357030401", + "type": "rectangle", + "x": 440, + "y": 100, + "width": 160, + "height": 56, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 321034818, + "versionNonce": 21074690, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1330397340", + "type": "text", + "x": 450, + "y": 108, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 706273742, + "versionNonce": 9756235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel scheduler", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel scheduler", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1279704816", + "type": "text", + "x": 450, + "y": 128, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1176630666, + "versionNonce": 490179011, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sleep \u2192 wake", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sleep \u2192 wake", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1039256552", + "type": "rectangle", + "x": 680, + "y": 100, + "width": 150, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 245844712, + "versionNonce": 1900219957, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1170117601", + "type": "text", + "x": 690, + "y": 108, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 974698822, + "versionNonce": 1421136377, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "write(2)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "write(2)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1415755333", + "type": "text", + "x": 690, + "y": 128, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1783797907, + "versionNonce": 1938631476, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "echo 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "echo 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1519797639", + "type": "rectangle", + "x": 40, + "y": 300, + "width": 120, + "height": 56, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1235202712, + "versionNonce": 1793831687, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t720009425", + "type": "text", + "x": 50, + "y": 308, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 945678405, + "versionNonce": 808670918, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "client", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "client", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t742637678", + "type": "text", + "x": 50, + "y": 328, + "width": 54.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 515470211, + "versionNonce": 315239258, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sends 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sends 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r292302941", + "type": "rectangle", + "x": 210, + "y": 294, + "width": 210, + "height": 68, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1204499860, + "versionNonce": 677424011, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t938065480", + "type": "text", + "x": 220, + "y": 302, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 714798845, + "versionNonce": 921902662, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "busy-poll recv(2)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "busy-poll recv(2)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1978816933", + "type": "text", + "x": 220, + "y": 322, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1118241278, + "versionNonce": 1232426824, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spin on EAGAIN, one reused buf", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spin on EAGAIN, one reused buf", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t395843310", + "type": "text", + "x": 220, + "y": 337, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1251254995, + "versionNonce": 1984576649, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "pinned: CPU 1", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "pinned: CPU 1", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1891588506", + "type": "rectangle", + "x": 470, + "y": 250, + "width": 170, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1221828708, + "versionNonce": 365480535, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1696882328", + "type": "text", + "x": 480, + "y": 258, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1340274838, + "versionNonce": 614448764, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "scheduler \u2014 never entered", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "scheduler \u2014 never entered", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r860242700", + "type": "rectangle", + "x": 680, + "y": 300, + "width": 150, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 615807350, + "versionNonce": 1376656462, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t314697987", + "type": "text", + "x": 690, + "y": 308, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 102469286, + "versionNonce": 1258475255, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "write(2)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "write(2)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t602665476", + "type": "text", + "x": 690, + "y": 328, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1288354933, + "versionNonce": 1381139435, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "echo 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "echo 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a825307638", + "type": "arrow", + "x": 160, + "y": 128, + "width": 50, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1306277521, + "versionNonce": 687326192, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 50, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t903727161", + "type": "text", + "x": 185.0, + "y": 114.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 384820706, + "versionNonce": 1399273609, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "frame", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "frame", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a132798046", + "type": "arrow", + "x": 380, + "y": 128, + "width": 60, + "height": 0, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 554315064, + "versionNonce": 832038760, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 60, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1182251171", + "type": "text", + "x": 410.0, + "y": 114.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 540816138, + "versionNonce": 357327090, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sleep/wake", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sleep/wake", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a274240587", + "type": "arrow", + "x": 600, + "y": 128, + "width": 80, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1013574871, + "versionNonce": 1864507763, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 80, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t2967221", + "type": "text", + "x": 640.0, + "y": 114.0, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 114762533, + "versionNonce": 1469525940, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "data ready", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "data ready", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a849909807", + "type": "arrow", + "x": 755, + "y": 192, + "width": 655, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1912977285, + "versionNonce": 708609235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -655, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1728776140", + "type": "text", + "x": 427.5, + "y": 178.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 744944039, + "versionNonce": 1775546482, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "echo 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "echo 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1829572012", + "type": "arrow", + "x": 160, + "y": 328, + "width": 50, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 628949221, + "versionNonce": 1193247542, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 50, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1090425960", + "type": "text", + "x": 185.0, + "y": 314.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1121355400, + "versionNonce": 552469647, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "frame", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "frame", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1680538512", + "type": "arrow", + "x": 420, + "y": 324, + "width": 260, + "height": 4, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1052102755, + "versionNonce": 1632311055, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 260, + 4 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1752469465", + "type": "text", + "x": 550.0, + "y": 312.0, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1868293402, + "versionNonce": 620910391, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stays on-CPU", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stays on-CPU", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a202914998", + "type": "arrow", + "x": 360, + "y": 294, + "width": 160, + "height": 4, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1872779603, + "versionNonce": 1480588741, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 160, + -4 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1343505987", + "type": "text", + "x": 440.0, + "y": 278.0, + "width": 40, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 131282644, + "versionNonce": 904490442, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "never", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "never", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a481972939", + "type": "arrow", + "x": 755, + "y": 392, + "width": 655, + "height": 0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1024499924, + "versionNonce": 153890865, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -655, + 0 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t475699915", + "type": "text", + "x": 427.5, + "y": 378.0, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 461009355, + "versionNonce": 1230844397, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "echo 64B", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "echo 64B", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t984258140", + "type": "text", + "x": 862, + "y": 46, + "width": 199.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1819205170, + "versionNonce": 835522694, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "voluntary_ctxt_switches \u2248 1 / msg", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "voluntary_ctxt_switches \u2248 1 / msg", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t63903316", + "type": "text", + "x": 862, + "y": 246, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 569425243, + "versionNonce": 389714171, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "voluntary_ctxt_switches \u2248 0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "voluntary_ctxt_switches \u2248 0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t955626748", + "type": "text", + "x": 450, + "y": 216, + "width": 607.2, + "height": 15.0, + "angle": 0, + "strokeColor": "#666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1628076520, + "versionNonce": 448645330, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "same 64-byte frame, same round trip \u2014 the difference is who the thread asks for its CPU back", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "same 64-byte frame, same round trip \u2014 the difference is who the thread asks for its CPU back", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/40-fastpath-datapath.svg b/assets/diagrams/40-fastpath-datapath.svg new file mode 100644 index 0000000..0bfe131 --- /dev/null +++ b/assets/diagrams/40-fastpath-datapath.svg @@ -0,0 +1,58 @@ + + + +naive — blocking read, migratable + +fastpath — busy-poll, pinned core + +client +sends 64B + +read(2) — blocks +fresh heap buf / msg + +kernel scheduler +sleep → wake + +write(2) +echo 64B + +client +sends 64B + +busy-poll recv(2) +spin on EAGAIN, one reused buf +pinned: CPU 1 + +scheduler — never entered + +write(2) +echo 64B + +frame +frame + +sleep/wake +sleep/wake + +data ready +data ready + +echo 64B +echo 64B + +frame +frame + +stays on-CPU +stays on-CPU + +never +never + +echo 64B +echo 64B +voluntary_ctxt_switches ≈ 1 / msg +voluntary_ctxt_switches ≈ 0 +same 64-byte frame, same round trip — the difference is who the thread asks for its CPU back + \ No newline at end of file diff --git a/assets/diagrams/README.md b/assets/diagrams/README.md index ed1edda..071c083 100644 --- a/assets/diagrams/README.md +++ b/assets/diagrams/README.md @@ -72,3 +72,10 @@ here as you add them: | `34-image-layering` | 34 | `app`'s multi-stage Containerfile — the discarded UBI10 builder (220 MB measured) installing a toolchain and statically linking the binary, `COPY --from=builder` into the UBI10-micro runtime stage (24.3 MB base) — with the real measured final image sizes (cpp/go 27.2 MB, rust 24.9 MB) barely above the bare micro base | | `34-pid1-signal-flow` | 34 | `serve` vs `naive` as a container's PID 1 receiving SIGTERM — serve's signalfd/Notify/Signals wake, stop-spawning, forward-to-worker, reap, graceful exit sequence beside naive's kernel-masked default action that just drops the signal — annotated with the real measured `podman stop` timings (serve: cpp 192ms/go 60ms/rust 101ms; naive: cpp 2108ms/rust 2074ms SIGKILLed, go 60ms but silent) | | `34-ns-debugging-map` | 34 | the two independent paths that reach the same container PID 1 — `podman exec` from inside the container runtime and a rootless `podman unshare nsenter` from the bare host (plain `nsenter` alone fails EPERM) — both confirming `Name: app` and `/proc/1/exe`, beside the real subuid/subgid mapping and a live zero-zombie scan | +| `39-coordinated-omission` | 39 | benchlab's closed loop against what a fixed-rate caller would have seen — one recorded 1057 ns stall sample on top, the same stall backfilled below into the synthetic queued requests `co_correct` adds, showing why raw p99 buries the stall while `co_p99_ns` (always ≥ `p99_ns`) surfaces it | +| `39-benchmark-pipeline` | 39 | benchlab's real path (warmup discard → per-iteration monotonic timing → sort/percentile and the parallel `co_correct` branch → the printed table) beside `--lie`'s one unwarmed wall-clock call straight to `elapsed_ns` — same codec, same fixed workload, two harnesses telling a different story | +| `38-otlp-pipeline` | 38 | lsp-otel's telemetry path — serve's per-request parent span `request` wrapping `work` + `respond` children plus the `requests_total` counter and `request_duration` histogram, through the OTLP/HTTP exporter (batched ~2s, flushed on SIGINT) posting to the OTel collector on :4318, fanned to Tempo (traces) and Prometheus (metrics) and queried back through Grafana's datasource proxy — with cpp/go/rust each tagging service.name=lsp-otel- so the three never collide in the shared stack | +| `37-use-method-checklist` | 37 | loadmix analyze as a pipeline — four one-shot pre-samples (uptime/dmesg/top/ss) then five interval samplers (vmstat/mpstat/iostat/sar/pidstat) spawned concurrently so an N-second checklist costs ~N seconds not 5N, each parsed by column name from its own header row into the 4×3 USE signal matrix (cpu/mem/io/net × Utilization/Saturation/Errors), and the max-Saturation-ratio verdict — with the saturate side shown as a worker pool the concurrent analyze attributes | +| `40-fastpath-datapath` | 40 | one echo round trip, naive versus fastpath: the naive server's per-message heap buffer and blocking `read(2)` that sleeps through the kernel scheduler (one voluntary context switch per message) against the fastpath server's single reused buffer on a `sched_setaffinity`-pinned core busy-polling `recv(2)` over `EAGAIN`, never entering the scheduler (≈ zero voluntary context switches) — the measured 20500-vs-3 contrast that the chapter reads from `/proc//status` | +| `36-proc-datasource-map` | 36 | the six /proc + cgroup PSI sources sysagent reads (stat, loadavg, meminfo, diskstats, net/dev, PSI), counters marked apart from gauges, feeding `take_snapshot`'s t0-read / concurrent-reader / sleep / t1-read / delta sequence that turns two cumulative `/proc/stat` reads into one `cpu_util_pct` | +| `36-agent-architecture` | 36 | sysagent's `sample`/`serve` subcommands sharing one procfs reader layer, against the row of cross-check tools run at the same moment (vmstat, mpstat, free -m, ss -s, a direct `/proc/meminfo` read, `kill -INT`), with forward references to Chapter 37 (USE method), 38 (OTLP export), and 45 (PCP pmrep) | diff --git a/examples/36-proc-sys-and-the-agent/README.md b/examples/36-proc-sys-and-the-agent/README.md new file mode 100644 index 0000000..8e420ab --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/README.md @@ -0,0 +1,169 @@ +# 36-proc-sys-and-the-agent — sysagent v0 + +Chapter 36 opens Part 9 (observability). `sysagent` is a small USE-method +metrics collector — the same tool the rest of the Part builds on (chapter 37's +saturation drills, chapter 41's capstone fleet) — reading nothing but +`/proc`, `/proc/net`, `/proc/diskstats`, and cgroup PSI. No root, no +`/dev/kmsg`, no eBPF: everything here is a plain file a normal user can open. + +``` +sysagent sample [--json] [--interval-ms N] # one snapshot to stdout +sysagent serve --port P [--interval-ms N] # the same snapshot over HTTP + +opts: --interval-ms N (default 200) cpu utilization sampling window + --json machine-readable output for `sample` +``` + +## Behavior (identical across C++, Go, and Rust) + +- **`sample`** takes exactly one snapshot: it reads `/proc/stat` once, sleeps + `--interval-ms`, reads it again, and reports the CPU delta over that + window — alongside `/proc/loadavg`, `/proc/meminfo`, `/proc/diskstats`, + `/proc/net/dev`, and cgroup PSI, all gathered **concurrently** with the + sleep (C++ `std::jthread`, Go goroutines fanned in over a channel, Rust + scoped threads — three languages, one idea: don't serialize five + independent file reads behind a timer). Default text output is + `key=value` lines; `--json` emits the single-line JSON object below. +- **`serve --port P`** exposes the identical snapshot over a hand-rolled + HTTP/1.1 `GET /metrics` endpoint (Go uses `net/http`; C++ and Rust hand-roll + the same tiny surface on raw sockets + `poll(2)`, in keeping with the + book's earlier chatterd examples) — one fresh sample per request. Any + other path 404s. `SIGINT`/`SIGTERM` prints a shutdown line and exits 0. +- No/bad arguments print a usage line on stderr and exit 2. + +## The deterministic field schema + +Every field name below is byte-identical across `--json` and `/metrics` in +all three languages — that's the contract this example exists to prove out: +you can point one Grafana dashboard or one test suite at any of the three. + +| field | type | source | +|-----------------------|---------|--------------------------------------------------| +| `cpu_util_pct` | float | `/proc/stat` delta: 100 × busy⁄total | +| `cpu_user_pct` | float | `/proc/stat` delta: 100 × user⁄total | +| `cpu_system_pct` | float | `/proc/stat` delta: 100 × system⁄total | +| `load1/5/15` | float | `/proc/loadavg` fields 1–3 | +| `runnable` | int | `/proc/loadavg` "runnable" of runnable/total | +| `total_threads` | int | `/proc/loadavg` "total" of runnable/total | +| `mem_total_kb` | int | `/proc/meminfo` `MemTotal` | +| `mem_available_kb` | int | `/proc/meminfo` `MemAvailable` | +| `mem_used_kb` | int | `mem_total_kb - mem_available_kb` | +| `disks[]` | array | `/proc/diskstats`, one entry per non-loop/ram device: `name`, `reads`, `writes`, `read_sectors`, `write_sectors` | +| `net[]` | array | `/proc/net/dev`, one entry per interface: `iface`, `rx_bytes`, `tx_bytes` | +| `psi_available` | bool | whether any PSI file below was readable | +| `psi_cpu_some_avg10` | float | cgroup/system `cpu.pressure` "some avg10" | +| `psi_mem_some_avg10` | float | cgroup/system `memory.pressure` "some avg10" | +| `psi_io_some_avg10` | float | cgroup/system `io.pressure` "some avg10" | + +CPU percentages are computed the standard `/proc/stat` way: `busy` is +`user + nice + system + irq + softirq + steal`; `idle` is `idle + iowait`; +`total` is their sum, all as **deltas** between the two `--interval-ms`-apart +readings (a single reading of `/proc/stat` is a cumulative counter since +boot, not a percentage). + +PSI (Pressure Stall Information) is read from **this process's own cgroup** +first — resolved via the single `0::` line in `/proc/self/cgroup` +(cgroup v2 unified hierarchy) joined onto `/sys/fs/cgroup/{cpu,memory,io}.pressure` +— falling back to the system-wide `/proc/pressure/{cpu,memory,io}` files if +the cgroup-scoped ones aren't there. If neither is readable (PSI disabled in +the kernel, or a container without a cgroup-v2 delegate), `psi_available` is +`false` and the three `avg10` fields are `0.0` — that's the "if available" +in the chapter spec: a missing PSI source is not an error. + +## Example output + +``` +$ ./demo.sh cpp run sample --json +{"cpu_util_pct":10.69,"cpu_user_pct":9.12,"cpu_system_pct":1.26,"load1":1.21, + "load5":1.20,"load15":0.91,"runnable":3,"total_threads":3523, + "mem_total_kb":65545860,"mem_available_kb":36847108,"mem_used_kb":28698752, + "disks":[{"name":"nvme0n1","reads":698032,"writes":19501729, + "read_sectors":55243425,"write_sectors":738027946}, ...], + "net":[{"iface":"lo","rx_bytes":65086669,"tx_bytes":65086669}, ...], + "psi_available":true,"psi_cpu_some_avg10":0.09,"psi_mem_some_avg10":0.00, + "psi_io_some_avg10":0.00} + +$ ./demo.sh go run serve --port 9100 & +sysagent: listening on 0.0.0.0:9100 +$ curl -s http://127.0.0.1:9100/metrics | head -c 120 +{"cpu_util_pct":8.18,"cpu_user_pct":4.09,"cpu_system_pct":3.46,"load1":1.27,"load5":1.21,"load15":0.92,"runnable":3,"total_threads +$ kill -INT %1 +sysagent: shutting down +``` + +(That `sample` line reflects real load on the box this was written on — a +concurrent `cc1plus`-heavy build was running in the background, which is +exactly the kind of noise the verification below is built to tolerate.) + +## One subject, three concurrency styles + +The `/proc` parsing itself is deliberately unglamorous — `open`, read lines, +split fields — so the interesting difference is how each language overlaps +the five independent reads with the CPU-delta sampling window instead of +doing five sequential blocking calls: + +- **C++** — `std::jthread` per source (`loadavg`, `meminfo`, `diskstats`, + `netdev`, `psi`), an `std::atomic` success counter, and a small + `ProcFile` RAII wrapper around `std::ifstream` so every `/proc` handle + closes itself. Errors flow through `std::expected`. +- **Go** — one goroutine per source, fanned in over a single buffered + `chan namedErr` — the idiomatic Go answer to "wait for N concurrent + results and keep the first error." Errors are wrapped with `%w` throughout. +- **Rust** — `std::thread::scope` with one scoped closure per source, so the + borrows into the final `Snapshot` don't need `Arc`/`Mutex`; each closure + returns a `Result` collected via `JoinHandle::join()`. Errors + flow through `Result`/`?`. + +The HTTP layer follows the same three-languages-one-idea pattern as the +book's chatterd examples: Go leans on its stdlib `net/http`; C++ and Rust +hand-roll the request line + `poll(2)` + `signalfd`/`SignalFd` shutdown +doorbell that chapter 24 already established, because there's no reason to +pull in a framework for one endpoint. + +## Layout + +``` +36-proc-sys-and-the-agent/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated behavioral check (schema, cpu-busy delta, serve) +├── cpp/ # procfs.{hpp,cpp} + httpd.{hpp,cpp} + main.cpp, demo.sh +├── go/ # procfs.go + httpd.go + main.go, demo.sh +└── rust/ # src/procfs.rs + src/httpd.rs + src/main.rs, demo.sh +``` + +## The demo contract + +Each language directory has a `demo.sh` with the identical interface used +book-wide: `./demo.sh build`, `./demo.sh run [args]`, and a bare `./demo.sh` +that builds then runs. With `TARGET` set, `run` deploys to that lab VM +instead of running locally — unused here since this chapter is local-only +(`mode: local` in the manifest); it reads the current host's `/proc`, not a +VM's. `run` execs straight into the built binary so signals (`SIGINT` on +`serve`) reach it directly rather than a wrapping shell. + +## Verification + +`verify.lua` (run by the test harness with `LSP_LANG` set to one language at +a time) asserts observable behavior, not exit 0: + +1. Usage errors — no arguments, an unknown subcommand, `serve` without + `--port` — all exit 2 with a usage line. +2. `sample --json` carries every field in the schema above with a numeric + (or boolean, for `psi_available`) value, `mem_total_kb` is positive, + `cpu_util_pct` is in `[0, 100]`, the `disks` array has a named entry, and + the `net` array includes `lo`. +3. **A cpu-busy child measurably raises `cpu_util_pct` versus an idle + baseline.** The check spins up `nproc` tight busy loops for slightly + longer than a `sample --interval-ms 500` window and compares against an + immediately-prior idle sample, retried up to three times with a 12 + percentage-point margin (or an absolute ≥55% floor, for the case where the + host is already saturated) — robust against running on a real, shared + development box rather than an isolated benchmark rig. +4. `serve --port P` answers `GET /metrics` with the same JSON shape, 404s an + unknown path, and its `mem_total_kb` agrees **exactly** with a direct + `awk '/^MemTotal:/{print $2}' /proc/meminfo` read in the same process + (a static fact, unlike load or disk counters, so an exact match is the + right bar); `SIGINT` shuts it down cleanly (exit 0), logging the listen + address on start and a shutdown line on the way out. + +Exit 0 = pass, 1 = fail, 77 = skip. diff --git a/examples/36-proc-sys-and-the-agent/cpp/.clang-format b/examples/36-proc-sys-and-the-agent/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/36-proc-sys-and-the-agent/cpp/.clang-tidy b/examples/36-proc-sys-and-the-agent/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/36-proc-sys-and-the-agent/cpp/CMakeLists.txt b/examples/36-proc-sys-and-the-agent/cpp/CMakeLists.txt new file mode 100644 index 0000000..7fe4828 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/CMakeLists.txt @@ -0,0 +1,15 @@ +cmake_minimum_required(VERSION 3.25) +project(sysagent CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app + src/main.cpp + src/procfs.cpp + src/httpd.cpp +) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/36-proc-sys-and-the-agent/cpp/CMakePresets.json b/examples/36-proc-sys-and-the-agent/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/36-proc-sys-and-the-agent/cpp/demo.sh b/examples/36-proc-sys-and-the-agent/cpp/demo.sh new file mode 100755 index 0000000..81d3103 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/httpd.cpp b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.cpp new file mode 100644 index 0000000..7e6cd2a --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.cpp @@ -0,0 +1,154 @@ +#include "httpd.hpp" + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "procfs.hpp" + +namespace sysagent { + +namespace { + +// Same RAII-fd idiom used throughout the book: never a naked close(2). +class UniqueFd { + public: + UniqueFd() = default; + explicit UniqueFd(int fd) : fd_(fd) {} + UniqueFd(const UniqueFd&) = delete; + UniqueFd& operator=(const UniqueFd&) = delete; + UniqueFd(UniqueFd&& other) noexcept : fd_(std::exchange(other.fd_, -1)) {} + UniqueFd& operator=(UniqueFd&& other) noexcept { + if (this != &other) { + reset(); + fd_ = std::exchange(other.fd_, -1); + } + return *this; + } + ~UniqueFd() { reset(); } + [[nodiscard]] int get() const { return fd_; } + void reset() { + if (fd_ >= 0) ::close(fd_); + fd_ = -1; + } + + private: + int fd_ = -1; +}; + +void send_all(int fd, const std::string& data) { + size_t sent = 0; + while (sent < data.size()) { + auto n = ::send(fd, data.data() + sent, data.size() - sent, MSG_NOSIGNAL); + if (n <= 0) return; + sent += static_cast(n); + } +} + +std::string http_response(int status, std::string_view status_text, std::string_view content_type, + const std::string& body) { + return std::format("HTTP/1.1 {} {}\r\nContent-Type: {}\r\nContent-Length: {}\r\n" + "Connection: close\r\n\r\n{}", + status, status_text, content_type, body.size(), body); +} + +void handle_client(int cfd, int interval_ms) { + char buf[4096]; + auto n = ::recv(cfd, buf, sizeof(buf) - 1, 0); + if (n <= 0) return; + buf[n] = '\0'; + std::string_view req(buf, static_cast(n)); + + // Only care about the request line: "METHOD PATH HTTP/1.1". + auto line_end = req.find("\r\n"); + std::string_view line = line_end == std::string_view::npos ? req : req.substr(0, line_end); + auto sp1 = line.find(' '); + auto sp2 = sp1 == std::string_view::npos ? std::string_view::npos : line.find(' ', sp1 + 1); + std::string_view method = sp1 == std::string_view::npos ? "" : line.substr(0, sp1); + std::string_view path = + (sp1 == std::string_view::npos || sp2 == std::string_view::npos) + ? "" + : line.substr(sp1 + 1, sp2 - sp1 - 1); + + if (method == "GET" && path == "/metrics") { + auto snap = take_snapshot(interval_ms); + if (snap) { + send_all(cfd, http_response(200, "OK", "application/json", to_json(*snap))); + } else { + send_all(cfd, http_response(500, "Internal Server Error", "text/plain", snap.error())); + } + } else { + send_all(cfd, http_response(404, "Not Found", "text/plain", "not found\n")); + } +} + +} // namespace + +int serve(int port, int interval_ms) { + UniqueFd listen_fd(::socket(AF_INET, SOCK_STREAM, 0)); + if (listen_fd.get() < 0) { + std::println(stderr, "sysagent: socket: {}", std::strerror(errno)); + return 1; + } + int one = 1; + ::setsockopt(listen_fd.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_ANY); + addr.sin_port = htons(static_cast(port)); + if (::bind(listen_fd.get(), reinterpret_cast(&addr), sizeof(addr)) != 0) { + std::println(stderr, "sysagent: bind: {}", std::strerror(errno)); + return 1; + } + if (::listen(listen_fd.get(), 16) != 0) { + std::println(stderr, "sysagent: listen: {}", std::strerror(errno)); + return 1; + } + + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + sigprocmask(SIG_BLOCK, &mask, nullptr); + UniqueFd sig_fd(::signalfd(-1, &mask, SFD_NONBLOCK)); + + std::println("sysagent: listening on 0.0.0.0:{}", port); + + pollfd fds[2] = { + {.fd = listen_fd.get(), .events = POLLIN, .revents = 0}, + {.fd = sig_fd.get(), .events = POLLIN, .revents = 0}, + }; + + for (;;) { + int rc = ::poll(fds, 2, -1); + if (rc < 0) { + if (errno == EINTR) continue; + std::println(stderr, "sysagent: poll: {}", std::strerror(errno)); + return 1; + } + if (fds[1].revents & POLLIN) { + signalfd_siginfo info{}; + (void)::read(sig_fd.get(), &info, sizeof(info)); + std::println("sysagent: shutting down"); + return 0; + } + if (fds[0].revents & POLLIN) { + UniqueFd cfd(::accept(listen_fd.get(), nullptr, nullptr)); + if (cfd.get() >= 0) handle_client(cfd.get(), interval_ms); + } + } +} + +} // namespace sysagent diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp new file mode 100644 index 0000000..b265d0b --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/src/httpd.hpp @@ -0,0 +1,12 @@ +// sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework — +// same philosophy as the raw-socket chatterd examples earlier in the book. +#pragma once + +namespace sysagent { + +// Serve GET /metrics (JSON snapshot, see procfs.hpp) on 0.0.0.0:port until +// SIGINT/SIGTERM. Each request takes a fresh snapshot with the given +// interval_ms. Returns the process exit code (0 on a clean signal shutdown). +[[nodiscard]] int serve(int port, int interval_ms); + +} // namespace sysagent diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp b/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp new file mode 100644 index 0000000..beb11b7 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/src/main.cpp @@ -0,0 +1,101 @@ +// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// metrics collector reading /proc + /sys/fs/cgroup, no root required. +// +// sysagent sample [--json] [--interval-ms N] +// sysagent serve --port P [--interval-ms N] +// +// `sample` takes one snapshot: CPU utilization from an /proc/stat delta +// spanning --interval-ms (default 200), run-queue + load from +// /proc/loadavg, memory from /proc/meminfo, per-disk I/O from +// /proc/diskstats, network rx/tx from /proc/net/dev, and cgroup PSI +// (falling back to system-wide /proc/pressure) if the kernel exposes it. +// `serve` exposes the identical snapshot as JSON over a hand-rolled +// HTTP/1.1 /metrics endpoint, one fresh sample per request. +// +// The field names in both the --json and /metrics output are the +// deterministic, cross-language schema documented in README.md — Go and +// Rust sysagent emit byte-for-byte the same keys. +#include +#include +#include +#include +#include +#include + +#include "httpd.hpp" +#include "procfs.hpp" + +namespace { + +void usage() { + std::println(stderr, "usage: sysagent sample [--json] [--interval-ms N] | " + "serve --port P [--interval-ms N]"); +} + +// Minimal hand-rolled flag scan — no getopt_long, to keep the three +// languages' argument handling directly comparable. +struct Flags { + bool json = false; + int interval_ms = 200; + int port = -1; +}; + +bool parse_flags(int argc, char** argv, int start, Flags& out) { + for (int i = start; i < argc; ++i) { + std::string_view a = argv[i]; + if (a == "--json") { + out.json = true; + } else if (a == "--interval-ms") { + if (i + 1 >= argc) return false; + out.interval_ms = std::atoi(argv[++i]); + } else if (a == "--port") { + if (i + 1 >= argc) return false; + out.port = std::atoi(argv[++i]); + } else { + return false; + } + } + return true; +} + +int cmd_sample(int argc, char** argv) { + Flags flags; + if (!parse_flags(argc, argv, 2, flags) || flags.interval_ms <= 0) { + usage(); + return 2; + } + auto snap = sysagent::take_snapshot(flags.interval_ms); + if (!snap) { + std::println(stderr, "sysagent: error: {}", snap.error()); + return 1; + } + if (flags.json) { + std::println("{}", sysagent::to_json(*snap)); + } else { + std::print("{}", sysagent::to_text(*snap)); + } + return 0; +} + +int cmd_serve(int argc, char** argv) { + Flags flags; + if (!parse_flags(argc, argv, 2, flags) || flags.port <= 0 || flags.interval_ms <= 0) { + usage(); + return 2; + } + return sysagent::serve(flags.port, flags.interval_ms); +} + +} // namespace + +int main(int argc, char** argv) { + if (argc < 2) { + usage(); + return 2; + } + std::string_view cmd = argv[1]; + if (cmd == "sample") return cmd_sample(argc, argv); + if (cmd == "serve") return cmd_serve(argc, argv); + usage(); + return 2; +} diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp new file mode 100644 index 0000000..8f16dd8 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.cpp @@ -0,0 +1,388 @@ +#include "procfs.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace std::chrono_literals; + +namespace sysagent { + +namespace { + +// RAII wrapper around an ifstream: names the path (for error messages) and +// guarantees the fd is closed when the wrapper goes out of scope, exactly +// like every other UniqueFd-style RAII type in this book — just for the +// stdio-buffered stream form /proc parsing normally uses. +class ProcFile { + public: + explicit ProcFile(std::string path) : path_(std::move(path)), stream_(path_) {} + + [[nodiscard]] bool ok() const { return stream_.is_open(); } + [[nodiscard]] const std::string& path() const { return path_; } + std::ifstream& stream() { return stream_; } + + private: + std::string path_; + std::ifstream stream_; +}; + +[[nodiscard]] std::string open_error(const ProcFile& f) { + return "cannot open " + f.path(); +} + +// double from a "some avg10=X.XX avg60=Y.YY avg300=Z.ZZ total=N" PSI line. +double parse_avg10(const std::string& line) { + double v = 0.0; + std::sscanf(line.c_str(), "%*s avg10=%lf", &v); + return v; +} + +std::optional read_psi_from_dir(const std::string& dir) { + Psi psi{}; + bool any = false; + for (auto [file, out] : { + std::pair{"cpu.pressure", &psi.cpu_some_avg10}, + std::pair{"memory.pressure", &psi.mem_some_avg10}, + std::pair{"io.pressure", &psi.io_some_avg10}, + }) { + ProcFile f(dir + "/" + file); + if (!f.ok()) continue; + std::string line; + if (std::getline(f.stream(), line)) { + *out = parse_avg10(line); + any = true; + } + } + return any ? std::optional{psi} : std::nullopt; +} + +std::optional read_psi_from_proc_pressure() { + Psi psi{}; + bool any = false; + for (auto [file, out] : { + std::pair{"/proc/pressure/cpu", &psi.cpu_some_avg10}, + std::pair{"/proc/pressure/memory", &psi.mem_some_avg10}, + std::pair{"/proc/pressure/io", &psi.io_some_avg10}, + }) { + ProcFile f(file); + if (!f.ok()) continue; + std::string line; + if (std::getline(f.stream(), line)) { + *out = parse_avg10(line); + any = true; + } + } + return any ? std::optional{psi} : std::nullopt; +} + +} // namespace + +std::expected read_cpu_ticks() { + ProcFile f("/proc/stat"); + if (!f.ok()) return std::unexpected(open_error(f)); + std::string line; + if (!std::getline(f.stream(), line)) { + return std::unexpected("/proc/stat: empty"); + } + // "cpu user nice system idle iowait irq softirq steal guest guest_nice" + std::istringstream iss(line); + std::string label; + CpuTicks t{}; + iss >> label >> t.user >> t.nice >> t.system >> t.idle >> t.iowait >> t.irq >> t.softirq >> + t.steal; + if (label != "cpu") return std::unexpected("/proc/stat: unexpected first line"); + return t; +} + +std::expected read_loadavg() { + ProcFile f("/proc/loadavg"); + if (!f.ok()) return std::unexpected(open_error(f)); + LoadAvg la{}; + std::string running_total; + long long last_pid = 0; + f.stream() >> la.load1 >> la.load5 >> la.load15 >> running_total >> last_pid; + if (!f.stream() && !f.stream().eof()) return std::unexpected("/proc/loadavg: parse error"); + auto slash = running_total.find('/'); + if (slash == std::string::npos) return std::unexpected("/proc/loadavg: bad running/total field"); + la.runnable = std::stoll(running_total.substr(0, slash)); + la.total_threads = std::stoll(running_total.substr(slash + 1)); + return la; +} + +std::expected read_meminfo() { + ProcFile f("/proc/meminfo"); + if (!f.ok()) return std::unexpected(open_error(f)); + MemInfo mi{}; + bool have_total = false, have_avail = false; + std::string line; + while (std::getline(f.stream(), line)) { + std::istringstream iss(line); + std::string key; + int64_t kb = 0; + iss >> key >> kb; + if (key == "MemTotal:") { + mi.total_kb = kb; + have_total = true; + } else if (key == "MemAvailable:") { + mi.available_kb = kb; + have_avail = true; + } + if (have_total && have_avail) break; + } + if (!have_total || !have_avail) return std::unexpected("/proc/meminfo: missing keys"); + return mi; +} + +std::expected, std::string> read_diskstats() { + ProcFile f("/proc/diskstats"); + if (!f.ok()) return std::unexpected(open_error(f)); + std::vector out; + std::string line; + while (std::getline(f.stream(), line)) { + std::istringstream iss(line); + int64_t major = 0, minor = 0; + std::string name; + int64_t reads_completed = 0, reads_merged = 0, sectors_read = 0, ms_reading = 0; + int64_t writes_completed = 0, writes_merged = 0, sectors_written = 0; + iss >> major >> minor >> name >> reads_completed >> reads_merged >> sectors_read >> + ms_reading >> writes_completed >> writes_merged >> sectors_written; + if (name.empty()) continue; + // Skip loop/ram devices and partitions with no traffic at all — keeps + // the array focused on the disks that actually matter for USE-method + // reads, without hardcoding any device-naming scheme. + if (name.starts_with("loop") || name.starts_with("ram")) continue; + out.push_back(DiskStat{ + .name = name, + .reads = reads_completed, + .writes = writes_completed, + .read_sectors = sectors_read, + .write_sectors = sectors_written, + }); + } + return out; +} + +std::expected, std::string> read_netdev() { + ProcFile f("/proc/net/dev"); + if (!f.ok()) return std::unexpected(open_error(f)); + std::vector out; + std::string line; + int lineno = 0; + while (std::getline(f.stream(), line)) { + ++lineno; + if (lineno <= 2) continue; // two header lines + auto colon = line.find(':'); + if (colon == std::string::npos) continue; + std::string iface = line.substr(0, colon); + iface.erase(std::remove(iface.begin(), iface.end(), ' '), iface.end()); + std::istringstream iss(line.substr(colon + 1)); + int64_t rx_bytes = 0, tx_bytes = 0, field = 0; + iss >> rx_bytes; + for (int i = 0; i < 7; ++i) iss >> field; // rx packets..rx multicast + iss >> tx_bytes; + out.push_back(NetStat{.iface = iface, .rx_bytes = rx_bytes, .tx_bytes = tx_bytes}); + } + return out; +} + +std::optional read_psi() { + // Prefer this process's own cgroup (v2 unified hierarchy: a single + // "0::/path" line in /proc/self/cgroup) so the numbers reflect exactly + // the pressure this program's cgroup is under. + ProcFile self_cg("/proc/self/cgroup"); + if (self_cg.ok()) { + std::string line; + while (std::getline(self_cg.stream(), line)) { + if (!line.starts_with("0::")) continue; + std::string rel = line.substr(3); + if (auto psi = read_psi_from_dir("/sys/fs/cgroup" + rel)) return psi; + break; + } + } + // Fall back to the system-wide files (still cgroup2 PSI under the hood). + return read_psi_from_proc_pressure(); +} + +std::expected take_snapshot(int interval_ms) { + auto t0 = read_cpu_ticks(); + if (!t0) return std::unexpected(t0.error()); + + // Gather the non-CPU sources concurrently with the sampling sleep, using + // jthreads (auto-joining, cooperatively cancellable) rather than raw + // pthreads; ok_count is an atomic so every worker can report success + // without a mutex. + Snapshot snap{}; + std::atomic ok_count{0}; + std::string first_error; + std::mutex err_mu; + auto note_error = [&](std::string msg) { + std::lock_guard lock(err_mu); + if (first_error.empty()) first_error = std::move(msg); + }; + + std::jthread load_thread([&] { + if (auto la = read_loadavg()) { + snap.load1 = la->load1; + snap.load5 = la->load5; + snap.load15 = la->load15; + snap.runnable = la->runnable; + snap.total_threads = la->total_threads; + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(la.error()); + } + }); + std::jthread mem_thread([&] { + if (auto mi = read_meminfo()) { + snap.mem_total_kb = mi->total_kb; + snap.mem_available_kb = mi->available_kb; + snap.mem_used_kb = mi->total_kb - mi->available_kb; + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(mi.error()); + } + }); + std::jthread disk_thread([&] { + if (auto ds = read_diskstats()) { + snap.disks = std::move(*ds); + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(ds.error()); + } + }); + std::jthread net_thread([&] { + if (auto ns = read_netdev()) { + snap.net = std::move(*ns); + ok_count.fetch_add(1, std::memory_order_relaxed); + } else { + note_error(ns.error()); + } + }); + std::jthread psi_thread([&] { + if (auto psi = read_psi()) { + snap.psi_available = true; + snap.psi_cpu_some_avg10 = psi->cpu_some_avg10; + snap.psi_mem_some_avg10 = psi->mem_some_avg10; + snap.psi_io_some_avg10 = psi->io_some_avg10; + } + ok_count.fetch_add(1, std::memory_order_relaxed); + }); + + std::this_thread::sleep_for(std::chrono::milliseconds(interval_ms)); + + // jthread destructors join as each goes out of scope at the end of this + // block; explicitly joining here just makes the ordering-before-t1 clear. + load_thread.join(); + mem_thread.join(); + disk_thread.join(); + net_thread.join(); + psi_thread.join(); + + if (ok_count.load(std::memory_order_relaxed) < 4) { + return std::unexpected(first_error.empty() ? "a /proc source failed" : first_error); + } + + auto t1 = read_cpu_ticks(); + if (!t1) return std::unexpected(t1.error()); + + auto busy0 = t0->user + t0->nice + t0->system; + auto busy1 = t1->user + t1->nice + t1->system; + auto idle0 = t0->idle + t0->iowait; + auto idle1 = t1->idle + t1->iowait; + auto total0 = busy0 + idle0 + t0->irq + t0->softirq + t0->steal; + auto total1 = busy1 + idle1 + t1->irq + t1->softirq + t1->steal; + + auto d_total = static_cast(total1 - total0); + auto d_busy = static_cast((busy1 - busy0) + (t1->irq - t0->irq) + + (t1->softirq - t0->softirq) + (t1->steal - t0->steal)); + auto d_user = static_cast(t1->user - t0->user); + auto d_system = static_cast(t1->system - t0->system); + + if (d_total > 0.0) { + snap.cpu_util_pct = 100.0 * d_busy / d_total; + snap.cpu_user_pct = 100.0 * d_user / d_total; + snap.cpu_system_pct = 100.0 * d_system / d_total; + } + + return snap; +} + +namespace { + +std::string fmt2(double v) { + char buf[64]; + std::snprintf(buf, sizeof(buf), "%.2f", v); + return buf; +} + +} // namespace + +std::string to_json(const Snapshot& s) { + std::ostringstream o; + o << "{"; + o << "\"cpu_util_pct\":" << fmt2(s.cpu_util_pct) << ","; + o << "\"cpu_user_pct\":" << fmt2(s.cpu_user_pct) << ","; + o << "\"cpu_system_pct\":" << fmt2(s.cpu_system_pct) << ","; + o << "\"load1\":" << fmt2(s.load1) << ","; + o << "\"load5\":" << fmt2(s.load5) << ","; + o << "\"load15\":" << fmt2(s.load15) << ","; + o << "\"runnable\":" << s.runnable << ","; + o << "\"total_threads\":" << s.total_threads << ","; + o << "\"mem_total_kb\":" << s.mem_total_kb << ","; + o << "\"mem_available_kb\":" << s.mem_available_kb << ","; + o << "\"mem_used_kb\":" << s.mem_used_kb << ","; + o << "\"disks\":["; + for (size_t i = 0; i < s.disks.size(); ++i) { + const auto& d = s.disks[i]; + if (i) o << ","; + o << "{\"name\":\"" << d.name << "\",\"reads\":" << d.reads << ",\"writes\":" << d.writes + << ",\"read_sectors\":" << d.read_sectors << ",\"write_sectors\":" << d.write_sectors + << "}"; + } + o << "],"; + o << "\"net\":["; + for (size_t i = 0; i < s.net.size(); ++i) { + const auto& n = s.net[i]; + if (i) o << ","; + o << "{\"iface\":\"" << n.iface << "\",\"rx_bytes\":" << n.rx_bytes + << ",\"tx_bytes\":" << n.tx_bytes << "}"; + } + o << "],"; + o << "\"psi_available\":" << (s.psi_available ? "true" : "false") << ","; + o << "\"psi_cpu_some_avg10\":" << fmt2(s.psi_cpu_some_avg10) << ","; + o << "\"psi_mem_some_avg10\":" << fmt2(s.psi_mem_some_avg10) << ","; + o << "\"psi_io_some_avg10\":" << fmt2(s.psi_io_some_avg10); + o << "}"; + return o.str(); +} + +std::string to_text(const Snapshot& s) { + std::ostringstream o; + o << "cpu_util_pct=" << fmt2(s.cpu_util_pct) << " cpu_user_pct=" << fmt2(s.cpu_user_pct) + << " cpu_system_pct=" << fmt2(s.cpu_system_pct) << "\n"; + o << "load1=" << fmt2(s.load1) << " load5=" << fmt2(s.load5) << " load15=" << fmt2(s.load15) + << " runnable=" << s.runnable << " total_threads=" << s.total_threads << "\n"; + o << "mem_total_kb=" << s.mem_total_kb << " mem_available_kb=" << s.mem_available_kb + << " mem_used_kb=" << s.mem_used_kb << "\n"; + for (const auto& d : s.disks) { + o << "disk name=" << d.name << " reads=" << d.reads << " writes=" << d.writes + << " read_sectors=" << d.read_sectors << " write_sectors=" << d.write_sectors << "\n"; + } + for (const auto& n : s.net) { + o << "net iface=" << n.iface << " rx_bytes=" << n.rx_bytes << " tx_bytes=" << n.tx_bytes + << "\n"; + } + o << "psi_available=" << (s.psi_available ? "true" : "false") + << " psi_cpu_some_avg10=" << fmt2(s.psi_cpu_some_avg10) + << " psi_mem_some_avg10=" << fmt2(s.psi_mem_some_avg10) + << " psi_io_some_avg10=" << fmt2(s.psi_io_some_avg10) << "\n"; + return o.str(); +} + +} // namespace sysagent diff --git a/examples/36-proc-sys-and-the-agent/cpp/src/procfs.hpp b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.hpp new file mode 100644 index 0000000..d6b111d --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/cpp/src/procfs.hpp @@ -0,0 +1,104 @@ +// sysagent: types + readers for /proc, /sys/fs/cgroup PSI, and the snapshot +// they compose into. See procfs.cpp for the parsing. +#pragma once + +#include +#include +#include +#include +#include + +namespace sysagent { + +struct DiskStat { + std::string name; + int64_t reads = 0; + int64_t writes = 0; + int64_t read_sectors = 0; + int64_t write_sectors = 0; +}; + +struct NetStat { + std::string iface; + int64_t rx_bytes = 0; + int64_t tx_bytes = 0; +}; + +// One fully-formed sample. Field names here are the canonical, deterministic +// names shared with the Go and Rust implementations (see README.md). +struct Snapshot { + double cpu_util_pct = 0.0; + double cpu_user_pct = 0.0; + double cpu_system_pct = 0.0; + + double load1 = 0.0; + double load5 = 0.0; + double load15 = 0.0; + int64_t runnable = 0; + int64_t total_threads = 0; + + int64_t mem_total_kb = 0; + int64_t mem_available_kb = 0; + int64_t mem_used_kb = 0; + + std::vector disks; + std::vector net; + + bool psi_available = false; + double psi_cpu_some_avg10 = 0.0; + double psi_mem_some_avg10 = 0.0; + double psi_io_some_avg10 = 0.0; +}; + +// Raw /proc/stat "cpu " aggregate line, in jiffies. +struct CpuTicks { + int64_t user = 0; + int64_t nice = 0; + int64_t system = 0; + int64_t idle = 0; + int64_t iowait = 0; + int64_t irq = 0; + int64_t softirq = 0; + int64_t steal = 0; +}; + +[[nodiscard]] std::expected read_cpu_ticks(); + +struct LoadAvg { + double load1 = 0.0; + double load5 = 0.0; + double load15 = 0.0; + int64_t runnable = 0; + int64_t total_threads = 0; +}; +[[nodiscard]] std::expected read_loadavg(); + +struct MemInfo { + int64_t total_kb = 0; + int64_t available_kb = 0; +}; +[[nodiscard]] std::expected read_meminfo(); + +[[nodiscard]] std::expected, std::string> read_diskstats(); +[[nodiscard]] std::expected, std::string> read_netdev(); + +struct Psi { + double cpu_some_avg10 = 0.0; + double mem_some_avg10 = 0.0; + double io_some_avg10 = 0.0; +}; +// Cgroup-scoped PSI when the calling process's cgroup exposes pressure files, +// falling back to system-wide /proc/pressure/*. std::nullopt if PSI is not +// available on this kernel/host at all (no error text: this is expected). +[[nodiscard]] std::optional read_psi(); + +// Take deltas of two CpuTicks readings `interval_ms` apart and fill in every +// field of a Snapshot. This is the one function main.cpp calls for `sample`. +[[nodiscard]] std::expected take_snapshot(int interval_ms); + +// Render as the deterministic single-line JSON object (see README.md). +[[nodiscard]] std::string to_json(const Snapshot& s); +// Render as human-readable key=value lines. +[[nodiscard]] std::string to_text(const Snapshot& s); + +} // namespace sysagent diff --git a/examples/36-proc-sys-and-the-agent/demo.sh b/examples/36-proc-sys-and-the-agent/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/36-proc-sys-and-the-agent/go/demo.sh b/examples/36-proc-sys-and-the-agent/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/36-proc-sys-and-the-agent/go/go.mod b/examples/36-proc-sys-and-the-agent/go/go.mod new file mode 100644 index 0000000..498738d --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/36-proc-sys-and-the-agent/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/36-proc-sys-and-the-agent/go/httpd.go b/examples/36-proc-sys-and-the-agent/go/httpd.go new file mode 100644 index 0000000..2b2b6d0 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/go/httpd.go @@ -0,0 +1,65 @@ +// sysagent: the /metrics HTTP endpoint, using net/http (Go's stdlib server — +// the C++ and Rust versions hand-roll the same tiny surface on raw sockets). +package main + +import ( + "context" + "errors" + "fmt" + "net" + "net/http" + "os" + "os/signal" + "syscall" + "time" +) + +// Serve exposes GET /metrics (JSON snapshot) on 0.0.0.0:port until +// SIGINT/SIGTERM. Each request takes a fresh snapshot with the given +// intervalMs. Returns the process exit code (0 on a clean signal shutdown). +func Serve(port, intervalMs int) int { + mux := http.NewServeMux() + mux.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet { + http.NotFound(w, r) + return + } + snap, err := TakeSnapshot(intervalMs) + if err != nil { + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + w.Header().Set("Content-Type", "application/json") + fmt.Fprintln(w, ToJSON(snap)) + }) + + ln, err := net.Listen("tcp", fmt.Sprintf("0.0.0.0:%d", port)) + if err != nil { + fmt.Fprintf(os.Stderr, "sysagent: listen: %v\n", err) + return 1 + } + srv := &http.Server{Handler: mux} + + serveErr := make(chan error, 1) + go func() { serveErr <- srv.Serve(ln) }() + + fmt.Printf("sysagent: listening on 0.0.0.0:%d\n", port) + + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + + select { + case <-sigCh: + fmt.Println("sysagent: shutting down") + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(ctx) + return 0 + case err := <-serveErr: + if err != nil && !errors.Is(err, http.ErrServerClosed) { + fmt.Fprintf(os.Stderr, "sysagent: serve: %v\n", err) + return 1 + } + return 0 + } +} diff --git a/examples/36-proc-sys-and-the-agent/go/main.go b/examples/36-proc-sys-and-the-agent/go/main.go new file mode 100644 index 0000000..37eef7e --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/go/main.go @@ -0,0 +1,116 @@ +// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// metrics collector reading /proc + /sys/fs/cgroup, no root required. +// +// sysagent sample [--json] [--interval-ms N] +// sysagent serve --port P [--interval-ms N] +// +// `sample` takes one snapshot: CPU utilization from a /proc/stat delta +// spanning --interval-ms (default 200), run-queue + load from +// /proc/loadavg, memory from /proc/meminfo, per-disk I/O from +// /proc/diskstats, network rx/tx from /proc/net/dev, and cgroup PSI +// (falling back to system-wide /proc/pressure) if the kernel exposes it. +// `serve` exposes the identical snapshot as JSON over an HTTP /metrics +// endpoint, one fresh sample per request. +// +// The field names in both the --json and /metrics output are the +// deterministic, cross-language schema documented in README.md — C++ and +// Rust sysagent emit byte-for-byte the same keys. +package main + +import ( + "fmt" + "os" + "strconv" +) + +func usage() { + fmt.Fprintln(os.Stderr, + "usage: sysagent sample [--json] [--interval-ms N] | serve --port P [--interval-ms N]") +} + +// flags is the minimal hand-rolled scan shared in shape with the C++ and +// Rust argument handling, so the three stay directly comparable. +type flags struct { + json bool + intervalMs int + port int +} + +func parseFlags(args []string) (flags, bool) { + f := flags{intervalMs: 200, port: -1} + for i := 0; i < len(args); i++ { + switch args[i] { + case "--json": + f.json = true + case "--interval-ms": + if i+1 >= len(args) { + return f, false + } + i++ + v, err := strconv.Atoi(args[i]) + if err != nil { + return f, false + } + f.intervalMs = v + case "--port": + if i+1 >= len(args) { + return f, false + } + i++ + v, err := strconv.Atoi(args[i]) + if err != nil { + return f, false + } + f.port = v + default: + return f, false + } + } + return f, true +} + +func cmdSample(args []string) int { + f, ok := parseFlags(args) + if !ok || f.intervalMs <= 0 { + usage() + return 2 + } + snap, err := TakeSnapshot(f.intervalMs) + if err != nil { + fmt.Fprintln(os.Stderr, "sysagent: error:", err) + return 1 + } + if f.json { + fmt.Println(ToJSON(snap)) + } else { + fmt.Print(ToText(snap)) + } + return 0 +} + +func cmdServe(args []string) int { + f, ok := parseFlags(args) + if !ok || f.port <= 0 || f.intervalMs <= 0 { + usage() + return 2 + } + return Serve(f.port, f.intervalMs) +} + +func main() { + if len(os.Args) < 2 { + usage() + os.Exit(2) + } + var code int + switch os.Args[1] { + case "sample": + code = cmdSample(os.Args[2:]) + case "serve": + code = cmdServe(os.Args[2:]) + default: + usage() + code = 2 + } + os.Exit(code) +} diff --git a/examples/36-proc-sys-and-the-agent/go/procfs.go b/examples/36-proc-sys-and-the-agent/go/procfs.go new file mode 100644 index 0000000..b736ef5 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/go/procfs.go @@ -0,0 +1,457 @@ +// sysagent: /proc + cgroup PSI readers and the snapshot they compose into. +package main + +import ( + "bufio" + "fmt" + "os" + "strconv" + "strings" + "time" +) + +// DiskStat is one line of /proc/diskstats reduced to the USE-method fields. +type DiskStat struct { + Name string + Reads int64 + Writes int64 + ReadSectors int64 + WriteSectors int64 +} + +// NetStat is one interface's counters from /proc/net/dev. +type NetStat struct { + Iface string + RxBytes int64 + TxBytes int64 +} + +// Snapshot is one fully-formed sample. Field names here are the canonical, +// deterministic names shared with the C++ and Rust implementations (see +// README.md) — only the encoding (Go struct vs C++ struct vs Rust struct) +// differs. +type Snapshot struct { + CPUUtilPct float64 + CPUUserPct float64 + CPUSystemPct float64 + + Load1 float64 + Load5 float64 + Load15 float64 + Runnable int64 + TotalThreads int64 + + MemTotalKB int64 + MemAvailableKB int64 + MemUsedKB int64 + + Disks []DiskStat + Net []NetStat + + PSIAvailable bool + PSICPUSomeAvg10 float64 + PSIMemSomeAvg10 float64 + PSIIOSomeAvg10 float64 +} + +type cpuTicks struct { + user, nice, system, idle, iowait, irq, softirq, steal int64 +} + +// readCPUTicks parses the aggregate "cpu " line of /proc/stat. +func readCPUTicks() (cpuTicks, error) { + f, err := os.Open("/proc/stat") + if err != nil { + return cpuTicks{}, fmt.Errorf("open /proc/stat: %w", err) + } + defer f.Close() + + sc := bufio.NewScanner(f) + if !sc.Scan() { + return cpuTicks{}, fmt.Errorf("read /proc/stat: %w", sc.Err()) + } + fields := strings.Fields(sc.Text()) + if len(fields) < 9 || fields[0] != "cpu" { + return cpuTicks{}, fmt.Errorf("/proc/stat: unexpected first line %q", sc.Text()) + } + vals := make([]int64, 8) + for i := range vals { + v, err := strconv.ParseInt(fields[i+1], 10, 64) + if err != nil { + return cpuTicks{}, fmt.Errorf("/proc/stat: parse field %d: %w", i, err) + } + vals[i] = v + } + return cpuTicks{ + user: vals[0], nice: vals[1], system: vals[2], idle: vals[3], + iowait: vals[4], irq: vals[5], softirq: vals[6], steal: vals[7], + }, nil +} + +type loadAvg struct { + load1, load5, load15 float64 + runnable, totalThreads int64 +} + +func readLoadAvg() (loadAvg, error) { + data, err := os.ReadFile("/proc/loadavg") + if err != nil { + return loadAvg{}, fmt.Errorf("read /proc/loadavg: %w", err) + } + fields := strings.Fields(string(data)) + if len(fields) < 4 { + return loadAvg{}, fmt.Errorf("/proc/loadavg: unexpected shape %q", string(data)) + } + var la loadAvg + if la.load1, err = strconv.ParseFloat(fields[0], 64); err != nil { + return loadAvg{}, fmt.Errorf("/proc/loadavg: load1: %w", err) + } + if la.load5, err = strconv.ParseFloat(fields[1], 64); err != nil { + return loadAvg{}, fmt.Errorf("/proc/loadavg: load5: %w", err) + } + if la.load15, err = strconv.ParseFloat(fields[2], 64); err != nil { + return loadAvg{}, fmt.Errorf("/proc/loadavg: load15: %w", err) + } + run, total, ok := strings.Cut(fields[3], "/") + if !ok { + return loadAvg{}, fmt.Errorf("/proc/loadavg: bad running/total field %q", fields[3]) + } + if la.runnable, err = strconv.ParseInt(run, 10, 64); err != nil { + return loadAvg{}, fmt.Errorf("/proc/loadavg: runnable: %w", err) + } + if la.totalThreads, err = strconv.ParseInt(total, 10, 64); err != nil { + return loadAvg{}, fmt.Errorf("/proc/loadavg: total: %w", err) + } + return la, nil +} + +type memInfo struct { + totalKB, availableKB int64 +} + +func readMemInfo() (memInfo, error) { + f, err := os.Open("/proc/meminfo") + if err != nil { + return memInfo{}, fmt.Errorf("open /proc/meminfo: %w", err) + } + defer f.Close() + + var mi memInfo + haveTotal, haveAvail := false, false + sc := bufio.NewScanner(f) + for sc.Scan() && !(haveTotal && haveAvail) { + fields := strings.Fields(sc.Text()) + if len(fields) < 2 { + continue + } + v, err := strconv.ParseInt(fields[1], 10, 64) + if err != nil { + continue + } + switch fields[0] { + case "MemTotal:": + mi.totalKB = v + haveTotal = true + case "MemAvailable:": + mi.availableKB = v + haveAvail = true + } + } + if !haveTotal || !haveAvail { + return memInfo{}, fmt.Errorf("/proc/meminfo: missing MemTotal/MemAvailable") + } + return mi, nil +} + +func readDiskStats() ([]DiskStat, error) { + f, err := os.Open("/proc/diskstats") + if err != nil { + return nil, fmt.Errorf("open /proc/diskstats: %w", err) + } + defer f.Close() + + var out []DiskStat + sc := bufio.NewScanner(f) + for sc.Scan() { + fields := strings.Fields(sc.Text()) + // major minor name reads_completed reads_merged sectors_read + // ms_reading writes_completed writes_merged sectors_written ... + if len(fields) < 10 { + continue + } + name := fields[2] + if strings.HasPrefix(name, "loop") || strings.HasPrefix(name, "ram") { + continue + } + reads, _ := strconv.ParseInt(fields[3], 10, 64) + readSectors, _ := strconv.ParseInt(fields[5], 10, 64) + writes, _ := strconv.ParseInt(fields[7], 10, 64) + writeSectors, _ := strconv.ParseInt(fields[9], 10, 64) + out = append(out, DiskStat{ + Name: name, Reads: reads, Writes: writes, + ReadSectors: readSectors, WriteSectors: writeSectors, + }) + } + if err := sc.Err(); err != nil { + return nil, fmt.Errorf("read /proc/diskstats: %w", err) + } + return out, nil +} + +func readNetDev() ([]NetStat, error) { + f, err := os.Open("/proc/net/dev") + if err != nil { + return nil, fmt.Errorf("open /proc/net/dev: %w", err) + } + defer f.Close() + + var out []NetStat + sc := bufio.NewScanner(f) + line := 0 + for sc.Scan() { + line++ + if line <= 2 { + continue // two header lines + } + iface, rest, ok := strings.Cut(sc.Text(), ":") + if !ok { + continue + } + fields := strings.Fields(rest) + if len(fields) < 9 { + continue + } + rx, _ := strconv.ParseInt(fields[0], 10, 64) + tx, _ := strconv.ParseInt(fields[8], 10, 64) + out = append(out, NetStat{Iface: strings.TrimSpace(iface), RxBytes: rx, TxBytes: tx}) + } + if err := sc.Err(); err != nil { + return nil, fmt.Errorf("read /proc/net/dev: %w", err) + } + return out, nil +} + +type psi struct { + cpuSomeAvg10, memSomeAvg10, ioSomeAvg10 float64 +} + +// parseAvg10 pulls "avg10=X.XX" out of a PSI line such as +// "some avg10=0.00 avg60=0.00 avg300=0.00 total=12345". +func parseAvg10(line string) float64 { + for _, tok := range strings.Fields(line) { + if v, ok := strings.CutPrefix(tok, "avg10="); ok { + f, err := strconv.ParseFloat(v, 64) + if err == nil { + return f + } + } + } + return 0 +} + +func readPSIFromDir(dir string) (psi, bool) { + var p psi + any := false + for _, ent := range []struct { + file string + out *float64 + }{ + {"cpu.pressure", &p.cpuSomeAvg10}, + {"memory.pressure", &p.memSomeAvg10}, + {"io.pressure", &p.ioSomeAvg10}, + } { + data, err := os.ReadFile(dir + "/" + ent.file) + if err != nil { + continue + } + line, _, _ := strings.Cut(string(data), "\n") + *ent.out = parseAvg10(line) + any = true + } + return p, any +} + +// readPSI prefers this process's own cgroup (v2 unified hierarchy) and falls +// back to the system-wide /proc/pressure/* files. Returns ok=false if PSI is +// not exposed anywhere on this kernel/host. +func readPSI() (psi, bool) { + if data, err := os.ReadFile("/proc/self/cgroup"); err == nil { + for _, line := range strings.Split(string(data), "\n") { + rel, ok := strings.CutPrefix(line, "0::") + if !ok { + continue + } + if p, ok := readPSIFromDir("/sys/fs/cgroup" + rel); ok { + return p, true + } + break + } + } + return readPSIFromDir("/proc/pressure") +} + +// namedErr carries a source label so TakeSnapshot can report which /proc +// source failed, fanned in over a channel from the goroutines below. +type namedErr struct { + source string + err error +} + +// TakeSnapshot takes two /proc/stat readings intervalMs apart and fills in +// every other field concurrently in between, via goroutines fanned in over a +// channel — the Go analogue of the C++ jthread pool and the Rust scoped +// threads. +func TakeSnapshot(intervalMs int) (Snapshot, error) { + t0, err := readCPUTicks() + if err != nil { + return Snapshot{}, err + } + + var snap Snapshot + results := make(chan namedErr, 5) + + go func() { + la, err := readLoadAvg() + if err == nil { + snap.Load1, snap.Load5, snap.Load15 = la.load1, la.load5, la.load15 + snap.Runnable, snap.TotalThreads = la.runnable, la.totalThreads + } + results <- namedErr{"loadavg", err} + }() + go func() { + mi, err := readMemInfo() + if err == nil { + snap.MemTotalKB, snap.MemAvailableKB = mi.totalKB, mi.availableKB + snap.MemUsedKB = mi.totalKB - mi.availableKB + } + results <- namedErr{"meminfo", err} + }() + go func() { + disks, err := readDiskStats() + if err == nil { + snap.Disks = disks + } + results <- namedErr{"diskstats", err} + }() + go func() { + net, err := readNetDev() + if err == nil { + snap.Net = net + } + results <- namedErr{"netdev", err} + }() + go func() { + if p, ok := readPSI(); ok { + snap.PSIAvailable = true + snap.PSICPUSomeAvg10, snap.PSIMemSomeAvg10, snap.PSIIOSomeAvg10 = + p.cpuSomeAvg10, p.memSomeAvg10, p.ioSomeAvg10 + } + results <- namedErr{"psi", nil} + }() + + time.Sleep(time.Duration(intervalMs) * time.Millisecond) + + var firstErr error + required := 0 + for i := 0; i < 5; i++ { + r := <-results + if r.source == "psi" { + continue + } + required++ + if r.err != nil && firstErr == nil { + firstErr = fmt.Errorf("%s: %w", r.source, r.err) + } + } + if firstErr != nil { + return Snapshot{}, firstErr + } + _ = required + + t1, err := readCPUTicks() + if err != nil { + return Snapshot{}, err + } + + busy0 := t0.user + t0.nice + t0.system + busy1 := t1.user + t1.nice + t1.system + idle0 := t0.idle + t0.iowait + idle1 := t1.idle + t1.iowait + total0 := busy0 + idle0 + t0.irq + t0.softirq + t0.steal + total1 := busy1 + idle1 + t1.irq + t1.softirq + t1.steal + + dTotal := float64(total1 - total0) + dBusy := float64((busy1 - busy0) + (t1.irq - t0.irq) + (t1.softirq - t0.softirq) + (t1.steal - t0.steal)) + dUser := float64(t1.user - t0.user) + dSystem := float64(t1.system - t0.system) + + if dTotal > 0 { + snap.CPUUtilPct = 100 * dBusy / dTotal + snap.CPUUserPct = 100 * dUser / dTotal + snap.CPUSystemPct = 100 * dSystem / dTotal + } + + return snap, nil +} + +func f2(v float64) string { return strconv.FormatFloat(v, 'f', 2, 64) } + +// ToJSON renders the deterministic single-line JSON object (see README.md). +func ToJSON(s Snapshot) string { + var b strings.Builder + b.WriteByte('{') + fmt.Fprintf(&b, `"cpu_util_pct":%s,`, f2(s.CPUUtilPct)) + fmt.Fprintf(&b, `"cpu_user_pct":%s,`, f2(s.CPUUserPct)) + fmt.Fprintf(&b, `"cpu_system_pct":%s,`, f2(s.CPUSystemPct)) + fmt.Fprintf(&b, `"load1":%s,`, f2(s.Load1)) + fmt.Fprintf(&b, `"load5":%s,`, f2(s.Load5)) + fmt.Fprintf(&b, `"load15":%s,`, f2(s.Load15)) + fmt.Fprintf(&b, `"runnable":%d,`, s.Runnable) + fmt.Fprintf(&b, `"total_threads":%d,`, s.TotalThreads) + fmt.Fprintf(&b, `"mem_total_kb":%d,`, s.MemTotalKB) + fmt.Fprintf(&b, `"mem_available_kb":%d,`, s.MemAvailableKB) + fmt.Fprintf(&b, `"mem_used_kb":%d,`, s.MemUsedKB) + b.WriteString(`"disks":[`) + for i, d := range s.Disks { + if i > 0 { + b.WriteByte(',') + } + fmt.Fprintf(&b, `{"name":"%s","reads":%d,"writes":%d,"read_sectors":%d,"write_sectors":%d}`, + d.Name, d.Reads, d.Writes, d.ReadSectors, d.WriteSectors) + } + b.WriteString(`],"net":[`) + for i, n := range s.Net { + if i > 0 { + b.WriteByte(',') + } + fmt.Fprintf(&b, `{"iface":"%s","rx_bytes":%d,"tx_bytes":%d}`, n.Iface, n.RxBytes, n.TxBytes) + } + b.WriteString(`],`) + fmt.Fprintf(&b, `"psi_available":%t,`, s.PSIAvailable) + fmt.Fprintf(&b, `"psi_cpu_some_avg10":%s,`, f2(s.PSICPUSomeAvg10)) + fmt.Fprintf(&b, `"psi_mem_some_avg10":%s,`, f2(s.PSIMemSomeAvg10)) + fmt.Fprintf(&b, `"psi_io_some_avg10":%s`, f2(s.PSIIOSomeAvg10)) + b.WriteByte('}') + return b.String() +} + +// ToText renders human-readable key=value lines with the same field names. +func ToText(s Snapshot) string { + var b strings.Builder + fmt.Fprintf(&b, "cpu_util_pct=%s cpu_user_pct=%s cpu_system_pct=%s\n", + f2(s.CPUUtilPct), f2(s.CPUUserPct), f2(s.CPUSystemPct)) + fmt.Fprintf(&b, "load1=%s load5=%s load15=%s runnable=%d total_threads=%d\n", + f2(s.Load1), f2(s.Load5), f2(s.Load15), s.Runnable, s.TotalThreads) + fmt.Fprintf(&b, "mem_total_kb=%d mem_available_kb=%d mem_used_kb=%d\n", + s.MemTotalKB, s.MemAvailableKB, s.MemUsedKB) + for _, d := range s.Disks { + fmt.Fprintf(&b, "disk name=%s reads=%d writes=%d read_sectors=%d write_sectors=%d\n", + d.Name, d.Reads, d.Writes, d.ReadSectors, d.WriteSectors) + } + for _, n := range s.Net { + fmt.Fprintf(&b, "net iface=%s rx_bytes=%d tx_bytes=%d\n", n.Iface, n.RxBytes, n.TxBytes) + } + fmt.Fprintf(&b, "psi_available=%t psi_cpu_some_avg10=%s psi_mem_some_avg10=%s psi_io_some_avg10=%s\n", + s.PSIAvailable, f2(s.PSICPUSomeAvg10), f2(s.PSIMemSomeAvg10), f2(s.PSIIOSomeAvg10)) + return b.String() +} diff --git a/examples/36-proc-sys-and-the-agent/rust/Cargo.lock b/examples/36-proc-sys-and-the-agent/rust/Cargo.lock new file mode 100644 index 0000000..a3de523 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/Cargo.lock @@ -0,0 +1,46 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", +] + +[[package]] +name = "sysagent" +version = "0.1.0" +dependencies = [ + "nix", +] diff --git a/examples/36-proc-sys-and-the-agent/rust/Cargo.toml b/examples/36-proc-sys-and-the-agent/rust/Cargo.toml new file mode 100644 index 0000000..931d433 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "sysagent" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +nix = { version = "0.31", features = ["signal", "poll"] } diff --git a/examples/36-proc-sys-and-the-agent/rust/demo.sh b/examples/36-proc-sys-and-the-agent/rust/demo.sh new file mode 100755 index 0000000..c2ef7e4 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + exec "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + exec "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/36-proc-sys-and-the-agent/rust/rust-toolchain.toml b/examples/36-proc-sys-and-the-agent/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs b/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs new file mode 100644 index 0000000..9aff97b --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/src/httpd.rs @@ -0,0 +1,110 @@ +//! sysagent: a hand-rolled, single-endpoint HTTP/1.1 server. No framework — +//! same philosophy as the raw-socket chatterd examples earlier in the book, +//! and the same poll(2) + signalfd(2) shutdown doorbell as their Rust code. + +use std::io::{Read, Write}; +use std::net::{TcpListener, TcpStream}; +use std::os::fd::AsFd; + +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sys::signal::{SigSet, Signal}; +use nix::sys::signalfd::{SfdFlags, SignalFd}; + +use crate::procfs::{take_snapshot, to_json}; + +fn install_shutdown() -> SignalFd { + let mut mask = SigSet::empty(); + mask.add(Signal::SIGINT); + mask.add(Signal::SIGTERM); + mask.thread_block().expect("block signals"); + SignalFd::with_flags(&mask, SfdFlags::SFD_CLOEXEC).expect("signalfd") +} + +fn http_response(status: u16, status_text: &str, content_type: &str, body: &str) -> String { + format!( + "HTTP/1.1 {status} {status_text}\r\nContent-Type: {content_type}\r\n\ + Content-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ) +} + +fn handle_client(mut stream: TcpStream, interval_ms: u64) { + let mut buf = [0u8; 4096]; + let n = match stream.read(&mut buf) { + Ok(n) if n > 0 => n, + _ => return, + }; + let req = String::from_utf8_lossy(&buf[..n]); + let line = req.lines().next().unwrap_or(""); + let mut parts = line.split_whitespace(); + let method = parts.next().unwrap_or(""); + let path = parts.next().unwrap_or(""); + + let resp = if method == "GET" && path == "/metrics" { + match take_snapshot(interval_ms) { + Ok(snap) => http_response(200, "OK", "application/json", &to_json(&snap)), + Err(e) => http_response(500, "Internal Server Error", "text/plain", &e), + } + } else { + http_response(404, "Not Found", "text/plain", "not found\n") + }; + let _ = stream.write_all(resp.as_bytes()); +} + +/// Serves GET /metrics (JSON snapshot, see procfs.rs) on 0.0.0.0:port until +/// SIGINT/SIGTERM. Each request takes a fresh snapshot with the given +/// interval_ms. Returns the process exit code (0 on a clean signal shutdown). +pub fn serve(port: u16, interval_ms: u64) -> i32 { + let listener = match TcpListener::bind(("0.0.0.0", port)) { + Ok(l) => l, + Err(e) => { + eprintln!("sysagent: bind: {e}"); + return 1; + } + }; + if let Err(e) = listener.set_nonblocking(true) { + eprintln!("sysagent: set_nonblocking: {e}"); + return 1; + } + + let sfd = install_shutdown(); + println!("sysagent: listening on 0.0.0.0:{port}"); + + loop { + let mut pfds = [ + PollFd::new(sfd.as_fd(), PollFlags::POLLIN), + PollFd::new(listener.as_fd(), PollFlags::POLLIN), + ]; + match poll(&mut pfds, PollTimeout::NONE) { + Ok(_) => {} + Err(nix::errno::Errno::EINTR) => continue, + Err(e) => { + eprintln!("sysagent: poll: {e}"); + return 1; + } + } + let sig_ready = pfds[0] + .revents() + .unwrap_or(PollFlags::empty()) + .contains(PollFlags::POLLIN); + let conn_ready = pfds[1] + .revents() + .unwrap_or(PollFlags::empty()) + .contains(PollFlags::POLLIN); + + if sig_ready { + println!("sysagent: shutting down"); + return 0; + } + if conn_ready { + match listener.accept() { + Ok((stream, _)) => { + let _ = stream.set_nonblocking(false); + handle_client(stream, interval_ms); + } + Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {} + Err(_) => {} + } + } + } +} diff --git a/examples/36-proc-sys-and-the-agent/rust/src/main.rs b/examples/36-proc-sys-and-the-agent/rust/src/main.rs new file mode 100644 index 0000000..6e75397 --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/src/main.rs @@ -0,0 +1,106 @@ +// sysagent v0 (chapter 36: /proc, /sys, and the agent) — a USE-method +// metrics collector reading /proc + /sys/fs/cgroup, no root required. +// +// sysagent sample [--json] [--interval-ms N] +// sysagent serve --port P [--interval-ms N] +// +// `sample` takes one snapshot: CPU utilization from a /proc/stat delta +// spanning --interval-ms (default 200), run-queue + load from +// /proc/loadavg, memory from /proc/meminfo, per-disk I/O from +// /proc/diskstats, network rx/tx from /proc/net/dev, and cgroup PSI +// (falling back to system-wide /proc/pressure) if the kernel exposes it. +// `serve` exposes the identical snapshot as JSON over a hand-rolled +// HTTP/1.1 /metrics endpoint, one fresh sample per request. +// +// The field names in both the --json and /metrics output are the +// deterministic, cross-language schema documented in README.md — C++ and +// Go sysagent emit byte-for-byte the same keys. +mod httpd; +mod procfs; + +use std::process::ExitCode; + +fn usage() { + eprintln!( + "usage: sysagent sample [--json] [--interval-ms N] | serve --port P [--interval-ms N]" + ); +} + +/// Minimal hand-rolled flag scan, kept in the same shape as the C++ and Go +/// argument handling so the three stay directly comparable. +struct Flags { + json: bool, + interval_ms: u64, + port: i64, +} + +fn parse_flags(args: &[String]) -> Option { + let mut f = Flags { + json: false, + interval_ms: 200, + port: -1, + }; + let mut i = 0; + while i < args.len() { + match args[i].as_str() { + "--json" => f.json = true, + "--interval-ms" => { + i += 1; + f.interval_ms = args.get(i)?.parse().ok()?; + } + "--port" => { + i += 1; + f.port = args.get(i)?.parse().ok()?; + } + _ => return None, + } + i += 1; + } + Some(f) +} + +fn cmd_sample(args: &[String]) -> u8 { + let Some(f) = parse_flags(args).filter(|f| f.interval_ms > 0) else { + usage(); + return 2; + }; + match procfs::take_snapshot(f.interval_ms) { + Ok(snap) => { + if f.json { + println!("{}", procfs::to_json(&snap)); + } else { + print!("{}", procfs::to_text(&snap)); + } + 0 + } + Err(e) => { + eprintln!("sysagent: error: {e}"); + 1 + } + } +} + +fn cmd_serve(args: &[String]) -> u8 { + let Some(f) = parse_flags(args).filter(|f| f.port > 0 && f.interval_ms > 0) else { + usage(); + return 2; + }; + httpd::serve(f.port as u16, f.interval_ms) as u8 +} + +fn main() -> ExitCode { + let args: Vec = std::env::args().collect(); + if args.len() < 2 { + usage(); + return ExitCode::from(2); + } + let code = match args[1].as_str() { + "sample" => cmd_sample(&args[2..]), + "serve" => cmd_serve(&args[2..]), + _ => { + usage(); + 2 + } + }; + ExitCode::from(code) +} diff --git a/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs b/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs new file mode 100644 index 0000000..bc605af --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/rust/src/procfs.rs @@ -0,0 +1,445 @@ +//! sysagent: /proc + cgroup PSI readers and the snapshot they compose into. + +use std::fs; +use std::thread; +use std::time::Duration; + +/// One line of /proc/diskstats reduced to the USE-method fields. +pub struct DiskStat { + pub name: String, + pub reads: i64, + pub writes: i64, + pub read_sectors: i64, + pub write_sectors: i64, +} + +/// One interface's counters from /proc/net/dev. +pub struct NetStat { + pub iface: String, + pub rx_bytes: i64, + pub tx_bytes: i64, +} + +/// One fully-formed sample. Field names here are the canonical, deterministic +/// names shared with the C++ and Go implementations (see README.md) — only +/// the encoding (Rust struct vs C++ struct vs Go struct) differs. +#[derive(Default)] +pub struct Snapshot { + pub cpu_util_pct: f64, + pub cpu_user_pct: f64, + pub cpu_system_pct: f64, + + pub load1: f64, + pub load5: f64, + pub load15: f64, + pub runnable: i64, + pub total_threads: i64, + + pub mem_total_kb: i64, + pub mem_available_kb: i64, + pub mem_used_kb: i64, + + pub disks: Vec, + pub net: Vec, + + pub psi_available: bool, + pub psi_cpu_some_avg10: f64, + pub psi_mem_some_avg10: f64, + pub psi_io_some_avg10: f64, +} + +#[derive(Default, Clone, Copy)] +struct CpuTicks { + user: i64, + nice: i64, + system: i64, + idle: i64, + iowait: i64, + irq: i64, + softirq: i64, + steal: i64, +} + +/// Parses the aggregate "cpu " line of /proc/stat. +fn read_cpu_ticks() -> Result { + let text = fs::read_to_string("/proc/stat").map_err(|e| format!("read /proc/stat: {e}"))?; + let line = text.lines().next().ok_or("/proc/stat: empty")?; + let fields: Vec<&str> = line.split_whitespace().collect(); + if fields.len() < 9 || fields[0] != "cpu" { + return Err(format!("/proc/stat: unexpected first line {line:?}")); + } + let n = |i: usize| -> Result { + fields[i] + .parse() + .map_err(|e| format!("/proc/stat: parse field {i}: {e}")) + }; + Ok(CpuTicks { + user: n(1)?, + nice: n(2)?, + system: n(3)?, + idle: n(4)?, + iowait: n(5)?, + irq: n(6)?, + softirq: n(7)?, + steal: n(8)?, + }) +} + +struct LoadAvg { + load1: f64, + load5: f64, + load15: f64, + runnable: i64, + total_threads: i64, +} + +fn read_loadavg() -> Result { + let text = + fs::read_to_string("/proc/loadavg").map_err(|e| format!("read /proc/loadavg: {e}"))?; + let fields: Vec<&str> = text.split_whitespace().collect(); + if fields.len() < 4 { + return Err(format!("/proc/loadavg: unexpected shape {text:?}")); + } + let load1 = fields[0] + .parse() + .map_err(|e| format!("/proc/loadavg: load1: {e}"))?; + let load5 = fields[1] + .parse() + .map_err(|e| format!("/proc/loadavg: load5: {e}"))?; + let load15 = fields[2] + .parse() + .map_err(|e| format!("/proc/loadavg: load15: {e}"))?; + let (run, total) = fields[3] + .split_once('/') + .ok_or_else(|| format!("/proc/loadavg: bad running/total field {:?}", fields[3]))?; + let runnable = run + .parse() + .map_err(|e| format!("/proc/loadavg: runnable: {e}"))?; + let total_threads = total + .parse() + .map_err(|e| format!("/proc/loadavg: total: {e}"))?; + Ok(LoadAvg { + load1, + load5, + load15, + runnable, + total_threads, + }) +} + +struct MemInfo { + total_kb: i64, + available_kb: i64, +} + +fn read_meminfo() -> Result { + let text = + fs::read_to_string("/proc/meminfo").map_err(|e| format!("read /proc/meminfo: {e}"))?; + let mut total_kb = None; + let mut available_kb = None; + for line in text.lines() { + let mut it = line.split_whitespace(); + let (Some(key), Some(val)) = (it.next(), it.next()) else { + continue; + }; + let Ok(v) = val.parse::() else { continue }; + match key { + "MemTotal:" => total_kb = Some(v), + "MemAvailable:" => available_kb = Some(v), + _ => {} + } + if total_kb.is_some() && available_kb.is_some() { + break; + } + } + match (total_kb, available_kb) { + (Some(total_kb), Some(available_kb)) => Ok(MemInfo { + total_kb, + available_kb, + }), + _ => Err("/proc/meminfo: missing MemTotal/MemAvailable".into()), + } +} + +fn read_diskstats() -> Result, String> { + let text = + fs::read_to_string("/proc/diskstats").map_err(|e| format!("read /proc/diskstats: {e}"))?; + let mut out = Vec::new(); + for line in text.lines() { + let fields: Vec<&str> = line.split_whitespace().collect(); + // major minor name reads_completed reads_merged sectors_read + // ms_reading writes_completed writes_merged sectors_written ... + if fields.len() < 10 { + continue; + } + let name = fields[2]; + if name.starts_with("loop") || name.starts_with("ram") { + continue; + } + let p = |i: usize| fields[i].parse::().unwrap_or(0); + out.push(DiskStat { + name: name.to_string(), + reads: p(3), + writes: p(7), + read_sectors: p(5), + write_sectors: p(9), + }); + } + Ok(out) +} + +fn read_netdev() -> Result, String> { + let text = + fs::read_to_string("/proc/net/dev").map_err(|e| format!("read /proc/net/dev: {e}"))?; + let mut out = Vec::new(); + for line in text.lines().skip(2) { + let Some((iface, rest)) = line.split_once(':') else { + continue; + }; + let fields: Vec<&str> = rest.split_whitespace().collect(); + if fields.len() < 9 { + continue; + } + let rx_bytes = fields[0].parse().unwrap_or(0); + let tx_bytes = fields[8].parse().unwrap_or(0); + out.push(NetStat { + iface: iface.trim().to_string(), + rx_bytes, + tx_bytes, + }); + } + Ok(out) +} + +#[derive(Default)] +struct Psi { + cpu_some_avg10: f64, + mem_some_avg10: f64, + io_some_avg10: f64, +} + +/// Pulls "avg10=X.XX" out of a PSI line such as +/// "some avg10=0.00 avg60=0.00 avg300=0.00 total=12345". +fn parse_avg10(line: &str) -> f64 { + for tok in line.split_whitespace() { + if let Some(v) = tok.strip_prefix("avg10=") + && let Ok(f) = v.parse() + { + return f; + } + } + 0.0 +} + +/// Reads one PSI file's first ("some") line and parses its avg10, if the +/// file exists and is readable. +fn read_one_pressure(path: &str) -> Option { + let text = fs::read_to_string(path).ok()?; + let line = text.lines().next()?; + Some(parse_avg10(line)) +} + +fn read_psi_from_dir(dir: &str) -> Option { + let cpu = read_one_pressure(&format!("{dir}/cpu.pressure")); + let mem = read_one_pressure(&format!("{dir}/memory.pressure")); + let io = read_one_pressure(&format!("{dir}/io.pressure")); + if cpu.is_none() && mem.is_none() && io.is_none() { + return None; + } + Some(Psi { + cpu_some_avg10: cpu.unwrap_or(0.0), + mem_some_avg10: mem.unwrap_or(0.0), + io_some_avg10: io.unwrap_or(0.0), + }) +} + +/// Prefers this process's own cgroup (v2 unified hierarchy: a single +/// "0::/path" line in /proc/self/cgroup) and falls back to the system-wide +/// /proc/pressure/* files. Returns None if PSI is not exposed anywhere on +/// this kernel/host. +fn read_psi() -> Option { + if let Ok(text) = fs::read_to_string("/proc/self/cgroup") { + for line in text.lines() { + if let Some(rel) = line.strip_prefix("0::") { + if let Some(psi) = read_psi_from_dir(&format!("/sys/fs/cgroup{rel}")) { + return Some(psi); + } + break; + } + } + } + read_psi_from_dir("/proc/pressure") +} + +/// Takes two /proc/stat readings `interval_ms` apart and fills in every other +/// field concurrently in between, via scoped threads — the Rust analogue of +/// the C++ jthread pool and the Go goroutines fanned in over a channel. +pub fn take_snapshot(interval_ms: u64) -> Result { + let t0 = read_cpu_ticks()?; + + let mut snap = Snapshot::default(); + let mut first_err: Option = None; + + thread::scope(|s| { + let load_h = s.spawn(read_loadavg); + let mem_h = s.spawn(read_meminfo); + let disk_h = s.spawn(read_diskstats); + let net_h = s.spawn(read_netdev); + let psi_h = s.spawn(read_psi); + + thread::sleep(Duration::from_millis(interval_ms)); + + match load_h.join().unwrap() { + Ok(la) => { + snap.load1 = la.load1; + snap.load5 = la.load5; + snap.load15 = la.load15; + snap.runnable = la.runnable; + snap.total_threads = la.total_threads; + } + Err(e) => { + first_err.get_or_insert(e); + } + } + match mem_h.join().unwrap() { + Ok(mi) => { + snap.mem_total_kb = mi.total_kb; + snap.mem_available_kb = mi.available_kb; + snap.mem_used_kb = mi.total_kb - mi.available_kb; + } + Err(e) => { + first_err.get_or_insert(e); + } + } + match disk_h.join().unwrap() { + Ok(disks) => snap.disks = disks, + Err(e) => { + first_err.get_or_insert(e); + } + } + match net_h.join().unwrap() { + Ok(net) => snap.net = net, + Err(e) => { + first_err.get_or_insert(e); + } + } + if let Some(psi) = psi_h.join().unwrap() { + snap.psi_available = true; + snap.psi_cpu_some_avg10 = psi.cpu_some_avg10; + snap.psi_mem_some_avg10 = psi.mem_some_avg10; + snap.psi_io_some_avg10 = psi.io_some_avg10; + } + }); + + if let Some(e) = first_err { + return Err(e); + } + + let t1 = read_cpu_ticks()?; + + let busy0 = t0.user + t0.nice + t0.system; + let busy1 = t1.user + t1.nice + t1.system; + let idle0 = t0.idle + t0.iowait; + let idle1 = t1.idle + t1.iowait; + let total0 = busy0 + idle0 + t0.irq + t0.softirq + t0.steal; + let total1 = busy1 + idle1 + t1.irq + t1.softirq + t1.steal; + + let d_total = (total1 - total0) as f64; + let d_busy = + ((busy1 - busy0) + (t1.irq - t0.irq) + (t1.softirq - t0.softirq) + (t1.steal - t0.steal)) + as f64; + let d_user = (t1.user - t0.user) as f64; + let d_system = (t1.system - t0.system) as f64; + + if d_total > 0.0 { + snap.cpu_util_pct = 100.0 * d_busy / d_total; + snap.cpu_user_pct = 100.0 * d_user / d_total; + snap.cpu_system_pct = 100.0 * d_system / d_total; + } + + Ok(snap) +} + +/// Renders the deterministic single-line JSON object (see README.md). +pub fn to_json(s: &Snapshot) -> String { + let mut o = String::new(); + o.push('{'); + o.push_str(&format!("\"cpu_util_pct\":{:.2},", s.cpu_util_pct)); + o.push_str(&format!("\"cpu_user_pct\":{:.2},", s.cpu_user_pct)); + o.push_str(&format!("\"cpu_system_pct\":{:.2},", s.cpu_system_pct)); + o.push_str(&format!("\"load1\":{:.2},", s.load1)); + o.push_str(&format!("\"load5\":{:.2},", s.load5)); + o.push_str(&format!("\"load15\":{:.2},", s.load15)); + o.push_str(&format!("\"runnable\":{},", s.runnable)); + o.push_str(&format!("\"total_threads\":{},", s.total_threads)); + o.push_str(&format!("\"mem_total_kb\":{},", s.mem_total_kb)); + o.push_str(&format!("\"mem_available_kb\":{},", s.mem_available_kb)); + o.push_str(&format!("\"mem_used_kb\":{},", s.mem_used_kb)); + o.push_str("\"disks\":["); + for (i, d) in s.disks.iter().enumerate() { + if i > 0 { + o.push(','); + } + o.push_str(&format!( + "{{\"name\":\"{}\",\"reads\":{},\"writes\":{},\"read_sectors\":{},\"write_sectors\":{}}}", + d.name, d.reads, d.writes, d.read_sectors, d.write_sectors + )); + } + o.push_str("],\"net\":["); + for (i, n) in s.net.iter().enumerate() { + if i > 0 { + o.push(','); + } + o.push_str(&format!( + "{{\"iface\":\"{}\",\"rx_bytes\":{},\"tx_bytes\":{}}}", + n.iface, n.rx_bytes, n.tx_bytes + )); + } + o.push_str("],"); + o.push_str(&format!("\"psi_available\":{},", s.psi_available)); + o.push_str(&format!( + "\"psi_cpu_some_avg10\":{:.2},", + s.psi_cpu_some_avg10 + )); + o.push_str(&format!( + "\"psi_mem_some_avg10\":{:.2},", + s.psi_mem_some_avg10 + )); + o.push_str(&format!("\"psi_io_some_avg10\":{:.2}", s.psi_io_some_avg10)); + o.push('}'); + o +} + +/// Renders human-readable key=value lines with the same field names. +pub fn to_text(s: &Snapshot) -> String { + let mut o = String::new(); + o.push_str(&format!( + "cpu_util_pct={:.2} cpu_user_pct={:.2} cpu_system_pct={:.2}\n", + s.cpu_util_pct, s.cpu_user_pct, s.cpu_system_pct + )); + o.push_str(&format!( + "load1={:.2} load5={:.2} load15={:.2} runnable={} total_threads={}\n", + s.load1, s.load5, s.load15, s.runnable, s.total_threads + )); + o.push_str(&format!( + "mem_total_kb={} mem_available_kb={} mem_used_kb={}\n", + s.mem_total_kb, s.mem_available_kb, s.mem_used_kb + )); + for d in &s.disks { + o.push_str(&format!( + "disk name={} reads={} writes={} read_sectors={} write_sectors={}\n", + d.name, d.reads, d.writes, d.read_sectors, d.write_sectors + )); + } + for n in &s.net { + o.push_str(&format!( + "net iface={} rx_bytes={} tx_bytes={}\n", + n.iface, n.rx_bytes, n.tx_bytes + )); + } + o.push_str(&format!( + "psi_available={} psi_cpu_some_avg10={:.2} psi_mem_some_avg10={:.2} psi_io_some_avg10={:.2}\n", + s.psi_available, s.psi_cpu_some_avg10, s.psi_mem_some_avg10, s.psi_io_some_avg10 + )); + o +} diff --git a/examples/36-proc-sys-and-the-agent/verify.lua b/examples/36-proc-sys-and-the-agent/verify.lua new file mode 100644 index 0000000..c1dcf0e --- /dev/null +++ b/examples/36-proc-sys-and-the-agent/verify.lua @@ -0,0 +1,193 @@ +-- Verify sysagent v0 (36-proc-sys-and-the-agent) for LSP_LANG. +-- +-- Asserts observable behavior, identical across cpp/go/rust: +-- 1. usage errors: no args / unknown subcommand / serve without --port all +-- exit 2 with a usage line on stderr. +-- 2. `sample --json` emits every expected key with a numeric value, in the +-- deterministic cross-language schema (README.md), with sane ranges. +-- 3. A cpu-busy child measurably raises cpu_util_pct vs an idle baseline — +-- compared with a robust margin and a couple of retries, since this +-- runs on a real, possibly-noisy host rather than an isolated lab VM. +-- 4. `serve --port P` answers GET /metrics with the same JSON shape, 404s +-- an unknown path, and its mem_total_kb agrees exactly with a direct +-- /proc/meminfo read (a static fact untouched by system load); SIGINT +-- shuts it down cleanly (exit 0). +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set; the interpreter may be lua or luajit. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- --------------------------------------------------------------------------- +-- 1. error shapes +-- --------------------------------------------------------------------------- + +local noargs = checks.run(demo) +checks.expect_exit(noargs, 2, lang .. ": no arguments exits 2") +checks.expect_match(noargs.out, "usage: sysagent", lang .. ": no arguments prints usage") + +local badsub = checks.run(demo .. " frobnicate") +checks.expect_exit(badsub, 2, lang .. ": unknown subcommand exits 2") + +local noport = checks.run(demo .. " serve") +checks.expect_exit(noport, 2, lang .. ": serve without --port exits 2") + +-- --------------------------------------------------------------------------- +-- 2. `sample --json` schema + sane ranges +-- --------------------------------------------------------------------------- + +local sample = checks.run(demo .. " sample --json --interval-ms 200") +checks.expect_exit(sample, 0, lang .. ": sample --json exits 0") + +local function expect_number_key(key, label) + local v = sample.out:match('"' .. key .. '":(%-?%d+%.?%d*)') + if v == nil then + checks.expect_match(sample.out, '"' .. key .. '":NEVER_MATCHES', label .. " (key present)") + else + checks.expect_match(tostring(tonumber(v) ~= nil), "true", label .. " (numeric)") + end +end + +for _, key in ipairs({ + "cpu_util_pct", "cpu_user_pct", "cpu_system_pct", + "load1", "load5", "load15", "runnable", "total_threads", + "mem_total_kb", "mem_available_kb", "mem_used_kb", + "psi_cpu_some_avg10", "psi_mem_some_avg10", "psi_io_some_avg10", +}) do + expect_number_key(key, lang .. ": sample --json has \"" .. key .. "\"") +end + +checks.expect_match(sample.out, '"psi_available":%a+', lang .. ": sample --json has psi_available bool") +checks.expect_match(sample.out, '"disks":%[.*"name":"[%w_%-/]+".-"reads":%d+.-"writes":%d+', + lang .. ": disks array has a named entry with reads/writes") +checks.expect_match(sample.out, '"net":%[.-"iface":"lo".-"rx_bytes":%d+.-"tx_bytes":%d+', + lang .. ": net array includes loopback rx/tx") + +local mem_total = tonumber(sample.out:match('"mem_total_kb":(%d+)')) +local util_pct = tonumber(sample.out:match('"cpu_util_pct":([%d%.]+)')) +checks.expect_match(tostring(mem_total ~= nil and mem_total > 0), "true", + lang .. ": mem_total_kb is a positive number") +checks.expect_match(tostring(util_pct ~= nil and util_pct >= 0 and util_pct <= 100.01), "true", + lang .. ": cpu_util_pct is within [0,100]") + +-- --------------------------------------------------------------------------- +-- 3. a cpu-busy child raises cpu_util_pct vs an idle baseline +-- --------------------------------------------------------------------------- + +local ncpu = tonumber(checks.run("nproc").out:match("%d+")) or 4 + +local function busy_vs_idle() + local script = string.format([[ +IDLE=$(%s sample --json --interval-ms 300 | grep -o '"cpu_util_pct":[0-9.]*' | cut -d: -f2) +echo "IDLE_PCT=$IDLE" +for i in $(seq %d); do + timeout 2 bash -c 'while :; do :; done' & +done +sleep 0.3 +BUSY=$(%s sample --json --interval-ms 500 | grep -o '"cpu_util_pct":[0-9.]*' | cut -d: -f2) +echo "BUSY_PCT=$BUSY" +wait 2>/dev/null +]], demo, ncpu, demo) + local r = checks.run(script) + return tonumber(r.out:match("IDLE_PCT=([%d%.]+)")), tonumber(r.out:match("BUSY_PCT=([%d%.]+)")) +end + +-- Retried because this runs on a real, possibly shared host: the margin is +-- generous (12 points) and there's a high-absolute-utilization escape hatch +-- (55%) for the case where the host is already so busy the idle baseline +-- itself is elevated. +local margin_ok, last_idle, last_busy = false, nil, nil +for _attempt = 1, 3 do + local idle, busy = busy_vs_idle() + last_idle, last_busy = idle, busy + if idle and busy and (busy - idle >= 12 or busy >= 55) then + margin_ok = true + break + end +end +print(string.format("cpu-busy check: idle=%s busy=%s margin_ok=%s", + tostring(last_idle), tostring(last_busy), tostring(margin_ok))) +checks.expect_match(margin_ok and "yes" or "no", "yes", + lang .. ": a cpu-busy child raises cpu_util_pct vs idle (margin>=12pp or busy>=55%)") + +-- --------------------------------------------------------------------------- +-- 4. `serve` — /metrics answers, 404s elsewhere, agrees with a direct +-- /proc read, and shuts down cleanly on SIGINT. +-- --------------------------------------------------------------------------- + +local port = checks.run( + "python3 -c 'import socket; s=socket.socket(); s.bind((\"127.0.0.1\",0)); " .. + "print(s.getsockname()[1]); s.close()'").out:gsub("%s+$", "") +if not port:match("^%d+$") then + checks.skip("could not allocate a TCP port") +end + +local wd = checks.run("mktemp -d").out:gsub("%s+$", "") +if wd == "" or not wd:match("^/") then + checks.skip("mktemp -d did not yield a directory") +end + +local function sh(script) + return checks.run((script:gsub("@WD@", wd):gsub("@PORT@", port):gsub("@DEMO@", demo))) +end + +local session = sh([[ +set -u +@DEMO@ serve --port @PORT@ > @WD@/srv.out 2>&1 & +SRV=$! +for _ in $(seq 1 200); do ss -ltn 2>/dev/null | grep -q ":@PORT@ " && break; sleep 0.05; done + +echo "===BODY===" +curl -s http://127.0.0.1:@PORT@/metrics +echo +echo "===ENDBODY===" + +echo "CODE404=$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:@PORT@/nope)" +echo "DIRECT_MEMTOTAL=$(awk '/^MemTotal:/{print $2}' /proc/meminfo)" + +kill -INT "$SRV" +for _ in $(seq 1 200); do kill -0 "$SRV" 2>/dev/null || break; sleep 0.05; done +if kill -0 "$SRV" 2>/dev/null; then kill -9 "$SRV"; echo "server_exit=timeout"; else + wait "$SRV"; echo "server_exit=$?" +fi +echo "===SERVER==="; cat @WD@/srv.out +]]) + +checks.expect_match(session.out, "CODE404=404", lang .. ": serve 404s an unknown path") + +local body = session.out:match("===BODY===\n(.-)\n===ENDBODY===") +checks.expect_match(tostring(body ~= nil and body:match('"cpu_util_pct"') ~= nil), "true", + lang .. ": /metrics answers with the JSON snapshot") + +local served_mem_total = body and tonumber(body:match('"mem_total_kb":(%d+)')) +local direct_mem_total = tonumber(session.out:match("DIRECT_MEMTOTAL=(%d+)")) +checks.expect_match( + tostring(served_mem_total ~= nil and direct_mem_total ~= nil and served_mem_total == direct_mem_total), + "true", lang .. ": /metrics mem_total_kb agrees exactly with a direct /proc/meminfo read") + +local disk_name = body and body:match('"name":"([%w_%-/]+)"') +if disk_name then + local found = checks.run("grep -q '" .. disk_name .. "' /proc/diskstats && echo yes || echo no") + checks.expect_match(found.out, "yes", + lang .. ": a disk name from /metrics (" .. disk_name .. ") is present in /proc/diskstats") +end + +checks.expect_match(session.out, "sysagent: listening on 0%.0%.0%.0:" .. port, + lang .. ": server announces the listen address") +checks.expect_match(session.out, "sysagent: shutting down", + lang .. ": server logs the shutdown") +checks.expect_match(session.out, "server_exit=0", + lang .. ": SIGINT shuts the server down cleanly and it exits 0") + +checks.run("rm -rf " .. wd) +checks.finish() diff --git a/examples/37-use-method-60s-analysis/README.md b/examples/37-use-method-60s-analysis/README.md new file mode 100644 index 0000000..39b523c --- /dev/null +++ b/examples/37-use-method-60s-analysis/README.md @@ -0,0 +1,51 @@ +# _template — hello-syscall + +The seed for every runnable example in the book. It implements the same tiny +program three times — C++23, Go, and Rust — each printing exactly one line: + +``` +pid on () +``` + +via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error +handling (`std::expected` / wrapped errors via `%w` / `rustix`). + +## Layout + +``` +_template/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build, demo.sh +├── go/ # go build, demo.sh +└── rust/ # cargo build, demo.sh +``` + +## The demo contract + +Every language directory has a `demo.sh` with the same interface, identical +across the whole book: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of + running locally. + +The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs +`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. + +## Verification + +`verify.lua` is run from this directory with `LSP_LANG` set to one language at +a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and +asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, +77 = skip. + +## Mapping to a chapter + +Each chapter's example is a copy of this template (`scripts/new-example.sh +NN-slug`) with the program body replaced by that chapter's code. Keep the +demo contract and `verify.lua` shape intact so CI and the manifest +(`examples/manifest.yaml`) keep working. diff --git a/examples/37-use-method-60s-analysis/cpp/.clang-format b/examples/37-use-method-60s-analysis/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/37-use-method-60s-analysis/cpp/.clang-tidy b/examples/37-use-method-60s-analysis/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/37-use-method-60s-analysis/cpp/CMakeLists.txt b/examples/37-use-method-60s-analysis/cpp/CMakeLists.txt new file mode 100644 index 0000000..72768a8 --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/CMakeLists.txt @@ -0,0 +1,11 @@ +cmake_minimum_required(VERSION 3.25) +project(loadmix CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app src/main.cpp) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/37-use-method-60s-analysis/cpp/CMakePresets.json b/examples/37-use-method-60s-analysis/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/37-use-method-60s-analysis/cpp/demo.sh b/examples/37-use-method-60s-analysis/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/37-use-method-60s-analysis/cpp/src/main.cpp b/examples/37-use-method-60s-analysis/cpp/src/main.cpp new file mode 100644 index 0000000..be5abef --- /dev/null +++ b/examples/37-use-method-60s-analysis/cpp/src/main.cpp @@ -0,0 +1,659 @@ +// loadmix (chapter 37): a saturation generator + guided-analysis harness for +// the classic Brendan Gregg 60-second USE-method checklist. +// +// app --resource cpu|mem|io|net --seconds N saturate that resource +// app analyze [--seconds N] run the checklist and name +// the saturated resource +// +// `analyze` shells out to the real system tools (uptime, dmesg, vmstat, +// mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the +// checklist would, parses their plain-text tables generically (by column +// *name*, read from each tool's own header row, never a hardcoded position), +// and for each of the four resources reports which USE signals — Utilization +// and Saturation — fired against a fixed threshold. Errors is reported too, +// fixed false: this chapter induces load, not kernel-logged faults, so an +// real checklist run finds none. +// +// C++23 idioms used throughout: std::jthread (RAII-joined worker pools), +// std::atomic counters shared across them, std::expected for the one truly +// fallible operation (spawning a subprocess and reading its output), and +// std::println for all output. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +// ------------------------------------------------------------- utilities -- + +unsigned nproc() { + unsigned n = std::thread::hardware_concurrency(); + return n == 0 ? 1 : n; +} + +std::string trim(const std::string& s) { + size_t a = s.find_first_not_of(" \t\r\n"); + if (a == std::string::npos) return ""; + size_t b = s.find_last_not_of(" \t\r\n"); + return s.substr(a, b - a + 1); +} + +std::vector split_ws(const std::string& line) { + std::vector out; + std::istringstream iss(line); + std::string tok; + while (iss >> tok) out.push_back(tok); + return out; +} + +std::vector split_lines(const std::string& text) { + std::vector out; + std::istringstream iss(text); + std::string line; + while (std::getline(iss, line)) out.push_back(line); + return out; +} + +// A parsed table row: column names (from a tool's own header line) zipped +// with this row's values, so every parser below reads fields *by name* +// rather than by a hardcoded position — robust to sysstat/procps column +// reordering across versions. +struct Row { + std::vector names; + std::vector values; + + [[nodiscard]] std::optional get(const std::string& name) const { + for (size_t i = 0; i < names.size() && i < values.size(); ++i) { + if (names[i] == name) { + try { + return std::stod(values[i]); + } catch (...) { + return std::nullopt; + } + } + } + return std::nullopt; + } +}; + +Row make_row(const std::vector& header_tokens, size_t header_skip, + const std::vector& data_tokens, size_t data_skip) { + Row r; + for (size_t i = header_skip; i < header_tokens.size(); ++i) r.names.push_back(header_tokens[i]); + for (size_t i = data_skip; i < data_tokens.size(); ++i) r.values.push_back(data_tokens[i]); + return r; +} + +// Run a short command to completion and capture combined stdout+stderr. +// Forced to the C locale so decimal points/dates are unambiguous regardless +// of the guest's configured locale. The one fallible operation in this file +// that returns std::expected, per book convention. +std::expected run_capture(const std::string& cmd) { + // Grouped in a subshell so a single trailing "2>&1" captures every stage + // of a pipeline (e.g. "dmesg | tail -n 5") — without the parens, 2>&1 + // binds only to the pipeline's last stage and an earlier stage's stderr + // (e.g. dmesg's own permission-denied message) leaks past the capture. + std::string full = "( env LC_ALL=C LANG=C " + cmd + " ) 2>&1"; + FILE* f = ::popen(full.c_str(), "r"); + if (!f) return std::unexpected(std::error_code{errno, std::system_category()}); + std::string out; + char buf[4096]; + size_t n; + while ((n = std::fread(buf, 1, sizeof buf, f)) > 0) out.append(buf, n); + ::pclose(f); + return out; +} + +// Start `env LC_ALL=C LANG=C ` in the background, stdout+stderr +// redirected to outfile; returns the child pid (or -1 on fork failure). Used +// for the five interval samplers so they run *concurrently* — a 12s analyze +// window costs ~12s wall-clock, not 5x that from running them one at a time. +pid_t spawn_to_file(const std::vector& argv, const std::string& outfile) { + pid_t pid = ::fork(); + if (pid == 0) { + int fd = ::open(outfile.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0600); + if (fd >= 0) { + ::dup2(fd, STDOUT_FILENO); + ::dup2(fd, STDERR_FILENO); + ::close(fd); + } + std::vector full = {"env", "LC_ALL=C", "LANG=C"}; + full.insert(full.end(), argv.begin(), argv.end()); + std::vector cargv; + cargv.reserve(full.size() + 1); + for (auto& s : full) cargv.push_back(s.data()); + cargv.push_back(nullptr); + ::execvp("env", cargv.data()); + ::_exit(127); + } + return pid; +} + +void wait_for(pid_t pid) { + if (pid <= 0) return; + int status = 0; + ::waitpid(pid, &status, 0); +} + +std::string read_all(const std::string& path) { + std::ifstream f(path); + std::ostringstream ss; + ss << f.rdbuf(); + return ss.str(); +} + +// ------------------------------------------------------- checklist parsers -- +// Each parser reads a header line's column names from the tool's own output +// and zips subsequent data lines against that name list — see Row::get above. + +struct VmstatResult { + double run_queue_max = 0.0; + double swap_io_max = 0.0; // si+so, KB/s +}; + +// vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its +// *first* sample is a since-boot average, not a live interval — both quirks +// are handled here: we scan every data row after the header, skip row 0, and +// take the max across the rest (the steady-state saturation signal, not +// diluted by an average across the whole window). +VmstatResult parse_vmstat(const std::string& text) { + VmstatResult res; + std::vector header; + std::vector> data_rows; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.empty()) continue; + if (header.empty()) { + bool has_swpd = std::any_of(toks.begin(), toks.end(), [](auto& t) { return t == "swpd"; }); + if (has_swpd) header = toks; + continue; + } + if (toks.size() + 2 >= header.size()) data_rows.push_back(toks); + } + for (size_t i = 1; i < data_rows.size(); ++i) { // skip row 0 (since-boot average) + Row r = make_row(header, 0, data_rows[i], 0); + res.run_queue_max = std::max(res.run_queue_max, r.get("r").value_or(0.0)); + double si = r.get("si").value_or(0.0), so = r.get("so").value_or(0.0); + res.swap_io_max = std::max(res.swap_io_max, si + so); + } + return res; +} + +struct MpstatResult { + double busy_pct = 0.0; + double iowait_pct = 0.0; +}; + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically — sysstat's job, not ours). +MpstatResult parse_mpstat(const std::string& text) { + MpstatResult res; + std::vector header, data; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.size() < 2) continue; + if (toks[0] == "Average:" && toks[1] == "CPU") header = toks; + else if (toks[0] == "Average:" && toks[1] == "all") data = toks; + } + if (header.empty() || data.empty()) return res; + Row r = make_row(header, 2, data, 2); + double idle = r.get("%idle").value_or(100.0); + res.iowait_pct = r.get("%iowait").value_or(0.0); + res.busy_pct = 100.0 - idle; + return res; +} + +double parse_free_used_pct(const std::string& text) { + std::vector header, data; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.empty()) continue; + bool has_available = std::any_of(toks.begin(), toks.end(), [](auto& t) { return t == "available"; }); + if (has_available) header = toks; + else if (toks[0] == "Mem:") data = toks; + } + if (header.empty() || data.empty()) return 0.0; + Row r = make_row(header, 0, data, 1); + double total = r.get("total").value_or(0.0); + double avail = r.get("available").value_or(0.0); + if (total <= 0.0) return 0.0; + return (total - avail) / total * 100.0; +} + +struct IostatResult { + double util_max = 0.0; + double await_max = 0.0; // w_await, ms +}; + +// iostat -xz has no Average line either, AND -z hides idle devices entirely +// (they just don't appear in a quiet interval) — so we scan every per-second +// block, skip block 0 (since-boot), skip loopback/removable/compressed +// pseudo-devices, and take the max over the guest's real block device. +IostatResult parse_iostat(const std::string& text) { + IostatResult res; + static const std::vector excl = {"zram", "sr", "loop", "dm-"}; + int block_idx = -1; + std::vector header; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.empty()) continue; + if (toks[0] == "avg-cpu:") { + ++block_idx; + continue; + } + if (toks[0] == "Device") { + header = toks; + continue; + } + if (header.empty() || block_idx < 1) continue; // no header yet, or since-boot block + const std::string& dev = toks[0]; + bool skip = std::any_of(excl.begin(), excl.end(), [&](auto& p) { return dev.rfind(p, 0) == 0; }); + if (skip) continue; + Row r = make_row(header, 1, toks, 1); + res.util_max = std::max(res.util_max, r.get("%util").value_or(0.0)); + res.await_max = std::max(res.await_max, r.get("w_await").value_or(0.0)); + } + return res; +} + +double parse_sar_pkts(const std::string& text, const std::string& iface) { + std::vector header, data; + for (auto& line : split_lines(text)) { + auto toks = split_ws(line); + if (toks.size() < 2) continue; + if (toks[0] == "Average:" && toks[1] == "IFACE") header = toks; + else if (toks[0] == "Average:" && toks[1] == iface) data = toks; + } + if (header.empty() || data.empty()) return 0.0; + Row r = make_row(header, 2, data, 2); + return r.get("rxpck/s").value_or(0.0) + r.get("txpck/s").value_or(0.0); +} + +// ---------------------------------------------------------------- saturate -- + +std::pair sat_cpu(int seconds, unsigned workers) { + auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + std::atomic total_iters{0}, checksum{0}; + { + std::vector pool; + pool.reserve(workers); + for (unsigned w = 0; w < workers; ++w) { + pool.emplace_back([&total_iters, &checksum, deadline, w](std::stop_token st) { + uint64_t x = 0x9E3779B97F4A7C15ULL ^ (w + 1); + uint64_t iters = 0; + while (!st.stop_requested()) { + x ^= x << 7; + x ^= x >> 9; + ++iters; + if ((iters & 0xFFFFF) == 0 && std::chrono::steady_clock::now() >= deadline) break; + } + total_iters.fetch_add(iters, std::memory_order_relaxed); + checksum.fetch_add(x, std::memory_order_relaxed); // keeps x provably live (no DCE) + }); + } + // Hold the scope open for the full duration. ~jthread runs + // request_stop() *before* join(), so without this wait the workers are + // told to stop the instant the spawn loop ends and the deadline check + // below never fires. The sleep makes that stop request the intended + // post-deadline signal rather than an immediate kill. + std::this_thread::sleep_until(deadline); + } // jthreads join here (RAII) + return {total_iters.load(), checksum.load()}; +} + +std::pair sat_mem(int seconds, unsigned workers) { + uint64_t mem_kb = 0; + { + std::ifstream mi("/proc/meminfo"); + std::string line; + while (std::getline(mi, line)) { + if (line.rfind("MemTotal:", 0) == 0) { + std::istringstream iss(line.substr(9)); + iss >> mem_kb; + break; + } + } + } + uint64_t target_bytes = mem_kb > 0 ? static_cast(mem_kb * 1024.0 * 1.35) + : (2ull << 30); // 2 GiB fallback if /proc/meminfo is unreadable + uint64_t chunk = std::max(target_bytes / workers, 4096 * 16); + + auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + std::atomic total_touches{0}; + { + std::vector pool; + pool.reserve(workers); + for (unsigned w = 0; w < workers; ++w) { + pool.emplace_back([&total_touches, deadline, chunk](std::stop_token st) { + std::vector buf(chunk, 0); + uint64_t touches = 0; + while (!st.stop_requested() && std::chrono::steady_clock::now() < deadline) { + for (size_t off = 0; off < buf.size(); off += 4096) { + buf[off] = static_cast(buf[off] + 1); + ++touches; + } + } + total_touches.fetch_add(touches, std::memory_order_relaxed); + }); + } + std::this_thread::sleep_until(deadline); // see sat_cpu: ~jthread stops before it joins + } + return {total_touches.load(), target_bytes}; +} + +std::pair sat_io(int seconds, unsigned workers) { + std::string dir = "/var/tmp/loadmix-io-" + std::to_string(::getpid()); + ::mkdir(dir.c_str(), 0700); + + auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + std::atomic total_writes{0}, total_bytes{0}; + { + std::vector pool; + pool.reserve(workers); + for (unsigned w = 0; w < workers; ++w) { + std::string path = dir + "/w" + std::to_string(w) + ".dat"; + pool.emplace_back([&total_writes, &total_bytes, deadline, path](std::stop_token st) { + int fd = ::open(path.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_DSYNC, 0600); + if (fd < 0) return; + constexpr size_t kBuf = 16384; + std::vector buf(kBuf); + // xorshift64*: cheap per-thread PRNG so the payload isn't + // trivially compressible (this guest's /var is btrfs with + // zstd:1 compression, which would otherwise let a real disk + // write shrink to nearly nothing and mute the iowait signal). + uint64_t seed = 0x2545F4914F6CDD1DULL ^ reinterpret_cast(buf.data()); + uint64_t writes = 0, bytes = 0; + while (!st.stop_requested() && std::chrono::steady_clock::now() < deadline) { + for (size_t i = 0; i < kBuf; i += 8) { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + std::memcpy(buf.data() + i, &seed, 8); + } + ssize_t n = ::pwrite(fd, buf.data(), buf.size(), 0); // same 16 KiB region: bounded file size + if (n > 0) { + ++writes; + bytes += static_cast(n); + } + } + ::close(fd); + ::unlink(path.c_str()); + total_writes.fetch_add(writes, std::memory_order_relaxed); + total_bytes.fetch_add(bytes, std::memory_order_relaxed); + }); + } + std::this_thread::sleep_until(deadline); // see sat_cpu: ~jthread stops before it joins + } + ::rmdir(dir.c_str()); + return {total_writes.load(), total_bytes.load()}; +} + +std::pair sat_net(int seconds, unsigned workers) { + int rfd = ::socket(AF_INET, SOCK_DGRAM, 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + ::bind(rfd, reinterpret_cast(&addr), sizeof addr); + socklen_t alen = sizeof addr; + ::getsockname(rfd, reinterpret_cast(&addr), &alen); + uint16_t port = ntohs(addr.sin_port); + timeval tv{0, 200000}; // 200ms so the receiver notices `stop` promptly + ::setsockopt(rfd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof tv); + + std::atomic stop{false}; + std::atomic total_pkts{0}, total_bytes{0}; + { + std::jthread receiver([&stop, rfd](std::stop_token) { + char buf[256]; + while (!stop.load(std::memory_order_relaxed)) ::recv(rfd, buf, sizeof buf, 0); + }); + auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + { + std::vector pool; + pool.reserve(workers); + for (unsigned w = 0; w < workers; ++w) { + pool.emplace_back([&total_pkts, &total_bytes, deadline, port](std::stop_token st) { + int sfd = ::socket(AF_INET, SOCK_DGRAM, 0); + sockaddr_in dst{}; + dst.sin_family = AF_INET; + dst.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + dst.sin_port = htons(port); + ::connect(sfd, reinterpret_cast(&dst), sizeof dst); + char payload[64] = {}; + uint64_t pkts = 0, bytes = 0; + while (!st.stop_requested() && std::chrono::steady_clock::now() < deadline) { + ssize_t n = ::send(sfd, payload, sizeof payload, 0); + if (n > 0) { + ++pkts; + bytes += static_cast(n); + } + } + ::close(sfd); + total_pkts.fetch_add(pkts, std::memory_order_relaxed); + total_bytes.fetch_add(bytes, std::memory_order_relaxed); + }); + } + std::this_thread::sleep_until(deadline); // see sat_cpu: ~jthread stops before it joins + } // sender pool joins here + stop.store(true, std::memory_order_relaxed); + } // receiver joins here (notices `stop` within ~200ms) + ::close(rfd); + return {total_pkts.load(), total_bytes.load()}; +} + +int cmd_saturate(const std::string& resource, int seconds) { + unsigned cpus = nproc(); + unsigned workers = (resource == "mem") ? std::max(2u, cpus) : std::max(4u, cpus * (resource == "cpu" ? 3u : 4u)); + + std::println("loadmix: start resource={} seconds={} workers={}", resource, seconds, workers); + std::pair r; + if (resource == "cpu") r = sat_cpu(seconds, workers); + else if (resource == "mem") r = sat_mem(seconds, workers); + else if (resource == "io") r = sat_io(seconds, workers); + else r = sat_net(seconds, workers); + std::println("loadmix: done resource={} seconds={} workers={} ops={} bytes={}", resource, seconds, workers, + r.first, r.second); + return 0; +} + +// ------------------------------------------------------------------ analyze -- + +int cmd_analyze(int seconds) { + unsigned cpus = nproc(); + std::println("analyze: start seconds={} cpus={}", seconds, cpus); + + auto up = run_capture("uptime"); + std::println("analyze: tool uptime raw=\"{}\"", up ? trim(*up) : std::string("unavailable")); + + auto dm = run_capture("dmesg | tail -n 5"); + std::string dm_status = "ok"; + if (!dm || dm->find("Operation not permitted") != std::string::npos) dm_status = "denied"; + std::println("analyze: tool dmesg status={}", dm_status); + + auto top_out = run_capture("top -b -n1"); + std::string cpu_line, tasks_line; + if (top_out) { + for (auto& l : split_lines(*top_out)) { + if (l.rfind("%Cpu", 0) == 0) cpu_line = trim(l); + if (l.rfind("Tasks:", 0) == 0) tasks_line = trim(l); + } + } + std::println("analyze: tool top cpu=\"{}\" tasks=\"{}\"", cpu_line, tasks_line); + + auto ss_out = run_capture("ss -s"); + std::string ss_line; + if (ss_out) { + auto lines = split_lines(*ss_out); + if (!lines.empty()) ss_line = trim(lines[0]); + } + std::println("analyze: tool ss raw=\"{}\"", ss_line); + + // The five interval samplers run CONCURRENTLY (each redirected to its own + // temp file) so an N-second checklist costs ~N seconds, not 5N. + std::string dir = "/tmp/loadmix-analyze-" + std::to_string(::getpid()); + ::mkdir(dir.c_str(), 0700); + std::string vfile = dir + "/vmstat.out", mfile = dir + "/mpstat.out", ifile = dir + "/iostat.out", + sfile = dir + "/sar.out", pfile = dir + "/pidstat.out"; + std::string n = std::to_string(seconds); + pid_t pv = spawn_to_file({"vmstat", "1", n}, vfile); + pid_t pm = spawn_to_file({"mpstat", "-P", "ALL", "1", n}, mfile); + pid_t pi = spawn_to_file({"iostat", "-xz", "1", n}, ifile); + pid_t ps = spawn_to_file({"sar", "-n", "DEV", "1", n}, sfile); + pid_t pp = spawn_to_file({"pidstat", "1", n}, pfile); + wait_for(pv); + wait_for(pm); + wait_for(pi); + wait_for(ps); + wait_for(pp); + std::string vtext = read_all(vfile), mtext = read_all(mfile), itext = read_all(ifile), stext = read_all(sfile), + ptext = read_all(pfile); + ::unlink(vfile.c_str()); + ::unlink(mfile.c_str()); + ::unlink(ifile.c_str()); + ::unlink(sfile.c_str()); + ::unlink(pfile.c_str()); + ::rmdir(dir.c_str()); + + int active = 0; + for (auto& l : split_lines(ptext)) { + auto t = split_ws(l); + if (t.size() >= 2 && t[0] == "Average:" && t[1] != "UID") ++active; + } + std::println("analyze: tool pidstat active_processes={}", active); + + auto free_out = run_capture("free -m"); + double mem_used_pct = free_out ? parse_free_used_pct(*free_out) : 0.0; + + VmstatResult vm = parse_vmstat(vtext); + MpstatResult mp = parse_mpstat(mtext); + IostatResult io = parse_iostat(itext); + double net_pkts = parse_sar_pkts(stext, "lo"); + + std::println("analyze: metric resource=cpu name=busy_pct value={:.2f} unit=pct", mp.busy_pct); + std::println("analyze: metric resource=cpu name=run_queue value={:.2f} unit=procs", vm.run_queue_max); + std::println("analyze: metric resource=mem name=used_pct value={:.2f} unit=pct", mem_used_pct); + std::println("analyze: metric resource=mem name=swap_io value={:.2f} unit=kbps", vm.swap_io_max); + std::println("analyze: metric resource=io name=util_pct value={:.2f} unit=pct", io.util_max); + std::println("analyze: metric resource=io name=iowait_pct value={:.2f} unit=pct", mp.iowait_pct); + std::println("analyze: metric resource=io name=await_ms value={:.2f} unit=ms", io.await_max); + std::println("analyze: metric resource=net name=pkts value={:.2f} unit=per_s", net_pkts); + + // signal thresholds: Utilization/Saturation pair per resource. cpu's + // saturation threshold scales with this guest's own core count — a run + // queue longer than (cpus + headroom) means threads are waiting for a + // CPU, the textbook USE definition of saturation. + struct Candidate { + std::string resource; + double u_val, u_thr, s_val, s_thr; + }; + std::vector table = { + {"cpu", mp.busy_pct, 60.0, vm.run_queue_max, cpus + 2.0}, + {"mem", mem_used_pct, 75.0, vm.swap_io_max, 300.0}, + {"io", io.util_max, 40.0, mp.iowait_pct, 8.0}, + {"net", net_pkts, 2000.0, net_pkts, 8000.0}, + }; + + std::string verdict; + double best_ratio = -1.0; + for (auto& c : table) { + bool ufired = c.u_val >= c.u_thr; + bool sfired = c.s_val >= c.s_thr; + std::println("analyze: signal resource={} type=Utilization fired={} value={:.2f} threshold={:.2f}", + c.resource, ufired ? "true" : "false", c.u_val, c.u_thr); + std::println("analyze: signal resource={} type=Saturation fired={} value={:.2f} threshold={:.2f}", + c.resource, sfired ? "true" : "false", c.s_val, c.s_thr); + std::println("analyze: signal resource={} type=Errors fired=false", c.resource); + double ratio = c.s_val / c.s_thr; + if (ratio > best_ratio) { + best_ratio = ratio; + verdict = c.resource; + } + } + std::println("analyze: verdict resource={} ratio={:.2f}", verdict, best_ratio); + std::println("analyze: done"); + return 0; +} + +void usage() { + std::println(stderr, "usage: app --resource cpu|mem|io|net --seconds N"); + std::println(stderr, " app analyze [--seconds N]"); +} + +} // namespace + +int main(int argc, char** argv) { + std::vector args(argv + 1, argv + argc); + + if (!args.empty() && args[0] == "analyze") { + int seconds = 60; + for (size_t i = 1; i < args.size(); ++i) { + if (args[i] == "--seconds" && i + 1 < args.size()) { + try { + seconds = std::stoi(args[++i]); + } catch (...) { + usage(); + return 2; + } + } else { + usage(); + return 2; + } + } + if (seconds <= 0) { + usage(); + return 2; + } + return cmd_analyze(seconds); + } + + std::string resource; + int seconds = -1; + for (size_t i = 0; i < args.size(); ++i) { + if (args[i] == "--resource" && i + 1 < args.size()) { + resource = args[++i]; + } else if (args[i] == "--seconds" && i + 1 < args.size()) { + try { + seconds = std::stoi(args[++i]); + } catch (...) { + usage(); + return 2; + } + } else { + usage(); + return 2; + } + } + if (resource != "cpu" && resource != "mem" && resource != "io" && resource != "net") { + usage(); + return 2; + } + if (seconds <= 0) { + usage(); + return 2; + } + return cmd_saturate(resource, seconds); +} diff --git a/examples/37-use-method-60s-analysis/demo.sh b/examples/37-use-method-60s-analysis/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/37-use-method-60s-analysis/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/37-use-method-60s-analysis/go/demo.sh b/examples/37-use-method-60s-analysis/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/37-use-method-60s-analysis/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/37-use-method-60s-analysis/go/go.mod b/examples/37-use-method-60s-analysis/go/go.mod new file mode 100644 index 0000000..58780d2 --- /dev/null +++ b/examples/37-use-method-60s-analysis/go/go.mod @@ -0,0 +1,7 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/37-use-method-60s-analysis/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 diff --git a/examples/37-use-method-60s-analysis/go/go.sum b/examples/37-use-method-60s-analysis/go/go.sum new file mode 100644 index 0000000..37ee2d4 --- /dev/null +++ b/examples/37-use-method-60s-analysis/go/go.sum @@ -0,0 +1,2 @@ +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= diff --git a/examples/37-use-method-60s-analysis/go/main.go b/examples/37-use-method-60s-analysis/go/main.go new file mode 100644 index 0000000..f7f8c33 --- /dev/null +++ b/examples/37-use-method-60s-analysis/go/main.go @@ -0,0 +1,779 @@ +// loadmix (chapter 37): a saturation generator + guided-analysis harness for +// the classic Brendan Gregg 60-second USE-method checklist. +// +// app --resource cpu|mem|io|net --seconds N saturate that resource +// app analyze [--seconds N] run the checklist and name +// the saturated resource +// +// `analyze` shells out to the real system tools (uptime, dmesg, vmstat, +// mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the +// checklist would, parses their plain-text tables generically (by column +// *name*, read from each tool's own header row, never a hardcoded position), +// and for each of the four resources reports which USE signals — Utilization +// and Saturation — fired against a fixed threshold. Errors is reported too, +// fixed false: this chapter induces load, not kernel-logged faults, so a +// real checklist run finds none. +// +// Go idioms used throughout, in place of the C++ reference's jthread pools, +// std::atomic counters, and std::expected: goroutines joined with +// sync.WaitGroup, atomic counters from sync/atomic, golang.org/x/sys/unix +// for the O_DSYNC pwrite path, and net.UDPConn with SetReadDeadline for the +// net saturator's receiver. Every observable line below is a byte-for-byte +// match of the C++ std::println output (field names, order, and spacing); +// only floating-point rounding may differ, which is why verify.lua checks +// shapes (%d+%.%d%d) rather than exact decimals. +package main + +import ( + "encoding/binary" + "fmt" + "net" + "os" + "os/exec" + "runtime" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" + + "golang.org/x/sys/unix" +) + +// ------------------------------------------------------------- utilities -- + +func nproc() int { + n := runtime.NumCPU() + if n == 0 { + return 1 + } + return n +} + +func maxInt(a, b int) int { + if a > b { + return a + } + return b +} + +func trim(s string) string { + return strings.Trim(s, " \t\r\n") +} + +func splitWs(line string) []string { + return strings.Fields(line) +} + +// splitLines mirrors std::getline over an istringstream: split on '\n', and +// drop the final empty element produced when the text ends with a trailing +// newline (getline never yields that last empty read). +func splitLines(text string) []string { + if text == "" { + return nil + } + parts := strings.Split(text, "\n") + if parts[len(parts)-1] == "" { + parts = parts[:len(parts)-1] + } + return parts +} + +// row is a parsed table row: column names (from a tool's own header line) +// zipped with this row's values, so every parser below reads fields *by +// name* rather than by a hardcoded position — robust to sysstat/procps +// column reordering across versions. +type row struct { + names []string + values []string +} + +func (r row) getOr(name string, def float64) float64 { + n := len(r.names) + if len(r.values) < n { + n = len(r.values) + } + for i := 0; i < n; i++ { + if r.names[i] == name { + v, err := strconv.ParseFloat(r.values[i], 64) + if err != nil { + return def + } + return v + } + } + return def +} + +func makeRow(headerTokens []string, headerSkip int, dataTokens []string, dataSkip int) row { + var r row + if headerSkip < len(headerTokens) { + r.names = append(r.names, headerTokens[headerSkip:]...) + } + if dataSkip < len(dataTokens) { + r.values = append(r.values, dataTokens[dataSkip:]...) + } + return r +} + +// captureResult is the Go analogue of std::expected: err +// is set only when the shell itself could not be started (the "truly +// fallible operation" per the book's convention), never when the captured +// command exits non-zero — its output is still valid text either way. +type captureResult struct { + out string + err error +} + +// runCapture runs a short command to completion and captures combined +// stdout+stderr. Forced to the C locale so decimal points/dates are +// unambiguous regardless of the guest's configured locale. Grouped in a +// subshell so a single trailing "2>&1" captures every stage of a pipeline +// (e.g. "dmesg | tail -n 5") — without the parens, 2>&1 binds only to the +// pipeline's last stage and an earlier stage's stderr (e.g. dmesg's own +// permission-denied message) leaks past the capture. +func runCapture(cmd string) captureResult { + full := "( env LC_ALL=C LANG=C " + cmd + " ) 2>&1" + out, err := exec.Command("sh", "-c", full).Output() + if err != nil { + if _, ok := err.(*exec.ExitError); ok { + // the command ran and produced output but exited non-zero -- + // that's not a capture failure, just like popen()/pclose() + // never surfacing the child's exit status to the caller here. + return captureResult{out: string(out)} + } + return captureResult{err: err} + } + return captureResult{out: string(out)} +} + +// runToFile runs `env LC_ALL=C LANG=C ` to completion, redirecting +// stdout+stderr to outfile. Used for the five interval samplers, each on its +// own goroutine, so they run *concurrently* — a 12s analyze window costs +// ~12s wall-clock, not 5x that from running them one at a time. +func runToFile(argv []string, outfile string) { + full := append([]string{"env", "LC_ALL=C", "LANG=C"}, argv...) + f, err := os.OpenFile(outfile, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600) + if err != nil { + return + } + defer f.Close() + cmd := exec.Command(full[0], full[1:]...) + cmd.Stdout = f + cmd.Stderr = f + _ = cmd.Run() +} + +func readAll(path string) string { + b, err := os.ReadFile(path) + if err != nil { + return "" + } + return string(b) +} + +// ------------------------------------------------------- checklist parsers -- +// Each parser reads a header line's column names from the tool's own output +// and zips subsequent data lines against that name list — see row.getOr +// above. + +type vmstatResult struct { + runQueueMax float64 + swapIOMax float64 // si+so, KB/s +} + +// vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its +// *first* sample is a since-boot average, not a live interval — both quirks +// are handled here: we scan every data row after the header, skip row 0, and +// take the max across the rest (the steady-state saturation signal, not +// diluted by an average across the whole window). +func parseVmstat(text string) vmstatResult { + var res vmstatResult + var header []string + var dataRows [][]string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) == 0 { + continue + } + if header == nil { + hasSwpd := false + for _, t := range toks { + if t == "swpd" { + hasSwpd = true + break + } + } + if hasSwpd { + header = toks + } + continue + } + if len(toks)+2 >= len(header) { + dataRows = append(dataRows, toks) + } + } + for i := 1; i < len(dataRows); i++ { // skip row 0 (since-boot average) + r := makeRow(header, 0, dataRows[i], 0) + res.runQueueMax = max(res.runQueueMax, r.getOr("r", 0.0)) + si, so := r.getOr("si", 0.0), r.getOr("so", 0.0) + res.swapIOMax = max(res.swapIOMax, si+so) + } + return res +} + +type mpstatResult struct { + busyPct float64 + iowaitPct float64 +} + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically — sysstat's job, not ours). +func parseMpstat(text string) mpstatResult { + res := mpstatResult{} + var header, data []string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) < 2 { + continue + } + if toks[0] == "Average:" && toks[1] == "CPU" { + header = toks + } else if toks[0] == "Average:" && toks[1] == "all" { + data = toks + } + } + if header == nil || data == nil { + return res + } + r := makeRow(header, 2, data, 2) + idle := r.getOr("%idle", 100.0) + res.iowaitPct = r.getOr("%iowait", 0.0) + res.busyPct = 100.0 - idle + return res +} + +func parseFreeUsedPct(text string) float64 { + var header, data []string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) == 0 { + continue + } + hasAvailable := false + for _, t := range toks { + if t == "available" { + hasAvailable = true + break + } + } + if hasAvailable { + header = toks + } else if toks[0] == "Mem:" { + data = toks + } + } + if header == nil || data == nil { + return 0.0 + } + r := makeRow(header, 0, data, 1) + total := r.getOr("total", 0.0) + avail := r.getOr("available", 0.0) + if total <= 0.0 { + return 0.0 + } + return (total - avail) / total * 100.0 +} + +type iostatResult struct { + utilMax float64 + awaitMax float64 // w_await, ms +} + +// iostat -xz has no Average line either, AND -z hides idle devices entirely +// (they just don't appear in a quiet interval) — so we scan every per-second +// block, skip block 0 (since-boot), skip loopback/removable/compressed +// pseudo-devices, and take the max over the guest's real block device. +func parseIostat(text string) iostatResult { + var res iostatResult + excl := []string{"zram", "sr", "loop", "dm-"} + blockIdx := -1 + var header []string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) == 0 { + continue + } + if toks[0] == "avg-cpu:" { + blockIdx++ + continue + } + if toks[0] == "Device" { + header = toks + continue + } + if header == nil || blockIdx < 1 { // no header yet, or since-boot block + continue + } + dev := toks[0] + skip := false + for _, p := range excl { + if strings.HasPrefix(dev, p) { + skip = true + break + } + } + if skip { + continue + } + r := makeRow(header, 1, toks, 1) + res.utilMax = max(res.utilMax, r.getOr("%util", 0.0)) + res.awaitMax = max(res.awaitMax, r.getOr("w_await", 0.0)) + } + return res +} + +func parseSarPkts(text, iface string) float64 { + var header, data []string + for _, line := range splitLines(text) { + toks := splitWs(line) + if len(toks) < 2 { + continue + } + if toks[0] == "Average:" && toks[1] == "IFACE" { + header = toks + } else if toks[0] == "Average:" && toks[1] == iface { + data = toks + } + } + if header == nil || data == nil { + return 0.0 + } + r := makeRow(header, 2, data, 2) + return r.getOr("rxpck/s", 0.0) + r.getOr("txpck/s", 0.0) +} + +// ---------------------------------------------------------------- saturate -- + +func satCPU(seconds, workers int) (uint64, uint64) { + deadline := time.Now().Add(time.Duration(seconds) * time.Second) + var totalIters, checksum uint64 + var wg sync.WaitGroup + wg.Add(workers) + for w := 0; w < workers; w++ { + go func(w int) { + defer wg.Done() + x := uint64(0x9E3779B97F4A7C15) ^ uint64(w+1) + var iters uint64 + for { + x ^= x << 7 + x ^= x >> 9 + iters++ + if iters&0xFFFFF == 0 && !time.Now().Before(deadline) { + break + } + } + atomic.AddUint64(&totalIters, iters) + atomic.AddUint64(&checksum, x) // keeps x provably live (no DCE) + }(w) + } + wg.Wait() // each goroutine only returns once its own deadline check fires + return totalIters, checksum +} + +func satMem(seconds, workers int) (uint64, uint64) { + var memKB uint64 + if data, err := os.ReadFile("/proc/meminfo"); err == nil { + for _, line := range strings.Split(string(data), "\n") { + if strings.HasPrefix(line, "MemTotal:") { + fields := splitWs(strings.TrimPrefix(line, "MemTotal:")) + if len(fields) > 0 { + if v, err := strconv.ParseUint(fields[0], 10, 64); err == nil { + memKB = v + } + } + break + } + } + } + var targetBytes uint64 + if memKB > 0 { + targetBytes = uint64(float64(memKB) * 1024.0 * 1.35) + } else { + targetBytes = 2 << 30 // 2 GiB fallback if /proc/meminfo is unreadable + } + chunk := targetBytes / uint64(workers) + if chunk < 4096*16 { + chunk = 4096 * 16 + } + + deadline := time.Now().Add(time.Duration(seconds) * time.Second) + var totalTouches uint64 + var wg sync.WaitGroup + wg.Add(workers) + for w := 0; w < workers; w++ { + go func() { + defer wg.Done() + buf := make([]byte, chunk) + var touches uint64 + for time.Now().Before(deadline) { + for off := 0; off < len(buf); off += 4096 { + buf[off]++ + touches++ + } + } + atomic.AddUint64(&totalTouches, touches) + }() + } + wg.Wait() + return totalTouches, targetBytes +} + +func satIO(seconds, workers int) (uint64, uint64) { + dir := fmt.Sprintf("/var/tmp/loadmix-io-%d", os.Getpid()) + _ = os.Mkdir(dir, 0700) + + deadline := time.Now().Add(time.Duration(seconds) * time.Second) + var totalWrites, totalBytes uint64 + var wg sync.WaitGroup + wg.Add(workers) + for w := 0; w < workers; w++ { + go func(w int) { + defer wg.Done() + path := fmt.Sprintf("%s/w%d.dat", dir, w) + fd, err := unix.Open(path, unix.O_WRONLY|unix.O_CREAT|unix.O_TRUNC|unix.O_DSYNC, 0600) + if err != nil { + return + } + const kBuf = 16384 + buf := make([]byte, kBuf) + // xorshift64*: cheap per-worker PRNG so the payload isn't + // trivially compressible (this guest's /var can be btrfs with + // zstd compression, which would otherwise let a real disk + // write shrink to nearly nothing and mute the iowait signal). + seed := uint64(0x2545F4914F6CDD1D) ^ uint64(w+1) + var writes, bytesWritten uint64 + for time.Now().Before(deadline) { + for i := 0; i < kBuf; i += 8 { + seed ^= seed << 13 + seed ^= seed >> 7 + seed ^= seed << 17 + binary.LittleEndian.PutUint64(buf[i:i+8], seed) + } + n, err := unix.Pwrite(fd, buf, 0) // same 16 KiB region: bounded file size + if err == nil && n > 0 { + writes++ + bytesWritten += uint64(n) + } + } + _ = unix.Close(fd) + _ = os.Remove(path) + atomic.AddUint64(&totalWrites, writes) + atomic.AddUint64(&totalBytes, bytesWritten) + }(w) + } + wg.Wait() + _ = os.Remove(dir) + return totalWrites, totalBytes +} + +func satNet(seconds, workers int) (uint64, uint64) { + rconn, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0}) + if err != nil { + return 0, 0 + } + defer rconn.Close() + port := rconn.LocalAddr().(*net.UDPAddr).Port + + var stop atomic.Bool + var recvWg sync.WaitGroup + recvWg.Add(1) + go func() { + defer recvWg.Done() + buf := make([]byte, 256) + for !stop.Load() { + _ = rconn.SetReadDeadline(time.Now().Add(200 * time.Millisecond)) // 200ms so the receiver notices stop promptly + _, _ = rconn.Read(buf) + } + }() + + deadline := time.Now().Add(time.Duration(seconds) * time.Second) + var totalPkts, totalBytes uint64 + var wg sync.WaitGroup + wg.Add(workers) + for w := 0; w < workers; w++ { + go func() { + defer wg.Done() + sconn, err := net.DialUDP("udp4", nil, &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: port}) + if err != nil { + return + } + defer sconn.Close() + payload := make([]byte, 64) + var pkts, bytesSent uint64 + for time.Now().Before(deadline) { + n, err := sconn.Write(payload) + if err == nil && n > 0 { + pkts++ + bytesSent += uint64(n) + } + } + atomic.AddUint64(&totalPkts, pkts) + atomic.AddUint64(&totalBytes, bytesSent) + }() + } + wg.Wait() // sender pool done + stop.Store(true) + recvWg.Wait() // receiver notices `stop` within ~200ms + return totalPkts, totalBytes +} + +func cmdSaturate(resource string, seconds int) int { + cpus := nproc() + var workers int + switch resource { + case "mem": + workers = maxInt(2, cpus) + case "cpu": + workers = maxInt(4, cpus*3) + default: // io, net + workers = maxInt(4, cpus*4) + } + + fmt.Printf("loadmix: start resource=%s seconds=%d workers=%d\n", resource, seconds, workers) + var ops, bytesN uint64 + switch resource { + case "cpu": + ops, bytesN = satCPU(seconds, workers) + case "mem": + ops, bytesN = satMem(seconds, workers) + case "io": + ops, bytesN = satIO(seconds, workers) + default: + ops, bytesN = satNet(seconds, workers) + } + fmt.Printf("loadmix: done resource=%s seconds=%d workers=%d ops=%d bytes=%d\n", + resource, seconds, workers, ops, bytesN) + return 0 +} + +// ------------------------------------------------------------------ analyze -- + +func boolStr(b bool) string { + if b { + return "true" + } + return "false" +} + +// runSamplers runs the five interval samplers CONCURRENTLY (each redirected +// to its own temp file), waits for all of them, then reads and deletes each +// file, returning the captured text keyed by tool name. +func runSamplers(seconds int, dir string) map[string]string { + n := strconv.Itoa(seconds) + type sampler struct { + name string + argv []string + file string + } + samplers := []sampler{ + {"vmstat", []string{"vmstat", "1", n}, dir + "/vmstat.out"}, + {"mpstat", []string{"mpstat", "-P", "ALL", "1", n}, dir + "/mpstat.out"}, + {"iostat", []string{"iostat", "-xz", "1", n}, dir + "/iostat.out"}, + {"sar", []string{"sar", "-n", "DEV", "1", n}, dir + "/sar.out"}, + {"pidstat", []string{"pidstat", "1", n}, dir + "/pidstat.out"}, + } + var wg sync.WaitGroup + wg.Add(len(samplers)) + for _, s := range samplers { + go func(s sampler) { + defer wg.Done() + runToFile(s.argv, s.file) + }(s) + } + wg.Wait() + + out := make(map[string]string, len(samplers)) + for _, s := range samplers { + out[s.name] = readAll(s.file) + _ = os.Remove(s.file) + } + return out +} + +func cmdAnalyze(seconds int) int { + cpus := nproc() + fmt.Printf("analyze: start seconds=%d cpus=%d\n", seconds, cpus) + + up := runCapture("uptime") + uptimeRaw := "unavailable" + if up.err == nil { + uptimeRaw = trim(up.out) + } + fmt.Printf("analyze: tool uptime raw=\"%s\"\n", uptimeRaw) + + dm := runCapture("dmesg | tail -n 5") + dmStatus := "ok" + if dm.err != nil || strings.Contains(dm.out, "Operation not permitted") { + dmStatus = "denied" + } + fmt.Printf("analyze: tool dmesg status=%s\n", dmStatus) + + topOut := runCapture("top -b -n1") + var cpuLine, tasksLine string + if topOut.err == nil { + for _, l := range splitLines(topOut.out) { + if strings.HasPrefix(l, "%Cpu") { + cpuLine = trim(l) + } + if strings.HasPrefix(l, "Tasks:") { + tasksLine = trim(l) + } + } + } + fmt.Printf("analyze: tool top cpu=\"%s\" tasks=\"%s\"\n", cpuLine, tasksLine) + + ssOut := runCapture("ss -s") + var ssLine string + if ssOut.err == nil { + lines := splitLines(ssOut.out) + if len(lines) > 0 { + ssLine = trim(lines[0]) + } + } + fmt.Printf("analyze: tool ss raw=\"%s\"\n", ssLine) + + // The five interval samplers run CONCURRENTLY so an N-second checklist + // costs ~N seconds, not 5N. + dir := fmt.Sprintf("/tmp/loadmix-analyze-%d", os.Getpid()) + _ = os.Mkdir(dir, 0700) + samplerOut := runSamplers(seconds, dir) + _ = os.Remove(dir) + + active := 0 + for _, l := range splitLines(samplerOut["pidstat"]) { + t := splitWs(l) + if len(t) >= 2 && t[0] == "Average:" && t[1] != "UID" { + active++ + } + } + fmt.Printf("analyze: tool pidstat active_processes=%d\n", active) + + freeOut := runCapture("free -m") + memUsedPct := 0.0 + if freeOut.err == nil { + memUsedPct = parseFreeUsedPct(freeOut.out) + } + + vm := parseVmstat(samplerOut["vmstat"]) + mp := parseMpstat(samplerOut["mpstat"]) + io := parseIostat(samplerOut["iostat"]) + netPkts := parseSarPkts(samplerOut["sar"], "lo") + + fmt.Printf("analyze: metric resource=cpu name=busy_pct value=%.2f unit=pct\n", mp.busyPct) + fmt.Printf("analyze: metric resource=cpu name=run_queue value=%.2f unit=procs\n", vm.runQueueMax) + fmt.Printf("analyze: metric resource=mem name=used_pct value=%.2f unit=pct\n", memUsedPct) + fmt.Printf("analyze: metric resource=mem name=swap_io value=%.2f unit=kbps\n", vm.swapIOMax) + fmt.Printf("analyze: metric resource=io name=util_pct value=%.2f unit=pct\n", io.utilMax) + fmt.Printf("analyze: metric resource=io name=iowait_pct value=%.2f unit=pct\n", mp.iowaitPct) + fmt.Printf("analyze: metric resource=io name=await_ms value=%.2f unit=ms\n", io.awaitMax) + fmt.Printf("analyze: metric resource=net name=pkts value=%.2f unit=per_s\n", netPkts) + + // signal thresholds: Utilization/Saturation pair per resource. cpu's + // saturation threshold scales with this guest's own core count — a run + // queue longer than (cpus + headroom) means threads are waiting for a + // CPU, the textbook USE definition of saturation. + type candidate struct { + resource string + uVal, uThr float64 + sVal, sThr float64 + } + table := []candidate{ + {"cpu", mp.busyPct, 60.0, vm.runQueueMax, float64(cpus) + 2.0}, + {"mem", memUsedPct, 75.0, vm.swapIOMax, 300.0}, + {"io", io.utilMax, 40.0, mp.iowaitPct, 8.0}, + {"net", netPkts, 2000.0, netPkts, 8000.0}, + } + + verdict := "" + bestRatio := -1.0 + for _, c := range table { + ufired := c.uVal >= c.uThr + sfired := c.sVal >= c.sThr + fmt.Printf("analyze: signal resource=%s type=Utilization fired=%s value=%.2f threshold=%.2f\n", + c.resource, boolStr(ufired), c.uVal, c.uThr) + fmt.Printf("analyze: signal resource=%s type=Saturation fired=%s value=%.2f threshold=%.2f\n", + c.resource, boolStr(sfired), c.sVal, c.sThr) + fmt.Printf("analyze: signal resource=%s type=Errors fired=false\n", c.resource) + ratio := c.sVal / c.sThr + if ratio > bestRatio { + bestRatio = ratio + verdict = c.resource + } + } + fmt.Printf("analyze: verdict resource=%s ratio=%.2f\n", verdict, bestRatio) + fmt.Println("analyze: done") + return 0 +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: app --resource cpu|mem|io|net --seconds N") + fmt.Fprintln(os.Stderr, " app analyze [--seconds N]") +} + +func main() { + args := os.Args[1:] + + if len(args) > 0 && args[0] == "analyze" { + seconds := 60 + for i := 1; i < len(args); { + if args[i] == "--seconds" && i+1 < len(args) { + v, err := strconv.Atoi(args[i+1]) + if err != nil { + usage() + os.Exit(2) + } + seconds = v + i += 2 + } else { + usage() + os.Exit(2) + } + } + if seconds <= 0 { + usage() + os.Exit(2) + } + os.Exit(cmdAnalyze(seconds)) + } + + resource := "" + seconds := -1 + for i := 0; i < len(args); { + switch { + case args[i] == "--resource" && i+1 < len(args): + resource = args[i+1] + i += 2 + case args[i] == "--seconds" && i+1 < len(args): + v, err := strconv.Atoi(args[i+1]) + if err != nil { + usage() + os.Exit(2) + } + seconds = v + i += 2 + default: + usage() + os.Exit(2) + } + } + if resource != "cpu" && resource != "mem" && resource != "io" && resource != "net" { + usage() + os.Exit(2) + } + if seconds <= 0 { + usage() + os.Exit(2) + } + os.Exit(cmdSaturate(resource, seconds)) +} diff --git a/examples/37-use-method-60s-analysis/rust/Cargo.lock b/examples/37-use-method-60s-analysis/rust/Cargo.lock new file mode 100644 index 0000000..306c006 --- /dev/null +++ b/examples/37-use-method-60s-analysis/rust/Cargo.lock @@ -0,0 +1,16 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "loadmix" +version = "0.1.0" +dependencies = [ + "libc", +] diff --git a/examples/37-use-method-60s-analysis/rust/Cargo.toml b/examples/37-use-method-60s-analysis/rust/Cargo.toml new file mode 100644 index 0000000..9c44c0b --- /dev/null +++ b/examples/37-use-method-60s-analysis/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "loadmix" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +libc = "0.2" diff --git a/examples/37-use-method-60s-analysis/rust/demo.sh b/examples/37-use-method-60s-analysis/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/37-use-method-60s-analysis/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/37-use-method-60s-analysis/rust/rust-toolchain.toml b/examples/37-use-method-60s-analysis/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/37-use-method-60s-analysis/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/37-use-method-60s-analysis/rust/src/main.rs b/examples/37-use-method-60s-analysis/rust/src/main.rs new file mode 100644 index 0000000..e7fffcc --- /dev/null +++ b/examples/37-use-method-60s-analysis/rust/src/main.rs @@ -0,0 +1,784 @@ +// loadmix (chapter 37): a saturation generator + guided-analysis harness for +// the classic Brendan Gregg 60-second USE-method checklist. +// +// app --resource cpu|mem|io|net --seconds N saturate that resource +// app analyze [--seconds N] run the checklist and name +// the saturated resource +// +// `analyze` shells out to the real system tools (uptime, dmesg, vmstat, +// mpstat, pidstat, iostat, free, sar, ss, top) exactly as a human running the +// checklist would, parses their plain-text tables generically (by column +// *name*, read from each tool's own header row, never a hardcoded position), +// and for each of the four resources reports which USE signals -- Utilization +// and Saturation -- fired against a fixed threshold. Errors is reported too, +// fixed false: this chapter induces load, not kernel-logged faults, so a +// real checklist run finds none. +// +// Rust idioms used throughout, in place of the C++ reference's jthread pools, +// std::atomic counters, and std::expected: std::thread::scope (the jthread +// analogue -- worker threads are RAII-joined when the scope block ends), +// AtomicU64 counters shared across them by reference, std::process::Command +// for every subprocess capture and spawn, and OpenOptionsExt::custom_flags +// plus FileExt::write_at for the O_DSYNC pwrite path. Every observable line +// below is a byte-for-byte match of the C++ std::println output (field +// names, order, and spacing); only floating-point rounding may differ, which +// is why verify.lua checks shapes (%d+%.%d%d) rather than exact decimals. + +use std::fs::{self, File, OpenOptions}; +use std::io; +use std::net::UdpSocket; +use std::os::unix::fs::{FileExt, OpenOptionsExt}; +use std::path::Path; +use std::process::{Child, Command}; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::time::{Duration, Instant}; + +// ------------------------------------------------------------- utilities -- + +fn nproc() -> usize { + std::thread::available_parallelism() + .map(|n| n.get()) + .unwrap_or(1) +} + +fn trim(s: &str) -> String { + s.trim_matches(|c: char| c == ' ' || c == '\t' || c == '\r' || c == '\n') + .to_string() +} + +fn split_ws(line: &str) -> Vec { + line.split_whitespace().map(String::from).collect() +} + +// Mirrors std::getline over an istringstream: split on '\n', and drop the +// final empty element produced when the text ends with a trailing newline +// (getline never yields that last empty read). +fn split_lines(text: &str) -> Vec { + if text.is_empty() { + return Vec::new(); + } + let mut parts: Vec = text.split('\n').map(String::from).collect(); + if parts.last().is_some_and(|s| s.is_empty()) { + parts.pop(); + } + parts +} + +// A parsed table row: column names (from a tool's own header line) zipped +// with this row's values, so every parser below reads fields *by name* +// rather than by a hardcoded position -- robust to sysstat/procps column +// reordering across versions. +struct Row { + names: Vec, + values: Vec, +} + +impl Row { + fn get(&self, name: &str) -> Option { + let n = self.names.len().min(self.values.len()); + for i in 0..n { + if self.names[i] == name { + return self.values[i].parse::().ok(); + } + } + None + } +} + +fn make_row( + header_tokens: &[String], + header_skip: usize, + data_tokens: &[String], + data_skip: usize, +) -> Row { + let names = header_tokens.get(header_skip..).unwrap_or(&[]).to_vec(); + let values = data_tokens.get(data_skip..).unwrap_or(&[]).to_vec(); + Row { names, values } +} + +// Run a short command to completion and capture combined stdout+stderr. +// Forced to the C locale so decimal points/dates are unambiguous regardless +// of the guest's configured locale. The one fallible operation in this file +// that returns a Result, per book convention (the C++ reference uses +// std::expected here). +fn run_capture(cmd: &str) -> io::Result { + // Grouped in a subshell so a single trailing "2>&1" captures every stage + // of a pipeline (e.g. "dmesg | tail -n 5") -- without the parens, 2>&1 + // binds only to the pipeline's last stage and an earlier stage's stderr + // (e.g. dmesg's own permission-denied message) leaks past the capture. + let full = format!("( env LC_ALL=C LANG=C {cmd} ) 2>&1"); + let output = Command::new("sh").arg("-c").arg(full).output()?; + Ok(String::from_utf8_lossy(&output.stdout).into_owned()) +} + +// Start `env LC_ALL=C LANG=C ` in the background, stdout+stderr +// redirected to outfile; returns the child (or None on spawn failure). Used +// for the five interval samplers so they run *concurrently* -- a 12s analyze +// window costs ~12s wall-clock, not 5x that from running them one at a time. +fn spawn_to_file(argv: &[&str], outfile: &Path) -> Option { + let out = File::create(outfile).ok()?; + let err = out.try_clone().ok()?; + Command::new("env") + .arg("LC_ALL=C") + .arg("LANG=C") + .args(argv) + .stdout(out) + .stderr(err) + .spawn() + .ok() +} + +fn wait_for(child: Option) { + if let Some(mut c) = child { + let _ = c.wait(); + } +} + +fn read_all(path: &str) -> String { + fs::read_to_string(path).unwrap_or_default() +} + +// ------------------------------------------------------- checklist parsers -- +// Each parser reads a header line's column names from the tool's own output +// and zips subsequent data lines against that name list -- see Row::get +// above. + +#[derive(Default)] +struct VmstatResult { + run_queue_max: f64, + swap_io_max: f64, // si+so, KB/s +} + +// vmstat has no aggregate "Average:" line (unlike mpstat/pidstat), and its +// *first* sample is a since-boot average, not a live interval -- both quirks +// are handled here: we scan every data row after the header, skip row 0, and +// take the max across the rest (the steady-state saturation signal, not +// diluted by an average across the whole window). +fn parse_vmstat(text: &str) -> VmstatResult { + let mut res = VmstatResult::default(); + let mut header: Vec = Vec::new(); + let mut data_rows: Vec> = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.is_empty() { + continue; + } + if header.is_empty() { + if toks.iter().any(|t| t == "swpd") { + header = toks; + } + continue; + } + if toks.len() + 2 >= header.len() { + data_rows.push(toks); + } + } + for row in data_rows.iter().skip(1) { + // skip row 0 (since-boot average) + let r = make_row(&header, 0, row, 0); + res.run_queue_max = res.run_queue_max.max(r.get("r").unwrap_or(0.0)); + let si = r.get("si").unwrap_or(0.0); + let so = r.get("so").unwrap_or(0.0); + res.swap_io_max = res.swap_io_max.max(si + so); + } + res +} + +#[derive(Default)] +struct MpstatResult { + busy_pct: f64, + iowait_pct: f64, +} + +// mpstat -P ALL's own "Average:" row over the whole window (excludes the +// since-boot first report automatically -- sysstat's job, not ours). +fn parse_mpstat(text: &str) -> MpstatResult { + let mut res = MpstatResult::default(); + let mut header: Vec = Vec::new(); + let mut data: Vec = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.len() < 2 { + continue; + } + if toks[0] == "Average:" && toks[1] == "CPU" { + header = toks; + } else if toks[0] == "Average:" && toks[1] == "all" { + data = toks; + } + } + if header.is_empty() || data.is_empty() { + return res; + } + let r = make_row(&header, 2, &data, 2); + let idle = r.get("%idle").unwrap_or(100.0); + res.iowait_pct = r.get("%iowait").unwrap_or(0.0); + res.busy_pct = 100.0 - idle; + res +} + +fn parse_free_used_pct(text: &str) -> f64 { + let mut header: Vec = Vec::new(); + let mut data: Vec = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.is_empty() { + continue; + } + if toks.iter().any(|t| t == "available") { + header = toks; + } else if toks[0] == "Mem:" { + data = toks; + } + } + if header.is_empty() || data.is_empty() { + return 0.0; + } + let r = make_row(&header, 0, &data, 1); + let total = r.get("total").unwrap_or(0.0); + let avail = r.get("available").unwrap_or(0.0); + if total <= 0.0 { + return 0.0; + } + (total - avail) / total * 100.0 +} + +#[derive(Default)] +struct IostatResult { + util_max: f64, + await_max: f64, // w_await, ms +} + +// iostat -xz has no Average line either, AND -z hides idle devices entirely +// (they just don't appear in a quiet interval) -- so we scan every per-second +// block, skip block 0 (since-boot), skip loopback/removable/compressed +// pseudo-devices, and take the max over the guest's real block device. +fn parse_iostat(text: &str) -> IostatResult { + let mut res = IostatResult::default(); + const EXCL: [&str; 4] = ["zram", "sr", "loop", "dm-"]; + let mut block_idx: i32 = -1; + let mut header: Vec = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.is_empty() { + continue; + } + if toks[0] == "avg-cpu:" { + block_idx += 1; + continue; + } + if toks[0] == "Device" { + header = toks; + continue; + } + if header.is_empty() || block_idx < 1 { + continue; // no header yet, or since-boot block + } + let dev = &toks[0]; + if EXCL.iter().any(|&p| dev.starts_with(p)) { + continue; + } + let r = make_row(&header, 1, &toks, 1); + res.util_max = res.util_max.max(r.get("%util").unwrap_or(0.0)); + res.await_max = res.await_max.max(r.get("w_await").unwrap_or(0.0)); + } + res +} + +fn parse_sar_pkts(text: &str, iface: &str) -> f64 { + let mut header: Vec = Vec::new(); + let mut data: Vec = Vec::new(); + for line in split_lines(text) { + let toks = split_ws(&line); + if toks.len() < 2 { + continue; + } + if toks[0] == "Average:" && toks[1] == "IFACE" { + header = toks; + } else if toks[0] == "Average:" && toks[1] == iface { + data = toks; + } + } + if header.is_empty() || data.is_empty() { + return 0.0; + } + let r = make_row(&header, 2, &data, 2); + r.get("rxpck/s").unwrap_or(0.0) + r.get("txpck/s").unwrap_or(0.0) +} + +// ---------------------------------------------------------------- saturate -- + +fn sat_cpu(seconds: u64, workers: usize) -> (u64, u64) { + let deadline = Instant::now() + Duration::from_secs(seconds); + let total_iters = AtomicU64::new(0); + let checksum = AtomicU64::new(0); + std::thread::scope(|s| { + for w in 0..workers { + let total_iters = &total_iters; + let checksum = &checksum; + s.spawn(move || { + let mut x: u64 = 0x9E3779B97F4A7C15u64 ^ (w as u64 + 1); + let mut iters: u64 = 0; + loop { + x ^= x << 7; + x ^= x >> 9; + iters += 1; + if iters & 0xFFFFF == 0 && Instant::now() >= deadline { + break; + } + } + total_iters.fetch_add(iters, Ordering::Relaxed); + checksum.fetch_add(x, Ordering::Relaxed); // keeps x provably live (no DCE) + }); + } + }); // worker threads join here -- thread::scope is the jthread analogue + ( + total_iters.load(Ordering::Relaxed), + checksum.load(Ordering::Relaxed), + ) +} + +fn sat_mem(seconds: u64, workers: usize) -> (u64, u64) { + let mut mem_kb: u64 = 0; + if let Ok(text) = fs::read_to_string("/proc/meminfo") { + for line in text.lines() { + if let Some(rest) = line.strip_prefix("MemTotal:") { + if let Some(tok) = rest.split_whitespace().next() { + mem_kb = tok.parse().unwrap_or(0); + } + break; + } + } + } + let target_bytes: u64 = if mem_kb > 0 { + (mem_kb as f64 * 1024.0 * 1.35) as u64 + } else { + 2u64 << 30 // 2 GiB fallback if /proc/meminfo is unreadable + }; + let chunk: u64 = (target_bytes / workers as u64).max(4096 * 16); + + let deadline = Instant::now() + Duration::from_secs(seconds); + let total_touches = AtomicU64::new(0); + std::thread::scope(|s| { + for _ in 0..workers { + let total_touches = &total_touches; + s.spawn(move || { + let mut buf = vec![0u8; chunk as usize]; + let mut touches: u64 = 0; + while Instant::now() < deadline { + let mut off = 0usize; + while off < buf.len() { + buf[off] = buf[off].wrapping_add(1); + touches += 1; + off += 4096; + } + } + total_touches.fetch_add(touches, Ordering::Relaxed); + }); + } + }); + (total_touches.load(Ordering::Relaxed), target_bytes) +} + +fn sat_io(seconds: u64, workers: usize) -> (u64, u64) { + let dir = format!("/var/tmp/loadmix-io-{}", std::process::id()); + let _ = fs::create_dir(&dir); + + let deadline = Instant::now() + Duration::from_secs(seconds); + let total_writes = AtomicU64::new(0); + let total_bytes = AtomicU64::new(0); + std::thread::scope(|s| { + for w in 0..workers { + let path = format!("{dir}/w{w}.dat"); + let total_writes = &total_writes; + let total_bytes = &total_bytes; + s.spawn(move || { + let file = match OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o600) + .custom_flags(libc::O_DSYNC) + .open(&path) + { + Ok(f) => f, + Err(_) => return, + }; + const K_BUF: usize = 16384; + let mut buf = [0u8; K_BUF]; + // xorshift64*: cheap per-thread PRNG so the payload isn't + // trivially compressible (this guest's /var is btrfs with + // zstd:1 compression, which would otherwise let a real disk + // write shrink to nearly nothing and mute the iowait signal). + let mut seed: u64 = 0x2545F4914F6CDD1Du64 ^ (w as u64 + 1); + let mut writes: u64 = 0; + let mut bytes: u64 = 0; + while Instant::now() < deadline { + let mut i = 0usize; + while i < K_BUF { + seed ^= seed << 13; + seed ^= seed >> 7; + seed ^= seed << 17; + buf[i..i + 8].copy_from_slice(&seed.to_le_bytes()); + i += 8; + } + if let Ok(n) = file.write_at(&buf, 0) { + // same 16 KiB region: bounded file size + if n > 0 { + writes += 1; + bytes += n as u64; + } + } + } + drop(file); + let _ = fs::remove_file(&path); + total_writes.fetch_add(writes, Ordering::Relaxed); + total_bytes.fetch_add(bytes, Ordering::Relaxed); + }); + } + }); + let _ = fs::remove_dir(&dir); + ( + total_writes.load(Ordering::Relaxed), + total_bytes.load(Ordering::Relaxed), + ) +} + +fn sat_net(seconds: u64, workers: usize) -> (u64, u64) { + let rsock = match UdpSocket::bind("127.0.0.1:0") { + Ok(s) => s, + Err(_) => return (0, 0), + }; + let _ = rsock.set_read_timeout(Some(Duration::from_millis(200))); // 200ms so the receiver notices `stop` promptly + let port = match rsock.local_addr() { + Ok(a) => a.port(), + Err(_) => return (0, 0), + }; + + let stop = AtomicBool::new(false); + let total_pkts = AtomicU64::new(0); + let total_bytes = AtomicU64::new(0); + std::thread::scope(|s| { + let stop_ref = &stop; + let rsock_ref = &rsock; + s.spawn(move || { + let mut buf = [0u8; 256]; + while !stop_ref.load(Ordering::Relaxed) { + let _ = rsock_ref.recv(&mut buf); + } + }); + let deadline = Instant::now() + Duration::from_secs(seconds); + std::thread::scope(|s2| { + for _ in 0..workers { + let total_pkts = &total_pkts; + let total_bytes = &total_bytes; + s2.spawn(move || { + let sock = match UdpSocket::bind("0.0.0.0:0") { + Ok(s) => s, + Err(_) => return, + }; + if sock.connect(("127.0.0.1", port)).is_err() { + return; + } + let payload = [0u8; 64]; + let mut pkts: u64 = 0; + let mut bytes: u64 = 0; + while Instant::now() < deadline { + if let Ok(n) = sock.send(&payload) + && n > 0 + { + pkts += 1; + bytes += n as u64; + } + } + total_pkts.fetch_add(pkts, Ordering::Relaxed); + total_bytes.fetch_add(bytes, Ordering::Relaxed); + }); + } + }); // sender pool joins here + stop.store(true, Ordering::Relaxed); + }); // receiver joins here (notices `stop` within ~200ms) + ( + total_pkts.load(Ordering::Relaxed), + total_bytes.load(Ordering::Relaxed), + ) +} + +fn cmd_saturate(resource: &str, seconds: i32) -> i32 { + let cpus = nproc(); + let workers = if resource == "mem" { + cpus.max(2) + } else if resource == "cpu" { + (cpus * 3).max(4) + } else { + (cpus * 4).max(4) + }; + + println!("loadmix: start resource={resource} seconds={seconds} workers={workers}"); + let (ops, bytes) = match resource { + "cpu" => sat_cpu(seconds as u64, workers), + "mem" => sat_mem(seconds as u64, workers), + "io" => sat_io(seconds as u64, workers), + _ => sat_net(seconds as u64, workers), + }; + println!( + "loadmix: done resource={resource} seconds={seconds} workers={workers} ops={ops} bytes={bytes}" + ); + 0 +} + +// ------------------------------------------------------------------ analyze -- + +fn cmd_analyze(seconds: i32) -> i32 { + let cpus = nproc(); + println!("analyze: start seconds={seconds} cpus={cpus}"); + + let uptime_raw = match run_capture("uptime") { + Ok(s) => trim(&s), + Err(_) => "unavailable".to_string(), + }; + println!("analyze: tool uptime raw=\"{uptime_raw}\""); + + let dm = run_capture("dmesg | tail -n 5"); + let dm_status = match &dm { + Ok(s) if !s.contains("Operation not permitted") => "ok", + _ => "denied", + }; + println!("analyze: tool dmesg status={dm_status}"); + + let top_out = run_capture("top -b -n1"); + let mut cpu_line = String::new(); + let mut tasks_line = String::new(); + if let Ok(text) = &top_out { + for l in split_lines(text) { + if l.starts_with("%Cpu") { + cpu_line = trim(&l); + } + if l.starts_with("Tasks:") { + tasks_line = trim(&l); + } + } + } + println!("analyze: tool top cpu=\"{cpu_line}\" tasks=\"{tasks_line}\""); + + let ss_out = run_capture("ss -s"); + let mut ss_line = String::new(); + if let Ok(text) = &ss_out + && let Some(first) = split_lines(text).first() + { + ss_line = trim(first); + } + println!("analyze: tool ss raw=\"{ss_line}\""); + + // The five interval samplers run CONCURRENTLY (each redirected to its own + // temp file) so an N-second checklist costs ~N seconds, not 5N. + let dir = format!("/tmp/loadmix-analyze-{}", std::process::id()); + let _ = fs::create_dir(&dir); + let vfile = format!("{dir}/vmstat.out"); + let mfile = format!("{dir}/mpstat.out"); + let ifile = format!("{dir}/iostat.out"); + let sfile = format!("{dir}/sar.out"); + let pfile = format!("{dir}/pidstat.out"); + let n = seconds.to_string(); + let pv = spawn_to_file(&["vmstat", "1", &n], Path::new(&vfile)); + let pm = spawn_to_file(&["mpstat", "-P", "ALL", "1", &n], Path::new(&mfile)); + let pi = spawn_to_file(&["iostat", "-xz", "1", &n], Path::new(&ifile)); + let ps = spawn_to_file(&["sar", "-n", "DEV", "1", &n], Path::new(&sfile)); + let pp = spawn_to_file(&["pidstat", "1", &n], Path::new(&pfile)); + wait_for(pv); + wait_for(pm); + wait_for(pi); + wait_for(ps); + wait_for(pp); + let vtext = read_all(&vfile); + let mtext = read_all(&mfile); + let itext = read_all(&ifile); + let stext = read_all(&sfile); + let ptext = read_all(&pfile); + let _ = fs::remove_file(&vfile); + let _ = fs::remove_file(&mfile); + let _ = fs::remove_file(&ifile); + let _ = fs::remove_file(&sfile); + let _ = fs::remove_file(&pfile); + let _ = fs::remove_dir(&dir); + + let mut active = 0; + for l in split_lines(&ptext) { + let t = split_ws(&l); + if t.len() >= 2 && t[0] == "Average:" && t[1] != "UID" { + active += 1; + } + } + println!("analyze: tool pidstat active_processes={active}"); + + let mem_used_pct = match run_capture("free -m") { + Ok(s) => parse_free_used_pct(&s), + Err(_) => 0.0, + }; + + let vm = parse_vmstat(&vtext); + let mp = parse_mpstat(&mtext); + let io = parse_iostat(&itext); + let net_pkts = parse_sar_pkts(&stext, "lo"); + + println!( + "analyze: metric resource=cpu name=busy_pct value={:.2} unit=pct", + mp.busy_pct + ); + println!( + "analyze: metric resource=cpu name=run_queue value={:.2} unit=procs", + vm.run_queue_max + ); + println!("analyze: metric resource=mem name=used_pct value={mem_used_pct:.2} unit=pct"); + println!( + "analyze: metric resource=mem name=swap_io value={:.2} unit=kbps", + vm.swap_io_max + ); + println!( + "analyze: metric resource=io name=util_pct value={:.2} unit=pct", + io.util_max + ); + println!( + "analyze: metric resource=io name=iowait_pct value={:.2} unit=pct", + mp.iowait_pct + ); + println!( + "analyze: metric resource=io name=await_ms value={:.2} unit=ms", + io.await_max + ); + println!("analyze: metric resource=net name=pkts value={net_pkts:.2} unit=per_s"); + + // signal thresholds: Utilization/Saturation pair per resource. cpu's + // saturation threshold scales with this guest's own core count -- a run + // queue longer than (cpus + headroom) means threads are waiting for a + // CPU, the textbook USE definition of saturation. + struct Candidate { + resource: &'static str, + u_val: f64, + u_thr: f64, + s_val: f64, + s_thr: f64, + } + let table = [ + Candidate { + resource: "cpu", + u_val: mp.busy_pct, + u_thr: 60.0, + s_val: vm.run_queue_max, + s_thr: cpus as f64 + 2.0, + }, + Candidate { + resource: "mem", + u_val: mem_used_pct, + u_thr: 75.0, + s_val: vm.swap_io_max, + s_thr: 300.0, + }, + Candidate { + resource: "io", + u_val: io.util_max, + u_thr: 40.0, + s_val: mp.iowait_pct, + s_thr: 8.0, + }, + Candidate { + resource: "net", + u_val: net_pkts, + u_thr: 2000.0, + s_val: net_pkts, + s_thr: 8000.0, + }, + ]; + + let mut verdict = String::new(); + let mut best_ratio = -1.0f64; + for c in &table { + let ufired = c.u_val >= c.u_thr; + let sfired = c.s_val >= c.s_thr; + println!( + "analyze: signal resource={} type=Utilization fired={} value={:.2} threshold={:.2}", + c.resource, ufired, c.u_val, c.u_thr + ); + println!( + "analyze: signal resource={} type=Saturation fired={} value={:.2} threshold={:.2}", + c.resource, sfired, c.s_val, c.s_thr + ); + println!( + "analyze: signal resource={} type=Errors fired=false", + c.resource + ); + let ratio = c.s_val / c.s_thr; + if ratio > best_ratio { + best_ratio = ratio; + verdict = c.resource.to_string(); + } + } + println!("analyze: verdict resource={verdict} ratio={best_ratio:.2}"); + println!("analyze: done"); + 0 +} + +fn usage() { + eprintln!("usage: app --resource cpu|mem|io|net --seconds N"); + eprintln!(" app analyze [--seconds N]"); +} + +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + + if args.first().map(String::as_str) == Some("analyze") { + let mut seconds: i32 = 60; + let mut i = 1; + while i < args.len() { + if args[i] == "--seconds" && i + 1 < args.len() { + match args[i + 1].parse::() { + Ok(v) => seconds = v, + Err(_) => { + usage(); + std::process::exit(2); + } + } + i += 2; + } else { + usage(); + std::process::exit(2); + } + } + if seconds <= 0 { + usage(); + std::process::exit(2); + } + std::process::exit(cmd_analyze(seconds)); + } + + let mut resource = String::new(); + let mut seconds: i32 = -1; + let mut i = 0; + while i < args.len() { + if args[i] == "--resource" && i + 1 < args.len() { + resource = args[i + 1].clone(); + i += 2; + } else if args[i] == "--seconds" && i + 1 < args.len() { + match args[i + 1].parse::() { + Ok(v) => seconds = v, + Err(_) => { + usage(); + std::process::exit(2); + } + } + i += 2; + } else { + usage(); + std::process::exit(2); + } + } + if !["cpu", "mem", "io", "net"].contains(&resource.as_str()) { + usage(); + std::process::exit(2); + } + if seconds <= 0 { + usage(); + std::process::exit(2); + } + std::process::exit(cmd_saturate(&resource, seconds)); +} diff --git a/examples/37-use-method-60s-analysis/verify.lua b/examples/37-use-method-60s-analysis/verify.lua new file mode 100644 index 0000000..d6600ed --- /dev/null +++ b/examples/37-use-method-60s-analysis/verify.lua @@ -0,0 +1,379 @@ +-- Verify loadmix (37-use-method-60s-analysis) for LSP_LANG. A vm-mode +-- example: driven directly against systems-target via plain scp/ssh to +-- /tmp/lsp37-/app, never `demo.sh run` under TARGET -- +-- deploy-to-vm.sh's `ssh -t` injects CRLF that breaks Lua patterns, and its +-- fixed /home/fedora/app path can be poisoned by a stale binary left over +-- from an unrelated example (this happened once during golden capture). +-- Every remote invocation below uses the full absolute remote_bin path -- +-- never a leading `cd` that a later `;`-separated command could silently +-- fail to inherit. +-- +-- Asserts observable behavior, identical in shape across cpp/go/rust: +-- 1. build, then stage to /tmp/lsp37-/app -- both must exit 0. +-- 2. error shapes: no args / bad --resource / --seconds 0 / bad analyze +-- flag all exit 2 with the two-line usage on stderr. +-- 3. saturate cpu/io/net (and mem) at a short --seconds: exit 0, the +-- start/done line shapes, the documented worker-count formula against +-- the guest's real nproc, ops>0 (mem excepted -- the golden capture +-- found ops legitimately 0 at a short duration, since allocation alone +-- can consume the whole budget), and the elapsed wall time actually +-- tracks --seconds for cpu/io/net -- this is exactly what would have +-- caught the pre-fix saturator bug where sat_* exited in milliseconds +-- regardless of the requested duration. mem gets a generous (x5) +-- ceiling instead of a tight window, since it can legitimately +-- *overrun* --seconds by several seconds under real swap pressure +-- (the inner touch loop only checks the stop flag between passes, per +-- the golden-capture calibration notes). +-- 4. idle `analyze --seconds N`: the exact skeleton -- 1 start line, 5 +-- tool lines, 8 metric lines (each %d+%.%d%d), 12 signal lines in +-- cpu/mem/io/net x Utilization/Saturation/Errors order, 1 verdict +-- line, and "analyze: done" as the last line of the app's own output. +-- 5. load attribution: a background `--resource cpu` saturator makes a +-- concurrent `analyze` see busy_pct and run_queue rise, Utilization +-- fire, and the verdict land on cpu -- retried up to 3 attempts +-- against a generous margin (calibrated from 3 real captures showing +-- busy_pct 1.34->100.00 and run_queue 3.00->7.00-8.00, both clearing +-- the gate by a large margin). +-- 6. wall-clock sanity: `analyze --seconds N` finishes in close to N +-- seconds, not 5N -- proving the five interval samplers +-- (vmstat/mpstat/iostat/sar/pidstat) run concurrently, not serially. +-- +-- Never compares formatted decimals across languages (C++, Go, and Rust +-- disagree on halfway rounding) -- only shapes (%d+%.%d%d) and numeric +-- ranges/ratios are asserted. +-- +-- Invoked from the example directory with LSP_LANG, TARGET, and REPO_ROOT +-- set (EXAMPLE_DIR optional, defaults to "."). + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local target = os.getenv("TARGET") +if not target or target == "" then + checks.skip("no TARGET set -- this is a vm example (run with --mode vm)") +end + +local example_dir = os.getenv("EXAMPLE_DIR") or "." +local home = os.getenv("HOME") or "/root" + +-- Local path of the freshly built binary for this language. +local bin_rel = ({ + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +})[lang] +local local_bin = example_dir .. "/" .. bin_rel + +-- Resolve the guest IP (same helper deploy-to-vm.sh uses). +local ipf = io.popen(repo_root .. "/scripts/lab/vm-ip.sh '" .. target .. "' 2>/dev/null") +local ip = ipf and ipf:read("*l") or nil +if ipf then ipf:close() end +if not ip or ip == "" then + checks.skip("cannot resolve IP for VM '" .. target .. "' (lab down?)") +end + +local ssh_opts = + "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=" .. home .. "/.ssh/known_hosts" + +-- Run a (single-quote-free) remote script through ssh. +local function ssh_run(remote_script) + return checks.run(string.format("ssh %s 'fedora@%s' '%s'", ssh_opts, ip, remote_script)) +end + +-- Boolean assertion via the checks.lua pattern-match API: match "ok" against +-- "ok" when true, or force a mismatch (with the real detail in the label) +-- when false -- same helper as example 39's verify.lua. +local function expect_true(cond, label, detail) + if cond then + checks.expect_match("ok", "ok", label) + else + local msg = label + if detail ~= nil then msg = msg .. " (" .. tostring(detail) .. ")" end + checks.expect_match("no", "yes", msg) + end +end + +local remote_dir = "/tmp/lsp37-" .. lang +local remote_bin = remote_dir .. "/app" + +-- --------------------------------------------------------------------------- +-- 1. build, then stage to /tmp/lsp37-/app -- named "app" so /proc comm +-- is "app", the book-wide rule. +-- --------------------------------------------------------------------------- + +local build = checks.run("./demo.sh " .. lang .. " build") +checks.expect_exit(build, 0, lang .. ": demo.sh build exits 0") + +local stage_cmd = string.format( + "ssh %s 'fedora@%s' 'mkdir -p %s' && " .. + "scp %s '%s' 'fedora@%s:%s' && " .. + "ssh %s 'fedora@%s' 'chmod 755 %s'", + ssh_opts, ip, remote_dir, + ssh_opts, local_bin, ip, remote_bin, + ssh_opts, ip, remote_bin) +local staged = checks.run(stage_cmd) +checks.expect_exit(staged, 0, lang .. ": stage app binary to " .. remote_dir .. " on the guest") + +-- The guest's real core count drives the worker-count formulas below. +local nproc_r = ssh_run("nproc") +local nproc = tonumber(nproc_r.out:match("%d+")) or 1 + +-- --------------------------------------------------------------------------- +-- 2. error shapes -- exit 2, usage on stderr. +-- --------------------------------------------------------------------------- + +local function expect_usage(args, label) + local r = ssh_run(remote_bin .. " " .. args) + checks.expect_exit(r, 2, lang .. ": " .. label .. " exits 2") + checks.expect_match(r.out, "usage: app %-%-resource", lang .. ": " .. label .. " prints usage") +end + +expect_usage("", "no args") +expect_usage("--resource bogus --seconds 5", "--resource bogus --seconds 5") +expect_usage("--resource cpu --seconds 0", "--resource cpu --seconds 0") +expect_usage("analyze --bogus", "analyze --bogus") + +-- --------------------------------------------------------------------------- +-- 3. saturate mode -- cpu/io/net (tight elapsed window) plus mem (generous +-- ceiling, no ops>0 requirement), per the golden-capture calibration. +-- --------------------------------------------------------------------------- + +local function workers_for(res, cpus) + if res == "mem" then return math.max(2, cpus) end + if res == "cpu" then return math.max(4, cpus * 3) end + return math.max(4, cpus * 4) -- io, net +end + +-- Run `--resource --seconds ` remotely, timed with date +%s%N +-- wrapped tightly around just the app invocation (not the ssh handshake). +local function saturate(res, secs) + local script = table.concat({ + "OUT=/tmp/lsp37_sat_" .. res .. ".$$", + "t0=$(date +%s%N)", + remote_bin .. " --resource " .. res .. " --seconds " .. secs .. " >$OUT 2>&1", + "RC=$?", + "t1=$(date +%s%N)", + "cat $OUT", + "rm -f $OUT", + "echo SAT_EXIT=$RC", + "echo SAT_ELAPSED_MS=$(( (t1 - t0) / 1000000 ))", + }, "\n") + return ssh_run(script) +end + +for _, res in ipairs({ "cpu", "io", "net" }) do + local secs = 4 + local r = saturate(res, secs) + + local sat_exit = tonumber(r.out:match("SAT_EXIT=(%d+)")) + expect_true(sat_exit == 0, lang .. ": --resource " .. res .. " --seconds " .. secs .. " exits 0", + tostring(sat_exit)) + + local workers = tonumber(r.out:match( + "loadmix: start resource=" .. res .. " seconds=" .. secs .. " workers=(%d+)")) + expect_true(workers ~= nil, lang .. ": " .. res .. " prints loadmix: start with workers=N") + + local want_workers = workers_for(res, nproc) + expect_true(workers == want_workers, + lang .. ": " .. res .. " workers matches the documented formula for nproc=" .. nproc, + string.format("got workers=%s want=%d", tostring(workers), want_workers)) + + local done_ops, done_bytes = r.out:match( + "loadmix: done resource=" .. res .. " seconds=" .. secs .. " workers=%d+ ops=(%d+) bytes=(%d+)") + done_ops, done_bytes = tonumber(done_ops), tonumber(done_bytes) + expect_true(done_ops ~= nil and done_bytes ~= nil, + lang .. ": " .. res .. " prints loadmix: done with ops= and bytes=") + expect_true(done_ops ~= nil and done_ops > 0, + lang .. ": " .. res .. " ops > 0", tostring(done_ops)) + + local elapsed_ms = tonumber(r.out:match("SAT_ELAPSED_MS=(%d+)")) + -- cpu/io/net track --seconds tightly on this guest (within ~0.2s per the + -- golden capture); a +/-2-4s window keeps this tight enough to have + -- caught the pre-fix bug (which exited in single-digit milliseconds) + -- while tolerating a slightly noisier guest. + expect_true(elapsed_ms ~= nil and elapsed_ms >= (secs * 1000 - 1500) and elapsed_ms <= (secs * 1000 + 4000), + lang .. ": " .. res .. " elapsed tracks --seconds=" .. secs .. + " (catches the pre-fix instant-exit saturator bug)", + tostring(elapsed_ms) .. "ms") +end + +do + local mem_secs = 3 + local r = saturate("mem", mem_secs) + + local sat_exit = tonumber(r.out:match("SAT_EXIT=(%d+)")) + expect_true(sat_exit == 0, + lang .. ": --resource mem --seconds " .. mem_secs .. " exits 0 (does not OOM the guest)", + tostring(sat_exit)) + + local workers = tonumber(r.out:match( + "loadmix: start resource=mem seconds=" .. mem_secs .. " workers=(%d+)")) + local want_workers = workers_for("mem", nproc) + expect_true(workers == want_workers, + lang .. ": mem workers matches the documented formula (max(2, nproc))", + string.format("got workers=%s want=%d", tostring(workers), want_workers)) + + local mem_ops, mem_bytes = r.out:match( + "loadmix: done resource=mem seconds=" .. mem_secs .. " workers=%d+ ops=(%d+) bytes=(%d+)") + mem_ops, mem_bytes = tonumber(mem_ops), tonumber(mem_bytes) + expect_true(mem_ops ~= nil and mem_bytes ~= nil, + lang .. ": mem prints loadmix: done with ops= and bytes=") + -- ops is deliberately NOT required to be > 0 here: the golden capture + -- found ops legitimately 0 at a short --seconds, since the initial buffer + -- allocation alone can consume the whole budget before any touch pass + -- completes. bytes (the ~1.35xMemTotal target) is the real sanity check. + expect_true(mem_bytes ~= nil and mem_bytes > 1024 * 1024 * 1024, + lang .. ": mem bytes= is the ~1.35xMemTotal target (>1GiB)", tostring(mem_bytes)) + + local elapsed_ms = tonumber(r.out:match("SAT_ELAPSED_MS=(%d+)")) + -- mem can OVERRUN --seconds by several seconds under real swap pressure + -- (a 12.6s actual for an 8s request was observed) -- a generous x5 + -- ceiling, not the tight cpu/io/net window. + expect_true(elapsed_ms ~= nil and elapsed_ms <= mem_secs * 5000, + lang .. ": mem finishes within a generous ceiling (secondsx5), tolerating the known swap overrun", + tostring(elapsed_ms) .. "ms") +end + +-- --------------------------------------------------------------------------- +-- 4 & 6. idle `analyze --seconds N`: exact line skeleton, plus wall-clock +-- sanity (samplers run concurrently, not serially). +-- --------------------------------------------------------------------------- + +local analyze_secs = 8 +local analyze_script = table.concat({ + "OUT=/tmp/lsp37_analyze.$$", + "t0=$(date +%s%N)", + remote_bin .. " analyze --seconds " .. analyze_secs .. " >$OUT 2>&1", + "RC=$?", + "t1=$(date +%s%N)", + "cat $OUT", + "rm -f $OUT", + "echo AN_EXIT=$RC", + "echo AN_ELAPSED_MS=$(( (t1 - t0) / 1000000 ))", +}, "\n") +local ar = ssh_run(analyze_script) + +local an_exit = tonumber(ar.out:match("AN_EXIT=(%d+)")) +expect_true(an_exit == 0, lang .. ": analyze --seconds " .. analyze_secs .. " exits 0", tostring(an_exit)) + +checks.expect_match(ar.out, "analyze: start seconds=" .. analyze_secs .. " cpus=%d+", + lang .. ": analyze prints the start line") +checks.expect_match(ar.out, "analyze: tool pidstat active_processes=%d+", + lang .. ": analyze prints the pidstat tool line") +checks.expect_match(ar.out, "analyze: tool dmesg status=%a+", + lang .. ": analyze prints the dmesg status line") + +for _, m in ipairs({ + "resource=cpu name=busy_pct value=%d+%.%d%d unit=pct", + "resource=cpu name=run_queue value=%d+%.%d%d unit=procs", + "resource=mem name=used_pct value=%d+%.%d%d unit=pct", + "resource=mem name=swap_io value=%d+%.%d%d unit=kbps", + "resource=io name=util_pct value=%d+%.%d%d unit=pct", + "resource=io name=iowait_pct value=%d+%.%d%d unit=pct", + "resource=io name=await_ms value=%d+%.%d%d unit=ms", + "resource=net name=pkts value=%d+%.%d%d unit=per_s", +}) do + checks.expect_match(ar.out, "analyze: metric " .. m, lang .. ": analyze metric " .. m) +end + +-- All 12 signal lines, in cpu/mem/io/net x Utilization/Saturation/Errors +-- order -- one pattern enforces both presence and order via lazy ".-" gaps. +local signal_order = {} +for _, res in ipairs({ "cpu", "mem", "io", "net" }) do + table.insert(signal_order, + "analyze: signal resource=" .. res .. " type=Utilization fired=%a+ value=%d+%.%d%d threshold=%d+%.%d%d") + table.insert(signal_order, + "analyze: signal resource=" .. res .. " type=Saturation fired=%a+ value=%d+%.%d%d threshold=%d+%.%d%d") + table.insert(signal_order, + "analyze: signal resource=" .. res .. " type=Errors fired=false") +end +checks.expect_match(ar.out, table.concat(signal_order, ".-"), + lang .. ": all 12 signal lines present, in cpu/mem/io/net x Utilization/Saturation/Errors order") + +checks.expect_match(ar.out, "analyze: verdict resource=%a+ ratio=%d+%.%d%d", + lang .. ": analyze prints the verdict line") +-- "analyze: done" is the last line of the app's OWN output (our injected +-- AN_EXIT=/AN_ELAPSED_MS= lines follow it in the captured buffer). +checks.expect_match(ar.out, "analyze: done\nAN_EXIT=%d+", + lang .. ": analyze: done is the last line of the app's output") + +local an_elapsed_ms = tonumber(ar.out:match("AN_ELAPSED_MS=(%d+)")) +expect_true(an_elapsed_ms ~= nil and an_elapsed_ms < analyze_secs * 5000, + lang .. ": analyze --seconds " .. analyze_secs .. " finishes well under 5x" .. analyze_secs .. + "s -- proves the 5 interval samplers run concurrently, not serially", + tostring(an_elapsed_ms) .. "ms") + +-- --------------------------------------------------------------------------- +-- 5. load attribution: a background --resource cpu saturator makes a +-- concurrent analyze attribute the load to cpu. Retried up to 3 attempts +-- (per the plan's own hedge, mirroring example 36's cpu-busy check) -- +-- the golden capture cleared this gate by a large margin every time +-- (busy_pct 100.00 vs. 60 threshold; run_queue 7-8 vs. nproc+2=4), so a +-- noisy guest should still pass, not produce a false red. +-- --------------------------------------------------------------------------- + +local function load_attribution_attempt() + local bg_secs, sleep_before, an_secs = 15, 2, 10 + local script = table.concat({ + "BGOUT=/tmp/lsp37_bg.$$", + "ANOUT=/tmp/lsp37_an.$$", + "nohup " .. remote_bin .. " --resource cpu --seconds " .. bg_secs .. " >$BGOUT 2>&1 &", + "BGPID=$!", + "sleep " .. sleep_before, + remote_bin .. " analyze --seconds " .. an_secs .. " >$ANOUT 2>&1", + "ANEXIT=$?", + "wait $BGPID", + "BGEXIT=$?", + "cat $ANOUT", + "echo AN_EXIT=$ANEXIT", + "cat $BGOUT", + "echo BG_EXIT=$BGEXIT", + "rm -f $BGOUT $ANOUT", + }, "\n") + return ssh_run(script) +end + +local attributed = false +local last_busy, last_rq, last_util_fired, last_verdict = nil, nil, nil, nil +for attempt = 1, 3 do + local r = load_attribution_attempt() + local an_exit2 = tonumber(r.out:match("AN_EXIT=(%d+)")) + local bg_exit = tonumber(r.out:match("BG_EXIT=(%d+)")) + local busy = tonumber(r.out:match("resource=cpu name=busy_pct value=(%d+%.%d%d)")) + local rq = tonumber(r.out:match("resource=cpu name=run_queue value=(%d+%.%d%d)")) + local util_fired = r.out:match("resource=cpu type=Utilization fired=(%a+)") + local verdict_res = r.out:match("verdict resource=(%a+)") + last_busy, last_rq, last_util_fired, last_verdict = busy, rq, util_fired, verdict_res + + print(string.format( + "%s: load-attribution attempt %d: an_exit=%s bg_exit=%s busy_pct=%s run_queue=%s util_fired=%s verdict=%s", + lang, attempt, tostring(an_exit2), tostring(bg_exit), tostring(busy), tostring(rq), + tostring(util_fired), tostring(verdict_res))) + + if an_exit2 == 0 and bg_exit == 0 and busy and rq + and (busy >= 60 or rq >= nproc + 2) + and util_fired == "true" and verdict_res == "cpu" then + attributed = true + break + end +end +expect_true(attributed, + lang .. ": under concurrent --resource cpu load, analyze attributes it " .. + "(busy_pct>=60 or run_queue>=nproc+2, Utilization fired=true, verdict=cpu) within 3 attempts", + string.format("last: busy_pct=%s run_queue=%s util_fired=%s verdict=%s", + tostring(last_busy), tostring(last_rq), tostring(last_util_fired), tostring(last_verdict))) + +-- --------------------------------------------------------------------------- +-- Cleanup. +-- --------------------------------------------------------------------------- + +checks.run(string.format("ssh %s 'fedora@%s' 'rm -rf %s'", ssh_opts, ip, remote_dir)) + +checks.finish() diff --git a/examples/38-opentelemetry-to-lgtm/README.md b/examples/38-opentelemetry-to-lgtm/README.md new file mode 100644 index 0000000..39b523c --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/README.md @@ -0,0 +1,51 @@ +# _template — hello-syscall + +The seed for every runnable example in the book. It implements the same tiny +program three times — C++23, Go, and Rust — each printing exactly one line: + +``` +pid on () +``` + +via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error +handling (`std::expected` / wrapped errors via `%w` / `rustix`). + +## Layout + +``` +_template/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build, demo.sh +├── go/ # go build, demo.sh +└── rust/ # cargo build, demo.sh +``` + +## The demo contract + +Every language directory has a `demo.sh` with the same interface, identical +across the whole book: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of + running locally. + +The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs +`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. + +## Verification + +`verify.lua` is run from this directory with `LSP_LANG` set to one language at +a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and +asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, +77 = skip. + +## Mapping to a chapter + +Each chapter's example is a copy of this template (`scripts/new-example.sh +NN-slug`) with the program body replaced by that chapter's code. Keep the +demo contract and `verify.lua` shape intact so CI and the manifest +(`examples/manifest.yaml`) keep working. diff --git a/examples/38-opentelemetry-to-lgtm/cpp/.clang-format b/examples/38-opentelemetry-to-lgtm/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/38-opentelemetry-to-lgtm/cpp/.clang-tidy b/examples/38-opentelemetry-to-lgtm/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/38-opentelemetry-to-lgtm/cpp/CMakeLists.txt b/examples/38-opentelemetry-to-lgtm/cpp/CMakeLists.txt new file mode 100644 index 0000000..0525a76 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/CMakeLists.txt @@ -0,0 +1,50 @@ +cmake_minimum_required(VERSION 3.25) +project(lsp_otel CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +# opentelemetry-cpp comes from Conan (see conanfile.txt): the CMakeDeps +# generator maps the package's cpp_info components onto +# opentelemetry-cpp:: imported targets. "trace" and "metrics" +# each carry their API *and* SDK pieces (there is no separate ::sdk +# target in this recipe); the exporters are named exporter_otlp_http and +# exporter_otlp_http_metric, not otlp_http_exporter. +find_package(opentelemetry-cpp CONFIG REQUIRED) +find_package(Threads REQUIRED) + +# The static archives conan pulls in have a genuine link-time cycle: e.g. +# libopentelemetry_exporter_otlp_http_client.a calls +# sdk::common::Base64Escape (defined in libopentelemetry_common.a), while +# conan's own generated CMake target graph makes nearly every component +# depend back on a shared aggregate target that includes the others -- +# a cycle at the CMake target-dependency level, not just the archive- +# symbol level, so plain target_link_libraries ordering (or even CMake's +# $, which refuses a group whose members depend +# back on the group itself) cannot fix it. GNU ld's own answer to +# circular static archives -- wrap the WHOLE library list in +# --start-group/--end-group so it keeps re-scanning until every symbol +# resolves -- has to be spliced directly into the link rule template, +# because CMake always emits the fully-resolved transitive library list +# () as one block *after* anything passed to +# target_link_libraries(), so no argument position in that call can +# bracket it. +set(CMAKE_CXX_LINK_EXECUTABLE + " -o -Wl,--start-group -Wl,--end-group" +) + +add_executable(app + src/main.cpp + src/telemetry.cpp +) + +target_link_libraries(app PRIVATE + opentelemetry-cpp::api + opentelemetry-cpp::trace + opentelemetry-cpp::metrics + opentelemetry-cpp::resources + opentelemetry-cpp::exporter_otlp_http + opentelemetry-cpp::exporter_otlp_http_metric + Threads::Threads +) diff --git a/examples/38-opentelemetry-to-lgtm/cpp/CMakePresets.json b/examples/38-opentelemetry-to-lgtm/cpp/CMakePresets.json new file mode 100644 index 0000000..bb52022 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/CMakePresets.json @@ -0,0 +1,63 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "toolchainFile": "${sourceDir}/build/release/conan_toolchain.cmake", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Release" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/38-opentelemetry-to-lgtm/cpp/conanfile.txt b/examples/38-opentelemetry-to-lgtm/cpp/conanfile.txt new file mode 100644 index 0000000..391f042 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/conanfile.txt @@ -0,0 +1,15 @@ +[requires] +opentelemetry-cpp/1.24.0 + +[options] +opentelemetry-cpp/*:shared=False +opentelemetry-cpp/*:with_otlp_http=True +opentelemetry-cpp/*:with_otlp_grpc=False +opentelemetry-cpp/*:with_zipkin=False +opentelemetry-cpp/*:with_prometheus=False +opentelemetry-cpp/*:with_elasticsearch=False +libcurl/*:with_ssl=False + +[generators] +CMakeDeps +CMakeToolchain diff --git a/examples/38-opentelemetry-to-lgtm/cpp/demo.sh b/examples/38-opentelemetry-to-lgtm/cpp/demo.sh new file mode 100755 index 0000000..70b341d --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/demo.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + conan install . --output-folder=build/release -s build_type=Release --build=missing + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp b/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp new file mode 100644 index 0000000..49005fa --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/main.cpp @@ -0,0 +1,481 @@ +// lsp-otel-cpp — a traced request/response loop (chapter 38: OpenTelemetry +// to LGTM). +// +// app serve --port PORT [--otel-endpoint URL] +// app drive --n N --addr HOST:PORT +// +// serve accepts one persistent line-oriented connection and treats every +// line as a "request": a parent span ("request") wraps two children +// ("work", "respond"), a monotonic counter (requests_total) is incremented, +// and a histogram (request_duration, milliseconds) records the parent +// span's wall time. drive is an untraced TCP client that sends N requests +// down one connection and waits for N replies -- the load generator for +// serve's telemetry, not a telemetry source itself. +// +// Every signal appears exactly once in the OTLP stream: serve's own +// SIGINT/SIGTERM handling flushes and shuts down the SDK before exit, the +// C++ way of RAII (an explicit flush_and_shutdown() call from the signal +// path, since a detached accept thread and a process-ending exit give no +// destructor a guaranteed chance to run -- the same reasoning the Rust port +// documents for its own explicit shutdown() call). +#include "telemetry.hpp" + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +namespace { + +namespace trace_api = opentelemetry::trace; + +[[noreturn]] void usage() { + std::println(stderr, "usage: app serve --port PORT [--otel-endpoint URL]"); + std::println(stderr, " app drive --n N --addr HOST:PORT"); + std::exit(2); +} + +[[noreturn]] void die(const std::string& msg) { + std::println(stderr, "app: error: {}", msg); + std::exit(1); +} + +std::string_view trim(std::string_view s) { + const auto is_space = [](unsigned char c) { return c == ' ' || c == '\t' || c == '\r' || c == '\n'; }; + while (!s.empty() && is_space(static_cast(s.front()))) { + s.remove_prefix(1); + } + while (!s.empty() && is_space(static_cast(s.back()))) { + s.remove_suffix(1); + } + return s; +} + +// write_all loops over write(2) so short writes (always possible on a +// socket) never truncate a reply -- the syscall-level equivalent of Go's +// bufio.Writer.Flush() / Rust's Write::write_all(). +bool write_all(int fd, std::string_view data) { + std::size_t off = 0; + while (off < data.size()) { + const ssize_t n = ::write(fd, data.data() + off, data.size() - off); + if (n < 0) { + if (errno == EINTR) { + continue; + } + return false; + } + off += static_cast(n); + } + return true; +} + +// LineReader is bufio.Scanner/BufReader::read_line's counterpart: it buffers +// raw read(2) chunks and hands back one newline-delimited line at a time, +// including a final unterminated line at EOF. +class LineReader { +public: + explicit LineReader(int fd) : fd_(fd) {} + + bool read_line(std::string& out) { + for (;;) { + if (const auto pos = buf_.find('\n'); pos != std::string::npos) { + out = buf_.substr(0, pos); + buf_.erase(0, pos + 1); + return true; + } + char chunk[4096]; + const ssize_t n = ::read(fd_, chunk, sizeof(chunk)); + if (n < 0) { + if (errno == EINTR) { + continue; + } + return false; + } + if (n == 0) { + if (!buf_.empty()) { + out = std::move(buf_); + buf_.clear(); + return true; + } + return false; + } + buf_.append(chunk, static_cast(n)); + } + } + +private: + int fd_; + std::string buf_; +}; + +// parse_i64 mirrors Go's strconv.Atoi / Rust's str::parse::(): the +// WHOLE trimmed string must be a valid integer, not merely a prefix of one. +bool parse_i64(std::string_view s, std::int64_t& out) { + if (s.empty()) { + return false; + } + const auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), out); + return ec == std::errc() && ptr == s.data() + s.size(); +} + +bool parse_port(std::string_view s, std::uint16_t& out) { + unsigned int v = 0; + const auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc() || ptr != s.data() + s.size() || v > 65535) { + return false; + } + out = static_cast(v); + return true; +} + +// ------------------------------------------------------------------- serve + +// handle_request is the parse->work->respond pipeline for one line of +// input. "parse" happens inline in serve_conn (cheap: one from_chars call); +// "work" and "respond" are the two child spans under the "request" parent, +// per the trace contract every language in this example follows +// identically. +bool handle_request(const telemetry::Handle& t, int fd, std::int64_t seq) { + const auto start = std::chrono::steady_clock::now(); + + auto parent_span = t.tracer->StartSpan("request", {{"request.seq", seq}}); + trace_api::Scope parent_scope(parent_span); + + // work: a tiny, deterministic-but-variable amount of simulated cost. + auto work_span = t.tracer->StartSpan("work", {{"request.seq", seq}}); + std::this_thread::sleep_for(std::chrono::milliseconds(1 + seq % 4)); + work_span->End(); + + // respond: write the reply line. + auto respond_span = t.tracer->StartSpan("respond", {{"request.seq", seq}}); + const std::string reply = "ok " + std::to_string(seq) + "\n"; + const bool wrote = write_all(fd, reply); + if (!wrote) { + respond_span->SetStatus(trace_api::StatusCode::kError, "write reply failed"); + } + respond_span->End(); + parent_span->End(); + + const double elapsed_ms = + std::chrono::duration(std::chrono::steady_clock::now() - start).count(); + t.requests_total->Add(1); + // Histogram::Record(T) alone is an ABI-v2-only overload; this build is + // pinned to ABI v1 (matching the rest of the conan cache), so an + // explicit empty Context is required to select an ABI-v1 overload. + t.request_duration->Record(elapsed_ms, opentelemetry::context::Context{}); + + return wrote; +} + +void serve_conn(const telemetry::Handle& t, int fd) { + LineReader reader(fd); + std::string line; + while (reader.read_line(line)) { + const std::string_view trimmed = trim(line); + if (trimmed.empty()) { + continue; + } + std::int64_t seq = 0; + if (!parse_i64(trimmed, seq)) { + if (!write_all(fd, "err bad-request\n")) { + break; + } + continue; + } + if (!handle_request(t, fd, seq)) { + std::println(stderr, "app: request error: write reply failed"); + break; + } + } + ::close(fd); +} + +// Event unifies the two sources serve waits on -- an incoming connection +// and a shutdown signal -- into the single queue a Go select over +// sigCh/connCh, or the Rust port's mpsc::channel over its Event enum, would +// use. +struct Event { + enum class Kind { Signal, Conn } kind; + std::string signal_name; + int conn_fd = -1; +}; + +// EventChannel is a minimal thread-safe queue: the signal-wait thread and +// the accept thread both push onto it, the main loop pops and dispatches. +class EventChannel { +public: + void push(Event e) { + { + std::lock_guard lock(mu_); + queue_.push_back(std::move(e)); + } + cv_.notify_one(); + } + + Event pop() { + std::unique_lock lock(mu_); + cv_.wait(lock, [this] { return !queue_.empty(); }); + Event e = std::move(queue_.front()); + queue_.pop_front(); + return e; + } + +private: + std::mutex mu_; + std::condition_variable cv_; + std::deque queue_; +}; + +int run_serve(const std::vector& args) { + std::string port; + std::string endpoint = "http://localhost:4318"; + if (const char* env = std::getenv("OTEL_ENDPOINT"); env != nullptr && *env != '\0') { + endpoint = env; + } + + for (std::size_t i = 0; i < args.size(); ++i) { + if (args[i] == "--port") { + if (++i >= args.size()) { + usage(); + } + port = args[i]; + } else if (args[i] == "--otel-endpoint") { + if (++i >= args.size()) { + usage(); + } + endpoint = args[i]; + } else { + usage(); + } + } + if (port.empty()) { + usage(); + } + + std::uint16_t port_num = 0; + if (!parse_port(port, port_num)) { + die("listen: invalid port '" + port + "'"); + } + + // A shell running "app serve ... &" as a background job (the exact + // pattern demo.sh/verify.lua use) sets SIGINT (and SIGQUIT) to SIG_IGN + // in the child before exec -- standard POSIX shell behavior, inherited + // across exec. Blocking + sigwait() alone does not override an + // inherited "ignore" disposition; installing a real handler does, which + // is exactly what Go's signal.Notify and Rust's signal-hook do under + // the hood. The handler itself never runs (the signal stays blocked + // below, so sigwait() -- not the handler -- is what actually consumes + // it); this call exists purely to force the disposition away from + // SIG_IGN. + struct sigaction sa {}; + sa.sa_handler = [](int) {}; + sigemptyset(&sa.sa_mask); + ::sigaction(SIGINT, &sa, nullptr); + ::sigaction(SIGTERM, &sa, nullptr); + + // Block SIGINT/SIGTERM in this thread BEFORE spawning (or triggering + // the SDK to spawn) any others: every thread created from here on -- + // including the BatchSpanProcessor/PeriodicExportingMetricReader + // background threads telemetry::init() starts -- inherits the blocked + // mask, so only the dedicated sigwait() thread below ever consumes + // them. Doing this BEFORE telemetry::init() matters: those SDK threads + // would otherwise start with the signal unblocked and the kernel could + // deliver SIGINT to one of them instead of the sigwait() thread. + sigset_t mask; + sigemptyset(&mask); + sigaddset(&mask, SIGINT); + sigaddset(&mask, SIGTERM); + if (const int rc = ::pthread_sigmask(SIG_BLOCK, &mask, nullptr); rc != 0) { + die("sigmask: " + std::string(std::strerror(rc))); + } + + telemetry::Handle t = telemetry::init(endpoint); + + const int listen_fd = ::socket(AF_INET, SOCK_STREAM, 0); + if (listen_fd < 0) { + die("listen: " + std::string(std::strerror(errno))); + } + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port_num); + if (::inet_pton(AF_INET, "127.0.0.1", &addr.sin_addr) != 1) { + die("listen: invalid address 127.0.0.1"); + } + if (::bind(listen_fd, reinterpret_cast(&addr), sizeof(addr)) < 0) { + die("listen: " + std::string(std::strerror(errno))); + } + if (::listen(listen_fd, 16) < 0) { + die("listen: " + std::string(std::strerror(errno))); + } + + EventChannel channel; + + // Signal watcher: sigwait() blocks (in this thread only) until one of + // the blocked signals is pending, then forwards it onto the shared + // event queue -- the same role Go's signal.Notify channel and the + // Rust port's signal-hook thread play. + std::thread signal_thread([&channel, mask] { + int sig = 0; + if (::sigwait(&mask, &sig) == 0) { + const char* name = sig == SIGINT ? "SIGINT" : sig == SIGTERM ? "SIGTERM" : "signal"; + channel.push(Event{Event::Kind::Signal, name, -1}); + } + }); + signal_thread.detach(); + + // Accept loop: forwards each accepted connection onto the same event + // queue. The process exits by returning all the way out of main, not + // by joining this thread -- same lifecycle as Go's accept goroutine + // (cut short by os.Exit) and the Rust port's accept thread (never + // joined). + std::thread accept_thread([&channel, listen_fd] { + for (;;) { + const int fd = ::accept(listen_fd, nullptr, nullptr); + if (fd < 0) { + return; // listener closed (or otherwise gone) during shutdown + } + channel.push(Event{Event::Kind::Conn, "", fd}); + } + }); + accept_thread.detach(); + + std::println("app: serve listening on 127.0.0.1:{}", port); + std::fflush(stdout); + + for (;;) { + Event e = channel.pop(); + if (e.kind == Event::Kind::Signal) { + std::println("app: shutting down ({})", e.signal_name); + std::fflush(stdout); + const bool ok = t.flush_and_shutdown(std::chrono::milliseconds(5000)); + if (!ok) { + std::println(stderr, "app: telemetry shutdown: one or more providers failed to flush/shutdown"); + } + return 0; + } + serve_conn(t, e.conn_fd); + } +} + +// ------------------------------------------------------------------- drive + +// run_drive is a plain, untraced TCP client: it generates the load that +// makes serve's telemetry interesting, but emits none of its own -- the +// service under observation is serve, not drive. +int run_drive(const std::vector& args) { + std::int64_t n = 0; + std::string addr_str = "127.0.0.1:8080"; + + for (std::size_t i = 0; i < args.size(); ++i) { + if (args[i] == "--n") { + if (++i >= args.size()) { + usage(); + } + if (!parse_i64(args[i], n)) { + die("--n: invalid integer '" + args[i] + "'"); + } + } else if (args[i] == "--addr") { + if (++i >= args.size()) { + usage(); + } + addr_str = args[i]; + } else { + usage(); + } + } + if (n <= 0) { + usage(); + } + + const auto colon = addr_str.rfind(':'); + if (colon == std::string::npos) { + die("dial " + addr_str + ": address must be HOST:PORT"); + } + const std::string host = addr_str.substr(0, colon); + std::uint16_t port_num = 0; + if (!parse_port(addr_str.substr(colon + 1), port_num)) { + die("dial " + addr_str + ": invalid port"); + } + + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + if (fd < 0) { + die("dial " + addr_str + ": " + std::strerror(errno)); + } + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port_num); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + die("dial " + addr_str + ": invalid host '" + host + "'"); + } + if (::connect(fd, reinterpret_cast(&addr), sizeof(addr)) < 0) { + const std::string msg = "dial " + addr_str + ": " + std::strerror(errno); + ::close(fd); + die(msg); + } + + LineReader reader(fd); + std::int64_t ok = 0; + for (std::int64_t seq = 1; seq <= n; ++seq) { + const std::string req = std::to_string(seq) + "\n"; + if (!write_all(fd, req)) { + const std::string msg = "send request " + std::to_string(seq) + ": " + std::strerror(errno); + ::close(fd); + die(msg); + } + std::string line; + if (!reader.read_line(line)) { + break; // EOF: server closed the connection early. + } + if (line.starts_with("ok ")) { + ++ok; + } + } + ::close(fd); + + std::println("app: drive sent {}/{} ok", ok, n); + std::fflush(stdout); + return ok == n ? 0 : 1; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc < 2) { + usage(); + } + const std::string_view subcommand = argv[1]; + const std::vector args(argv + 2, argv + argc); + + int code = 0; + if (subcommand == "serve") { + code = run_serve(args); + } else if (subcommand == "drive") { + code = run_drive(args); + } else { + usage(); + } + return code; +} diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp new file mode 100644 index 0000000..9aeada8 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.cpp @@ -0,0 +1,118 @@ +#include "telemetry.hpp" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace telemetry { + +namespace { + +namespace trace_api = opentelemetry::trace; +namespace trace_sdk = opentelemetry::sdk::trace; +namespace metrics_api = opentelemetry::metrics; +namespace metrics_sdk = opentelemetry::sdk::metrics; +namespace resource_sdk = opentelemetry::sdk::resource; +namespace otlp = opentelemetry::exporter::otlp; + +} // namespace + +bool Handle::flush_and_shutdown(std::chrono::milliseconds timeout) { + const auto us = std::chrono::duration_cast(timeout); + bool ok = true; + if (tracer_provider && !tracer_provider->ForceFlush(us)) { + ok = false; + } + if (tracer_provider && !tracer_provider->Shutdown(us)) { + ok = false; + } + if (meter_provider && !meter_provider->ForceFlush(us)) { + ok = false; + } + if (meter_provider && !meter_provider->Shutdown(us)) { + ok = false; + } + return ok; +} + +Handle init(const std::string& endpoint) { + // The SDK's default log handler writes Warning/Error to stderr and + // Info/Debug to stdout -- the latter would corrupt serve's stdout + // contract (exactly two lines, ever). A no-op handler keeps this build + // as quiet as the Go and Rust ports, which have no internal SDK logging + // to begin with. + opentelemetry::sdk::common::internal_log::GlobalLogHandler::SetLogHandler( + opentelemetry::nostd::shared_ptr( + new opentelemetry::sdk::common::internal_log::NoopLogHandler())); + opentelemetry::sdk::common::internal_log::GlobalLogHandler::SetLogLevel( + opentelemetry::sdk::common::internal_log::LogLevel::None); + + auto resource = resource_sdk::Resource::Create({ + {"service.name", service_name}, + {"service.version", "1.0"}, + {"deployment.environment", "local"}, + }); + + // --- traces: OTLP/HTTP exporter, batched every export_interval. --- + otlp::OtlpHttpExporterOptions trace_opts; + trace_opts.url = endpoint + "/v1/traces"; + auto trace_exporter = otlp::OtlpHttpExporterFactory::Create(trace_opts); + + trace_sdk::BatchSpanProcessorOptions batch_opts; + batch_opts.schedule_delay_millis = export_interval; + auto processor = trace_sdk::BatchSpanProcessorFactory::Create(std::move(trace_exporter), batch_opts); + + auto sampler = trace_sdk::AlwaysOnSamplerFactory::Create(); + // Keep the concrete sdk::trace::TracerProvider alive as a shared_ptr: + // Handle needs it (by its sdk type, for ForceFlush/Shutdown) and the + // global API registration needs it too (upcast to the API's + // trace::TracerProvider via nostd::shared_ptr's std::shared_ptr ctor). + std::shared_ptr tracer_provider( + trace_sdk::TracerProviderFactory::Create(std::move(processor), resource, std::move(sampler))); + trace_api::Provider::SetTracerProvider( + opentelemetry::nostd::shared_ptr(tracer_provider)); + + // --- metrics: OTLP/HTTP exporter, exported every export_interval. --- + otlp::OtlpHttpMetricExporterOptions metric_opts; + metric_opts.url = endpoint + "/v1/metrics"; + auto metric_exporter = otlp::OtlpHttpMetricExporterFactory::Create(metric_opts); + + metrics_sdk::PeriodicExportingMetricReaderOptions reader_opts; + reader_opts.export_interval_millis = export_interval; + reader_opts.export_timeout_millis = export_interval; + auto reader = + std::make_shared(std::move(metric_exporter), reader_opts); + + std::shared_ptr meter_provider( + metrics_sdk::MeterProviderFactory::Create(std::make_unique(), resource)); + meter_provider->AddMetricReader(reader); + metrics_api::Provider::SetMeterProvider( + opentelemetry::nostd::shared_ptr(meter_provider)); + + Handle h; + h.tracer_provider = tracer_provider; + h.meter_provider = meter_provider; + h.tracer = trace_api::Provider::GetTracerProvider()->GetTracer(service_name); + auto meter = metrics_api::Provider::GetMeterProvider()->GetMeter(service_name); + h.requests_total = meter->CreateUInt64Counter("requests_total", "Total requests handled", "1"); + h.request_duration = meter->CreateDoubleHistogram("request_duration", "Request duration", "ms"); + return h; +} + +} // namespace telemetry diff --git a/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp new file mode 100644 index 0000000..ba088fe --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/cpp/src/telemetry.hpp @@ -0,0 +1,67 @@ +// telemetry.hpp — OpenTelemetry wiring for lsp-otel-cpp (chapter 38: +// OpenTelemetry to LGTM). Mirrors the Go reference's initTelemetry: a +// TracerProvider batching spans over OTLP/HTTP, a MeterProvider exporting a +// counter and a histogram on the same short interval, and one Handle that +// owns both so serve's signal path can flush-then-shutdown them explicitly +// -- the C++ RAII equivalent of the Go program's deferred cleanup func, +// called out by hand rather than left to a destructor because "flush before +// exit" has a deadline the process must observe (see flush_and_shutdown). +#pragma once + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +namespace telemetry { + +// service_name tags every span and metric this binary emits; it MUST differ +// from the Go (lsp-otel-go) and Rust (lsp-otel-rust) builds so verify.lua can +// query the LGTM stack for this language's signals alone. +inline constexpr const char* service_name = "lsp-otel-cpp"; + +// export_interval is short on purpose: the demo and verify.lua both need the +// last batch flushed within a few seconds of shutdown, not the SDK defaults +// (5s span batches are fine; 60s metric batches are not). Matches the Go +// reference's exportInterval exactly so both languages' telemetry lands on +// comparable timescales. +inline constexpr std::chrono::milliseconds export_interval{2000}; + +// Handle owns the trace/metric providers plus the two instruments serve +// records against; it is move-only (Counter/Histogram are non-copyable +// nostd::unique_ptr) and deliberately has no destructor-driven shutdown -- +// flush_and_shutdown() must be called explicitly from the signal path so the +// caller controls the timeout instead of an unobservable process-exit race. +struct Handle { + std::shared_ptr tracer_provider; + std::shared_ptr meter_provider; + opentelemetry::nostd::shared_ptr tracer; + opentelemetry::nostd::unique_ptr> requests_total; + opentelemetry::nostd::unique_ptr> request_duration; + + // flush_and_shutdown force-flushes both providers (draining any batch + // still in flight) and then shuts them down for good, each bounded by + // `timeout`. Returns false if either step reports a failure; the caller + // logs but does not treat that as fatal (mirrors the Go reference, which + // logs a shutdown error to stderr and still exits 0). + bool flush_and_shutdown(std::chrono::milliseconds timeout); +}; + +// init sets up the resource (service.name/version/deployment.environment), +// the OTLP/HTTP trace and metric exporters against `endpoint` (a full URL, +// e.g. "http://localhost:4318" -- the /v1/traces and /v1/metrics paths are +// appended here), and the requests_total counter + request_duration +// histogram instruments. Aborts the process via die() semantics is the +// caller's job; init itself only throws opentelemetry-cpp's own exceptions +// (none, in practice -- the factories are noexcept-safe by construction). +Handle init(const std::string& endpoint); + +} // namespace telemetry diff --git a/examples/38-opentelemetry-to-lgtm/demo.sh b/examples/38-opentelemetry-to-lgtm/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/38-opentelemetry-to-lgtm/go/demo.sh b/examples/38-opentelemetry-to-lgtm/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/38-opentelemetry-to-lgtm/go/go.mod b/examples/38-opentelemetry-to-lgtm/go/go.mod new file mode 100644 index 0000000..d98f92c --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/go/go.mod @@ -0,0 +1,33 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/38-opentelemetry-to-lgtm/go + +go 1.26 + +toolchain go1.26.5 + +require ( + go.opentelemetry.io/otel v1.38.0 + go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.38.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.38.0 + go.opentelemetry.io/otel/metric v1.38.0 + go.opentelemetry.io/otel/sdk v1.38.0 + go.opentelemetry.io/otel/sdk/metric v1.38.0 + go.opentelemetry.io/otel/trace v1.38.0 +) + +require ( + github.com/cenkalti/backoff/v5 v5.0.3 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 // indirect + go.opentelemetry.io/auto/sdk v1.1.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0 // indirect + go.opentelemetry.io/proto/otlp v1.7.1 // indirect + golang.org/x/net v0.43.0 // indirect + golang.org/x/sys v0.35.0 // indirect + golang.org/x/text v0.28.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20250825161204-c5933d9347a5 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20250825161204-c5933d9347a5 // indirect + google.golang.org/grpc v1.75.0 // indirect + google.golang.org/protobuf v1.36.8 // indirect +) diff --git a/examples/38-opentelemetry-to-lgtm/go/go.sum b/examples/38-opentelemetry-to-lgtm/go/go.sum new file mode 100644 index 0000000..50e5a44 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/go/go.sum @@ -0,0 +1,61 @@ +github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM= +github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2 h1:8Tjv8EJ+pM1xP8mK6egEbD1OgnVTyacbefKhmbLhIhU= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.2/go.mod h1:pkJQ2tZHJ0aFOVEEot6oZmaVEZcRme73eIFmhiVuRWs= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= +go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= +go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8= +go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.38.0 h1:Oe2z/BCg5q7k4iXC3cqJxKYg0ieRiOqF0cecFYdPTwk= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.38.0/go.mod h1:ZQM5lAJpOsKnYagGg/zV2krVqTtaVdYdDkhMoX6Oalg= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0 h1:GqRJVj7UmLjCVyVJ3ZFLdPRmhDUp2zFmQe3RHIOsw24= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.38.0/go.mod h1:ri3aaHSmCTVYu2AWv44YMauwAQc0aqI9gHKIcSbI1pU= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.38.0 h1:aTL7F04bJHUlztTsNGJ2l+6he8c+y/b//eR0jjjemT4= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.38.0/go.mod h1:kldtb7jDTeol0l3ewcmd8SDvx3EmIE7lyvqbasU3QC4= +go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA= +go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI= +go.opentelemetry.io/otel/sdk v1.38.0 h1:l48sr5YbNf2hpCUj/FoGhW9yDkl+Ma+LrVl8qaM5b+E= +go.opentelemetry.io/otel/sdk v1.38.0/go.mod h1:ghmNdGlVemJI3+ZB5iDEuk4bWA3GkTpW+DOoZMYBVVg= +go.opentelemetry.io/otel/sdk/metric v1.38.0 h1:aSH66iL0aZqo//xXzQLYozmWrXxyFkBJ6qT5wthqPoM= +go.opentelemetry.io/otel/sdk/metric v1.38.0/go.mod h1:dg9PBnW9XdQ1Hd6ZnRz689CbtrUp0wMMs9iPcgT9EZA= +go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE= +go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs= +go.opentelemetry.io/proto/otlp v1.7.1 h1:gTOMpGDb0WTBOP8JaO72iL3auEZhVmAQg4ipjOVAtj4= +go.opentelemetry.io/proto/otlp v1.7.1/go.mod h1:b2rVh6rfI/s2pHWNlB7ILJcRALpcNDzKhACevjI+ZnE= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE= +golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg= +golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= +golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng= +golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU= +gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk= +gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E= +google.golang.org/genproto/googleapis/api v0.0.0-20250825161204-c5933d9347a5 h1:BIRfGDEjiHRrk0QKZe3Xv2ieMhtgRGeLcZQ0mIVn4EY= +google.golang.org/genproto/googleapis/api v0.0.0-20250825161204-c5933d9347a5/go.mod h1:j3QtIyytwqGr1JUDtYXwtMXWPKsEa5LtzIFN1Wn5WvE= +google.golang.org/genproto/googleapis/rpc v0.0.0-20250825161204-c5933d9347a5 h1:eaY8u2EuxbRv7c3NiGK0/NedzVsCcV6hDuU5qPX5EGE= +google.golang.org/genproto/googleapis/rpc v0.0.0-20250825161204-c5933d9347a5/go.mod h1:M4/wBTSeyLxupu3W3tJtOgB14jILAS/XWPSSa3TAlJc= +google.golang.org/grpc v1.75.0 h1:+TW+dqTd2Biwe6KKfhE5JpiYIBWq865PhKGSXiivqt4= +google.golang.org/grpc v1.75.0/go.mod h1:JtPAzKiq4v1xcAB2hydNlWI2RnF85XXcV0mhKXr2ecQ= +google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc= +google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/examples/38-opentelemetry-to-lgtm/go/main.go b/examples/38-opentelemetry-to-lgtm/go/main.go new file mode 100644 index 0000000..54096ca --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/go/main.go @@ -0,0 +1,374 @@ +// lsp-otel — a traced request/response loop (chapter 38: OpenTelemetry to LGTM). +// +// app serve --port PORT [--otel-endpoint URL] +// app drive --n N --addr HOST:PORT +// +// serve accepts one persistent line-oriented connection and treats every +// line as a "request": a parent span ("request") wraps two children +// ("work", "respond"), a monotonic counter (requests_total) is incremented, +// and a histogram (request_duration, milliseconds) records the parent +// span's wall time. drive is an untraced TCP client that sends N requests +// down one connection and waits for N replies — the load generator for +// serve's telemetry, not a telemetry source itself. +// +// Every signal appears exactly once in the OTLP stream: serve's own +// SIGINT/SIGTERM handling flushes and shuts down the SDK before exit, the +// Go way of RAII (deferred cleanup, no destructors to rely on). +package main + +import ( + "bufio" + "context" + "errors" + "fmt" + "io" + "net" + "os" + "os/signal" + "strconv" + "strings" + "syscall" + "time" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + semconv "go.opentelemetry.io/otel/semconv/v1.26.0" + "go.opentelemetry.io/otel/trace" +) + +const serviceName = "lsp-otel-go" + +// exportInterval is short on purpose: the demo and verify.lua both need the +// last batch flushed within a few seconds of shutdown, not the SDK +// defaults (5s span batches are fine; 60s metric batches are not). +const exportInterval = 2 * time.Second + +func usage() { + fmt.Fprintln(os.Stderr, "usage: app serve --port PORT [--otel-endpoint URL]") + fmt.Fprintln(os.Stderr, " app drive --n N --addr HOST:PORT") + os.Exit(2) +} + +func die(err error) { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + os.Exit(1) +} + +// ---------------------------------------------------------------- telemetry + +type telemetry struct { + tracerProvider *sdktrace.TracerProvider + meterProvider *sdkmetric.MeterProvider + tracer trace.Tracer + requestsTotal metric.Int64Counter + requestDur metric.Float64Histogram +} + +// otlpHostPort strips a URL scheme: the otlptrace/otlpmetric HTTP exporters +// take host:port, not a URL (they prepend the protocol themselves). +func otlpHostPort(endpoint string) string { + hp := strings.TrimPrefix(endpoint, "http://") + hp = strings.TrimPrefix(hp, "https://") + return hp +} + +func initTelemetry(ctx context.Context, endpoint string) (*telemetry, func(context.Context) error, error) { + res, err := resource.New(ctx, + resource.WithAttributes( + semconv.ServiceName(serviceName), + semconv.ServiceVersion("1.0"), + semconv.DeploymentEnvironment("local"), + ), + ) + if err != nil { + return nil, nil, fmt.Errorf("resource: %w", err) + } + + hostPort := otlpHostPort(endpoint) + + traceExp, err := otlptracehttp.New(ctx, + otlptracehttp.WithEndpoint(hostPort), + otlptracehttp.WithInsecure(), + ) + if err != nil { + return nil, nil, fmt.Errorf("trace exporter: %w", err) + } + tp := sdktrace.NewTracerProvider( + sdktrace.WithBatcher(traceExp, sdktrace.WithBatchTimeout(exportInterval)), + sdktrace.WithResource(res), + sdktrace.WithSampler(sdktrace.AlwaysSample()), + ) + otel.SetTracerProvider(tp) + + metricExp, err := otlpmetrichttp.New(ctx, + otlpmetrichttp.WithEndpoint(hostPort), + otlpmetrichttp.WithInsecure(), + ) + if err != nil { + return nil, nil, fmt.Errorf("metric exporter: %w", err) + } + mp := sdkmetric.NewMeterProvider( + sdkmetric.WithResource(res), + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp, + sdkmetric.WithInterval(exportInterval))), + ) + otel.SetMeterProvider(mp) + + meter := mp.Meter(serviceName) + requestsTotal, err := meter.Int64Counter("requests_total", + metric.WithDescription("Total requests handled"), + metric.WithUnit("1")) + if err != nil { + return nil, nil, fmt.Errorf("counter: %w", err) + } + requestDur, err := meter.Float64Histogram("request_duration", + metric.WithDescription("Request duration"), + metric.WithUnit("ms")) + if err != nil { + return nil, nil, fmt.Errorf("histogram: %w", err) + } + + t := &telemetry{ + tracerProvider: tp, + meterProvider: mp, + tracer: tp.Tracer(serviceName), + requestsTotal: requestsTotal, + requestDur: requestDur, + } + + shutdown := func(ctx context.Context) error { + var errs []error + if err := tp.Shutdown(ctx); err != nil { + errs = append(errs, err) + } + if err := mp.Shutdown(ctx); err != nil { + errs = append(errs, err) + } + return errors.Join(errs...) + } + return t, shutdown, nil +} + +// ------------------------------------------------------------------- serve + +// handleRequest is the parse->work->respond pipeline for one line of input. +// "parse" happens inline (cheap: one strconv.Atoi); "work" and "respond" +// are the two child spans under the "request" parent, per the trace +// contract every language in this example follows identically. +func handleRequest(ctx context.Context, t *telemetry, w *bufio.Writer, seq int) error { + start := time.Now() + ctx, parent := t.tracer.Start(ctx, "request", + trace.WithAttributes(attribute.Int("request.seq", seq))) + defer parent.End() + + // work: a tiny, deterministic-but-variable amount of simulated cost. + _, work := t.tracer.Start(ctx, "work", + trace.WithAttributes(attribute.Int("request.seq", seq))) + time.Sleep(time.Duration(1+seq%4) * time.Millisecond) + work.End() + + // respond: write the reply line. + _, respond := t.tracer.Start(ctx, "respond", + trace.WithAttributes(attribute.Int("request.seq", seq))) + if _, err := fmt.Fprintf(w, "ok %d\n", seq); err != nil { + respond.RecordError(err) + respond.End() + return fmt.Errorf("write reply: %w", err) + } + if err := w.Flush(); err != nil { + respond.RecordError(err) + respond.End() + return fmt.Errorf("flush reply: %w", err) + } + respond.End() + + elapsedMs := float64(time.Since(start)) / float64(time.Millisecond) + t.requestsTotal.Add(ctx, 1) + t.requestDur.Record(ctx, elapsedMs) + return nil +} + +func serveConn(t *telemetry, conn net.Conn) { + defer conn.Close() + r := bufio.NewScanner(conn) + w := bufio.NewWriter(conn) + for r.Scan() { + line := strings.TrimSpace(r.Text()) + if line == "" { + continue + } + seq, err := strconv.Atoi(line) + if err != nil { + fmt.Fprintf(w, "err bad-request\n") + w.Flush() + continue + } + if err := handleRequest(context.Background(), t, w, seq); err != nil { + fmt.Fprintln(os.Stderr, "app: request error:", err) + return + } + } +} + +func runServe(args []string) int { + port := "" + endpoint := os.Getenv("OTEL_ENDPOINT") + if endpoint == "" { + endpoint = "http://localhost:4318" + } + for i := 0; i < len(args); i++ { + switch args[i] { + case "--port": + i++ + if i >= len(args) { + usage() + } + port = args[i] + case "--otel-endpoint": + i++ + if i >= len(args) { + usage() + } + endpoint = args[i] + default: + usage() + } + } + if port == "" { + usage() + } + + ctx := context.Background() + t, shutdown, err := initTelemetry(ctx, endpoint) + if err != nil { + die(fmt.Errorf("telemetry init: %w", err)) + } + + ln, err := net.Listen("tcp", "127.0.0.1:"+port) + if err != nil { + die(fmt.Errorf("listen: %w", err)) + } + + sigCh := make(chan os.Signal, 1) + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) + + connCh := make(chan net.Conn) + go func() { + for { + conn, err := ln.Accept() + if err != nil { + return // listener closed during shutdown + } + connCh <- conn + } + }() + + fmt.Printf("app: serve listening on 127.0.0.1:%s\n", port) + + for { + select { + case sig := <-sigCh: + fmt.Printf("app: shutting down (%s)\n", sig) + ln.Close() + shutCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + if err := shutdown(shutCtx); err != nil { + fmt.Fprintln(os.Stderr, "app: telemetry shutdown:", err) + } + cancel() + return 0 + case conn := <-connCh: + serveConn(t, conn) + } + } +} + +// ------------------------------------------------------------------- drive + +// runDrive is a plain, untraced TCP client: it generates the load that +// makes serve's telemetry interesting, but emits none of its own — the +// service under observation is serve, not drive. +func runDrive(args []string) int { + n := 0 + addr := "127.0.0.1:8080" + for i := 0; i < len(args); i++ { + switch args[i] { + case "--n": + i++ + if i >= len(args) { + usage() + } + v, err := strconv.Atoi(args[i]) + if err != nil { + die(fmt.Errorf("--n: %w", err)) + } + n = v + case "--addr": + i++ + if i >= len(args) { + usage() + } + addr = args[i] + default: + usage() + } + } + if n <= 0 { + usage() + } + + conn, err := net.Dial("tcp", addr) + if err != nil { + die(fmt.Errorf("dial %s: %w", addr, err)) + } + defer conn.Close() + + w := bufio.NewWriter(conn) + r := bufio.NewScanner(conn) + ok := 0 + for seq := 1; seq <= n; seq++ { + if _, err := fmt.Fprintf(w, "%d\n", seq); err != nil { + die(fmt.Errorf("send request %d: %w", seq, err)) + } + if err := w.Flush(); err != nil { + die(fmt.Errorf("flush request %d: %w", seq, err)) + } + if !r.Scan() { + if err := r.Err(); err != nil && !errors.Is(err, io.EOF) { + die(fmt.Errorf("read reply %d: %w", seq, err)) + } + break + } + if strings.HasPrefix(r.Text(), "ok ") { + ok++ + } + } + fmt.Printf("app: drive sent %d/%d ok\n", ok, n) + if ok != n { + return 1 + } + return 0 +} + +// ---------------------------------------------------------------------main + +func main() { + if len(os.Args) < 2 { + usage() + } + var code int + switch os.Args[1] { + case "serve": + code = runServe(os.Args[2:]) + case "drive": + code = runDrive(os.Args[2:]) + default: + usage() + } + os.Exit(code) +} diff --git a/examples/38-opentelemetry-to-lgtm/rust/Cargo.lock b/examples/38-opentelemetry-to-lgtm/rust/Cargo.lock new file mode 100644 index 0000000..1f20458 --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/rust/Cargo.lock @@ -0,0 +1,1133 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "async-trait" +version = "0.1.91" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae36dc4177970ef04fde5178d3e2429882def40e57a451f919c098f72baa6cec" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "atomic-waker" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "displaydoc" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi", + "wasip2", +] + +[[package]] +name = "http" +version = "1.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "http-body" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" +dependencies = [ + "bytes", + "http", +] + +[[package]] +name = "http-body-util" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2" +dependencies = [ + "bytes", + "futures-core", + "http", + "http-body", + "pin-project-lite", +] + +[[package]] +name = "httparse" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" + +[[package]] +name = "hyper" +version = "1.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" +dependencies = [ + "atomic-waker", + "bytes", + "futures-channel", + "futures-core", + "http", + "http-body", + "httparse", + "itoa", + "pin-project-lite", + "smallvec", + "tokio", + "want", +] + +[[package]] +name = "hyper-util" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-util", + "http", + "http-body", + "hyper", + "ipnet", + "libc", + "percent-encoding", + "pin-project-lite", + "socket2", + "tokio", + "tower-service", + "tracing", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "ipnet" +version = "2.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "lsp-otel-rust" +version = "0.1.0" +dependencies = [ + "anyhow", + "opentelemetry", + "opentelemetry-otlp", + "opentelemetry_sdk", + "signal-hook", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "mio" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" +dependencies = [ + "libc", + "wasi", + "windows-sys", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opentelemetry" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b0142c63252a9e054e68a4c61a5778f7b14f576274d593f8ce883d191a099682" +dependencies = [ + "futures-core", + "futures-sink", + "js-sys", + "pin-project-lite", + "thiserror", + "tracing", +] + +[[package]] +name = "opentelemetry-http" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5683015d09e2df236ef005b17f6f196f0d5f6313c4fa43a7b6a53b52776e4331" +dependencies = [ + "async-trait", + "bytes", + "http", + "opentelemetry", + "reqwest", +] + +[[package]] +name = "opentelemetry-otlp" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9966929966d17620d7c316c643ba62631826e10021409357772d5eea84f62c35" +dependencies = [ + "http", + "opentelemetry", + "opentelemetry-http", + "opentelemetry-proto", + "opentelemetry_sdk", + "prost", + "reqwest", + "thiserror", +] + +[[package]] +name = "opentelemetry-proto" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "56d658ba1faf63f7b9c492cfbe6e0ec365440a16132d3270c1065f7b33f1b638" +dependencies = [ + "opentelemetry", + "opentelemetry_sdk", + "prost", +] + +[[package]] +name = "opentelemetry_sdk" +version = "0.32.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b59f80e1ac4d5ff7a2db8fb6c80badb7f0f3f858211fba08dd9aaec750894f9" +dependencies = [ + "futures-channel", + "futures-executor", + "futures-util", + "opentelemetry", + "percent-encoding", + "portable-atomic", + "rand", + "thiserror", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "prost" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" +dependencies = [ + "bytes", + "prost-derive", +] + +[[package]] +name = "prost-derive" +version = "0.14.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" +dependencies = [ + "anyhow", + "itertools", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom", +] + +[[package]] +name = "reqwest" +version = "0.13.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3" +dependencies = [ + "base64", + "bytes", + "futures-channel", + "futures-core", + "futures-util", + "http", + "http-body", + "http-body-util", + "hyper", + "hyper-util", + "js-sys", + "log", + "percent-encoding", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower", + "tower-http", + "tower-service", + "url", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "signal-hook" +version = "0.3.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2" +dependencies = [ + "libc", + "signal-hook-registry", +] + +[[package]] +name = "signal-hook-registry" +version = "1.4.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" +dependencies = [ + "errno", + "libc", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "socket2" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2fac314a64dc9a36e61a9eb4261a5e9bbfbc922b27e518af97bc32b926cf967" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "sync_wrapper" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" +dependencies = [ + "futures-core", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.0", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d988bcd52dbe076d3d46903332f58c912b87a2c49b1428419a5845154762ffee" +dependencies = [ + "libc", + "mio", + "pin-project-lite", + "socket2", + "windows-sys", +] + +[[package]] +name = "tower" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" +dependencies = [ + "futures-core", + "futures-util", + "pin-project-lite", + "sync_wrapper", + "tokio", + "tower-layer", + "tower-service", +] + +[[package]] +name = "tower-http" +version = "0.6.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" +dependencies = [ + "bitflags", + "bytes", + "futures-util", + "http", + "http-body", + "pin-project-lite", + "tower", + "tower-layer", + "tower-service", + "url", +] + +[[package]] +name = "tower-layer" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" + +[[package]] +name = "tower-service" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" + +[[package]] +name = "tracing" +version = "0.1.44" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" +dependencies = [ + "pin-project-lite", + "tracing-core", +] + +[[package]] +name = "tracing-core" +version = "0.1.36" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" +dependencies = [ + "once_cell", +] + +[[package]] +name = "try-lock" +version = "0.2.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "want" +version = "0.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" +dependencies = [ + "try-lock", +] + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-futures" +version = "0.4.76" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerocopy" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.54" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] diff --git a/examples/38-opentelemetry-to-lgtm/rust/Cargo.toml b/examples/38-opentelemetry-to-lgtm/rust/Cargo.toml new file mode 100644 index 0000000..e1ce92d --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/rust/Cargo.toml @@ -0,0 +1,20 @@ +[package] +name = "lsp-otel-rust" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +anyhow = "1" +opentelemetry = "0.32" +opentelemetry_sdk = "0.32" +opentelemetry-otlp = { version = "0.32", default-features = false, features = [ + "http-proto", + "reqwest-blocking-client", + "trace", + "metrics", +] } +signal-hook = "0.3" diff --git a/examples/38-opentelemetry-to-lgtm/rust/demo.sh b/examples/38-opentelemetry-to-lgtm/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/38-opentelemetry-to-lgtm/rust/rust-toolchain.toml b/examples/38-opentelemetry-to-lgtm/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/38-opentelemetry-to-lgtm/rust/src/main.rs b/examples/38-opentelemetry-to-lgtm/rust/src/main.rs new file mode 100644 index 0000000..66e064f --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/rust/src/main.rs @@ -0,0 +1,431 @@ +// lsp-otel-rust — a traced request/response loop (chapter 38: OpenTelemetry +// to LGTM). +// +// app serve --port PORT [--otel-endpoint URL] +// app drive --n N --addr HOST:PORT +// +// serve accepts one persistent line-oriented connection and treats every +// line as a "request": a parent span ("request") wraps two children +// ("work", "respond"), a monotonic counter (requests_total) is incremented, +// and a histogram (request_duration, milliseconds) records the parent +// span's wall time. drive is an untraced TCP client that sends N requests +// down one connection and waits for N replies — the load generator for +// serve's telemetry, not a telemetry source itself. +// +// Every signal appears exactly once in the OTLP stream: serve's own +// SIGINT/SIGTERM handling flushes and shuts down the SDK before exit, the +// Rust way of RAII (explicit shutdown calls, since there is no deferred +// cleanup guaranteed to run before `std::process::exit`). +use std::io::{BufRead, BufReader, Write}; +use std::net::{TcpListener, TcpStream}; +use std::sync::mpsc; +use std::thread; +use std::time::{Duration, Instant}; + +use anyhow::{Context as _, Result}; +use opentelemetry::metrics::{Counter, Histogram, MeterProvider as _}; +use opentelemetry::trace::{Span as _, TraceContextExt, Tracer as _, TracerProvider as _}; +use opentelemetry::{Context, KeyValue}; +use opentelemetry_otlp::WithExportConfig; +use opentelemetry_sdk::Resource; +use opentelemetry_sdk::metrics::{PeriodicReader, SdkMeterProvider}; +use opentelemetry_sdk::trace::{ + BatchConfigBuilder, BatchSpanProcessor, Sampler, SdkTracer, SdkTracerProvider, +}; +use signal_hook::consts::{SIGINT, SIGTERM}; +use signal_hook::iterator::Signals; + +const SERVICE_NAME: &str = "lsp-otel-rust"; + +// EXPORT_INTERVAL is short on purpose: the demo and verify.lua both need the +// last batch flushed within a few seconds of shutdown, not the SDK +// defaults (5s span batches are fine; 60s metric batches are not). +const EXPORT_INTERVAL: Duration = Duration::from_secs(2); + +fn usage() -> ! { + eprintln!("usage: app serve --port PORT [--otel-endpoint URL]"); + eprintln!(" app drive --n N --addr HOST:PORT"); + std::process::exit(2); +} + +fn die(msg: &str) -> ! { + eprintln!("app: error: {msg}"); + std::process::exit(1); +} + +// ---------------------------------------------------------------- telemetry + +struct Telemetry { + tracer_provider: SdkTracerProvider, + meter_provider: SdkMeterProvider, + tracer: SdkTracer, + requests_total: Counter, + request_duration: Histogram, +} + +impl Telemetry { + // shutdown flushes and closes both providers. Errors are reported but + // never fatal — the process is exiting regardless, this is best effort + // draining of whatever the batch/periodic exporters still hold. + fn shutdown(&self) { + if let Err(e) = self.tracer_provider.shutdown() { + eprintln!("app: telemetry shutdown: {e}"); + } + if let Err(e) = self.meter_provider.shutdown() { + eprintln!("app: telemetry shutdown: {e}"); + } + } +} + +// init_telemetry wires up the OTLP HTTP/protobuf exporters (reqwest blocking +// client — no tokio runtime needed), a short-interval batch span processor +// and periodic metric reader, and the two instruments serve records against. +// The per-signal OTLP HTTP exporters POST to exactly the URL given -- unlike +// the base-endpoint env-var path, an explicit with_endpoint() does NOT append +// the signal path in opentelemetry-otlp 0.32 -- so the caller's base +// (http://localhost:4318) is joined to /v1/traces and /v1/metrics here. +// Posting to the bare base instead 404s at the collector and the export +// silently fails (surfacing only as a shutdown error). +fn init_telemetry(endpoint: &str) -> Result { + let base = endpoint.trim_end_matches('/'); + let traces_url = format!("{base}/v1/traces"); + let metrics_url = format!("{base}/v1/metrics"); + + let resource = Resource::builder_empty() + .with_service_name(SERVICE_NAME) + .with_attribute(KeyValue::new("service.version", "1.0")) + .with_attribute(KeyValue::new("deployment.environment", "local")) + .build(); + + let span_exporter = opentelemetry_otlp::SpanExporter::builder() + .with_http() + .with_endpoint(&traces_url) + .build() + .context("trace exporter")?; + + let batch_processor = BatchSpanProcessor::builder(span_exporter) + .with_batch_config( + BatchConfigBuilder::default() + .with_scheduled_delay(EXPORT_INTERVAL) + .build(), + ) + .build(); + + let tracer_provider = SdkTracerProvider::builder() + .with_span_processor(batch_processor) + .with_resource(resource.clone()) + .with_sampler(Sampler::AlwaysOn) + .build(); + + let metric_exporter = opentelemetry_otlp::MetricExporter::builder() + .with_http() + .with_endpoint(&metrics_url) + .build() + .context("metric exporter")?; + + let reader = PeriodicReader::builder(metric_exporter) + .with_interval(EXPORT_INTERVAL) + .build(); + + let meter_provider = SdkMeterProvider::builder() + .with_resource(resource) + .with_reader(reader) + .build(); + + let tracer = tracer_provider.tracer(SERVICE_NAME); + let meter = meter_provider.meter(SERVICE_NAME); + + let requests_total = meter + .u64_counter("requests_total") + .with_description("Total requests handled") + .with_unit("1") + .build(); + let request_duration = meter + .f64_histogram("request_duration") + .with_description("Request duration") + .with_unit("ms") + .build(); + + Ok(Telemetry { + tracer_provider, + meter_provider, + tracer, + requests_total, + request_duration, + }) +} + +// ------------------------------------------------------------------- serve + +// handle_request is the parse->work->respond pipeline for one line of input. +// "parse" happens inline (cheap: one i64 parse, done by the caller); "work" +// and "respond" are the two child spans under the "request" parent, per the +// trace contract every language in this example follows identically. +fn handle_request(t: &Telemetry, w: &mut TcpStream, seq: i64) -> std::io::Result<()> { + let start = Instant::now(); + let parent_span = t + .tracer + .span_builder("request") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &Context::new()); + let parent_cx = Context::current_with_span(parent_span); + + // work: a tiny, deterministic-but-variable amount of simulated cost. + let mut work_span = t + .tracer + .span_builder("work") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &parent_cx); + thread::sleep(Duration::from_millis((1 + seq % 4) as u64)); + work_span.end(); + + // respond: write the reply line. + let mut respond_span = t + .tracer + .span_builder("respond") + .with_attributes(vec![KeyValue::new("request.seq", seq)]) + .start_with_context(&t.tracer, &parent_cx); + let result = (|| -> std::io::Result<()> { + writeln!(w, "ok {seq}")?; + w.flush() + })(); + if let Err(ref e) = result { + respond_span.record_error(e); + } + respond_span.end(); + parent_cx.span().end(); + result?; + + let elapsed_ms = start.elapsed().as_secs_f64() * 1000.0; + t.requests_total.add(1, &[]); + t.request_duration.record(elapsed_ms, &[]); + Ok(()) +} + +fn serve_conn(t: &Telemetry, mut stream: TcpStream) { + let reader_side = match stream.try_clone() { + Ok(s) => s, + Err(_) => return, + }; + let mut reader = BufReader::new(reader_side); + let mut line = String::new(); + loop { + line.clear(); + let n = match reader.read_line(&mut line) { + Ok(n) => n, + Err(_) => return, + }; + if n == 0 { + return; // EOF: client closed the connection. + } + let trimmed = line.trim(); + if trimmed.is_empty() { + continue; + } + match trimmed.parse::() { + Ok(seq) => { + if let Err(e) = handle_request(t, &mut stream, seq) { + eprintln!("app: request error: {e}"); + return; + } + } + Err(_) => { + if stream.write_all(b"err bad-request\n").is_err() { + return; + } + } + } + } +} + +// Event unifies the two sources serve waits on — an incoming connection and +// a shutdown signal — into the single channel a Go select over sigCh/connCh +// would use. +enum Event { + Signal(&'static str), + Conn(TcpStream), +} + +fn run_serve(args: &[String]) -> i32 { + let mut port = String::new(); + let mut endpoint = + std::env::var("OTEL_ENDPOINT").unwrap_or_else(|_| "http://localhost:4318".to_string()); + let mut i = 0; + while i < args.len() { + match args[i].as_str() { + "--port" => { + i += 1; + if i >= args.len() { + usage(); + } + port = args[i].clone(); + } + "--otel-endpoint" => { + i += 1; + if i >= args.len() { + usage(); + } + endpoint = args[i].clone(); + } + _ => usage(), + } + i += 1; + } + if port.is_empty() { + usage(); + } + + let t = match init_telemetry(&endpoint) { + Ok(t) => t, + Err(e) => die(&format!("telemetry init: {e}")), + }; + + let listener = match TcpListener::bind(format!("127.0.0.1:{port}")) { + Ok(l) => l, + Err(e) => die(&format!("listen: {e}")), + }; + + let (tx, rx) = mpsc::channel::(); + + // Signal watcher: a dedicated thread blocks on signal-hook's iterator + // and forwards the first SIGINT/SIGTERM onto the shared event channel. + { + let tx = tx.clone(); + thread::spawn(move || { + let mut signals = match Signals::new([SIGINT, SIGTERM]) { + Ok(s) => s, + Err(_) => return, + }; + if let Some(sig) = signals.forever().next() { + let name = match sig { + SIGINT => "SIGINT", + SIGTERM => "SIGTERM", + _ => "signal", + }; + let _ = tx.send(Event::Signal(name)); + } + }); + } + + // Accept loop: a dedicated thread forwards each accepted connection onto + // the same event channel. The process exits (via std::process::exit) + // from the shutdown path below rather than closing the listener, so this + // thread never needs to be joined — same lifecycle as Go's accept + // goroutine, which os.Exit simply cuts short. + { + let tx = tx.clone(); + thread::spawn(move || { + for stream in listener.incoming() { + match stream { + Ok(s) => { + if tx.send(Event::Conn(s)).is_err() { + return; + } + } + Err(_) => return, // listener closed during shutdown + } + } + }); + } + + println!("app: serve listening on 127.0.0.1:{port}"); + + loop { + match rx.recv() { + Ok(Event::Signal(name)) => { + println!("app: shutting down ({name})"); + t.shutdown(); + return 0; + } + Ok(Event::Conn(stream)) => { + serve_conn(&t, stream); + } + Err(_) => return 0, + } + } +} + +// ------------------------------------------------------------------- drive + +// run_drive is a plain, untraced TCP client: it generates the load that +// makes serve's telemetry interesting, but emits none of its own — the +// service under observation is serve, not drive. +fn run_drive(args: &[String]) -> i32 { + let mut n: i64 = 0; + let mut addr = "127.0.0.1:8080".to_string(); + let mut i = 0; + while i < args.len() { + match args[i].as_str() { + "--n" => { + i += 1; + if i >= args.len() { + usage(); + } + match args[i].parse::() { + Ok(v) => n = v, + Err(e) => die(&format!("--n: {e}")), + } + } + "--addr" => { + i += 1; + if i >= args.len() { + usage(); + } + addr = args[i].clone(); + } + _ => usage(), + } + i += 1; + } + if n <= 0 { + usage(); + } + + let stream = match TcpStream::connect(&addr) { + Ok(s) => s, + Err(e) => die(&format!("dial {addr}: {e}")), + }; + let mut writer = match stream.try_clone() { + Ok(s) => s, + Err(e) => die(&format!("dial {addr}: {e}")), + }; + let mut reader = BufReader::new(stream); + + let mut ok: i64 = 0; + let mut line = String::new(); + for seq in 1..=n { + if let Err(e) = writeln!(writer, "{seq}") { + die(&format!("send request {seq}: {e}")); + } + if let Err(e) = writer.flush() { + die(&format!("flush request {seq}: {e}")); + } + line.clear(); + match reader.read_line(&mut line) { + Ok(0) => break, // EOF + Ok(_) => { + if line.trim_end().starts_with("ok ") { + ok += 1; + } + } + Err(e) => die(&format!("read reply {seq}: {e}")), + } + } + println!("app: drive sent {ok}/{n} ok"); + if ok != n { 1 } else { 0 } +} + +// ---------------------------------------------------------------------main + +fn main() { + let args: Vec = std::env::args().collect(); + if args.len() < 2 { + usage(); + } + let code = match args[1].as_str() { + "serve" => run_serve(&args[2..]), + "drive" => run_drive(&args[2..]), + _ => usage(), + }; + std::process::exit(code); +} diff --git a/examples/38-opentelemetry-to-lgtm/verify.lua b/examples/38-opentelemetry-to-lgtm/verify.lua new file mode 100644 index 0000000..1b805ba --- /dev/null +++ b/examples/38-opentelemetry-to-lgtm/verify.lua @@ -0,0 +1,178 @@ +-- Verify lsp-otel (38-opentelemetry-to-lgtm) for LSP_LANG. A LOCAL-mode +-- example that requires the LGTM stack (podman container `lsp-lgtm`, OTLP on +-- :4318, Grafana on :3000). Unlike the vm examples this runs entirely on the +-- host: it builds the binary, runs `serve` in the background, drives N traced +-- requests through it, sends SIGINT so the SDK flushes and shuts down, then +-- proves the telemetry actually LANDED IN THE STACK by querying Prometheus +-- (metrics) and Tempo (traces) through Grafana's datasource proxy -- not by +-- trusting the program's own "I exported" claim. +-- +-- The query surface: only Grafana (:3000, admin/admin) is published off the +-- otel-lgtm container; it proxies to the internal Prometheus (uid=prometheus, +-- metrics) and Tempo (uid=tempo, traces). Each language tags its telemetry +-- with a distinct service.name -- lsp-otel-cpp / lsp-otel-go / lsp-otel-rust +-- -- so the three languages' signals never collide in the shared stack and +-- each run is queried by its own service_name. +-- +-- Asserts observable behavior, identical in shape across cpp/go/rust: +-- 1. build exits 0. +-- 2. error shapes: no args, `serve` without --port, `drive` --n 0 all exit +-- 2 with the two-line usage; `drive` against a dead address exits 1 with +-- `app: error:`. +-- 3. serve prints `app: serve listening on 127.0.0.1:PORT`; drive prints +-- `app: drive sent N/N ok` and exits 0; SIGINT makes serve print +-- `app: shutting down` and exit 0 (the SDK flush-then-shutdown path). +-- 4. metrics reached Prometheus: requests_total{service_name=...} is >= N +-- (the counter the parent span increments once per request) and +-- request_duration_milliseconds_count{service_name=...} is >= N (the +-- histogram records one observation per request). +-- 5. traces reached Tempo: at least one trace with rootServiceName=... and +-- rootTraceName=request (the parent span wrapping work+respond). +-- Metrics/traces are polled with a bounded retry because the SDK batches on a +-- ~2s interval and the collector/backends add their own ingest lag. +-- +-- If the LGTM stack is not reachable the whole example SKIPs (per the runner +-- convention: a down stack is not a failing example), matching manifest.yaml +-- `requires: [lgtm]`. +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local example_dir = os.getenv("EXAMPLE_DIR") or "." +local grafana = "http://localhost:3000" +local svc = "lsp-otel-" .. lang +local port = 47380 +local n = 10 + +-- Boolean assertion via the checks.lua pattern-match API (same helper as the +-- other chapters' verify.lua): match on success, force a labelled mismatch on +-- failure so it surfaces exactly like an expect_match failure. +local function expect_true(cond, label, detail) + if cond then + checks.expect_match("ok", "ok", label) + else + local msg = label + if detail ~= nil then msg = msg .. " (" .. tostring(detail) .. ")" end + checks.expect_match("no", "yes", msg) + end +end + +-- LGTM reachability: a down stack SKIPs the whole example, it does not fail. +local health = checks.run( + "curl -s -o /dev/null -w '%{http_code}' -u admin:admin " .. grafana .. "/api/health") +if not health.out:match("200") then + checks.skip("LGTM stack not reachable at " .. grafana .. " (requires the lsp-lgtm container)") +end + +-- Local path of the freshly built binary for this language. +local bin_rel = ({ + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +})[lang] +local bin = example_dir .. "/" .. bin_rel + +-- --------------------------------------------------------------------------- +-- 1. build. +-- --------------------------------------------------------------------------- +local build = checks.run("./demo.sh " .. lang .. " build") +checks.expect_exit(build, 0, lang .. ": demo.sh build exits 0") + +-- --------------------------------------------------------------------------- +-- 2. error shapes. +-- --------------------------------------------------------------------------- +local no_args = checks.run(bin) +checks.expect_exit(no_args, 2, lang .. ": no args exits 2") +checks.expect_match(no_args.out, "usage: app serve", lang .. ": no args prints usage") + +local serve_noport = checks.run(bin .. " serve") +checks.expect_exit(serve_noport, 2, lang .. ": serve without --port exits 2") + +local drive_zero = checks.run(bin .. " drive --n 0 --addr 127.0.0.1:1") +checks.expect_exit(drive_zero, 2, lang .. ": drive --n 0 exits 2") + +local drive_dead = checks.run(bin .. " drive --n 5 --addr 127.0.0.1:1") +checks.expect_exit(drive_dead, 1, lang .. ": drive against a dead address exits 1") +checks.expect_match(drive_dead.out, "app: error:", lang .. ": dead-address drive prints app: error:") + +-- --------------------------------------------------------------------------- +-- 3-5. run serve+drive+flush, then confirm telemetry landed in the stack. +-- One shell script so the background serve, the drive, the SIGINT flush, and +-- the bounded metric/trace polling all share process state; it echoes simple +-- KEY=value tokens this file then parses. +-- --------------------------------------------------------------------------- +local q = grafana .. "/api/datasources/proxy/uid/prometheus/api/v1/query?query=" +local tempo = grafana .. "/api/datasources/proxy/uid/tempo/api/search?tags=service.name%3D" .. svc .. "&limit=1" + +-- URL-encoded PromQL: metric{service_name="lsp-otel-"} +local prom_req = "requests_total%7Bservice_name%3D%22" .. svc .. "%22%7D" +local prom_hist = "request_duration_milliseconds_count%7Bservice_name%3D%22" .. svc .. "%22%7D" + +local script = table.concat({ + "set -u", + "OUT=/tmp/lsp38_" .. lang .. ".$$", + "SRV=$OUT.serve", + bin .. " serve --port " .. port .. " >$SRV 2>&1 &", + "SPID=$!", + "for i in $(seq 1 50); do grep -q 'serve listening' $SRV 2>/dev/null && break; sleep 0.1; done", + bin .. " drive --n " .. n .. " --addr 127.0.0.1:" .. port .. " >$OUT.drive 2>&1", + "echo DRIVE_EXIT=$?", + "cat $OUT.drive", + "kill -INT $SPID 2>/dev/null", + "if timeout 10 tail --pid=$SPID -f /dev/null 2>/dev/null; then wait $SPID; echo SERVE_EXIT=$?; else echo SERVE_EXIT=124; kill -9 $SPID 2>/dev/null; fi", + "cat $SRV", + -- poll Prometheus for the counter (bounded ~15s for batch + ingest lag) + "REQ=; for i in $(seq 1 30); do", + " REQ=$(curl -s -u admin:admin '" .. q .. prom_req .. "' | python3 -c 'import sys,json;r=json.load(sys.stdin)[\"data\"][\"result\"];print(int(float(r[0][\"value\"][1])) if r else \"\")' 2>/dev/null)", + " [ -n \"$REQ\" ] && break; sleep 0.5; done", + "echo PROM_REQUESTS=$REQ", + "HCNT=$(curl -s -u admin:admin '" .. q .. prom_hist .. "' | python3 -c 'import sys,json;r=json.load(sys.stdin)[\"data\"][\"result\"];print(int(float(r[0][\"value\"][1])) if r else \"\")' 2>/dev/null)", + "echo PROM_HIST_COUNT=$HCNT", + -- poll Tempo for a trace + "TR=0; for i in $(seq 1 20); do", + " TR=$(curl -s -u admin:admin '" .. tempo .. "' | python3 -c 'import sys,json;print(len(json.load(sys.stdin).get(\"traces\",[])))' 2>/dev/null)", + " [ \"$TR\" != \"0\" ] && [ -n \"$TR\" ] && break; sleep 0.5; done", + "echo TEMPO_TRACES=$TR", + "rm -f $OUT $OUT.serve $OUT.drive", +}, "\n") + +local r = checks.run(script) + +-- serve/drive observable behavior +checks.expect_match(r.out, "app: serve listening on 127%.0%.0%.1:" .. port, + lang .. ": serve prints the listening line") +checks.expect_match(r.out, "app: drive sent " .. n .. "/" .. n .. " ok", + lang .. ": drive reports N/N ok") +local drive_exit = r.out:match("DRIVE_EXIT=(%d+)") +expect_true(drive_exit == "0", lang .. ": drive exits 0", "DRIVE_EXIT=" .. tostring(drive_exit)) +local serve_exit = r.out:match("SERVE_EXIT=(%d+)") +expect_true(serve_exit == "0", + lang .. ": serve exits 0 after SIGINT (SDK flush-then-shutdown)", "SERVE_EXIT=" .. tostring(serve_exit)) +checks.expect_match(r.out, "app: shutting down", lang .. ": serve prints the shutdown line") + +-- metrics landed in Prometheus +local prom_req_v = tonumber(r.out:match("PROM_REQUESTS=(%d+)")) +expect_true(prom_req_v ~= nil and prom_req_v >= n, + lang .. ": requests_total{service_name=" .. svc .. "} reached Prometheus and is >= " .. n, + "PROM_REQUESTS=" .. tostring(r.out:match("PROM_REQUESTS=(%S*)"))) +local prom_hist_v = tonumber(r.out:match("PROM_HIST_COUNT=(%d+)")) +expect_true(prom_hist_v ~= nil and prom_hist_v >= n, + lang .. ": request_duration_milliseconds_count{service_name=" .. svc .. "} is >= " .. n, + "PROM_HIST_COUNT=" .. tostring(r.out:match("PROM_HIST_COUNT=(%S*)"))) + +-- traces landed in Tempo +local tempo_v = tonumber(r.out:match("TEMPO_TRACES=(%d+)")) +expect_true(tempo_v ~= nil and tempo_v >= 1, + lang .. ": at least one trace with service.name=" .. svc .. " reached Tempo", + "TEMPO_TRACES=" .. tostring(r.out:match("TEMPO_TRACES=(%S*)"))) + +checks.finish() diff --git a/examples/39-benchmarking-without-lies/README.md b/examples/39-benchmarking-without-lies/README.md new file mode 100644 index 0000000..86d52cb --- /dev/null +++ b/examples/39-benchmarking-without-lies/README.md @@ -0,0 +1,181 @@ +# 39-benchmarking-without-lies + +Chapter 39's example: `benchlab`, a microbenchmark harness for the chatterd +frame codec (`encode_frame`/`decode_frame`, introduced ch21) — implemented +three times, C++23, Go, and Rust, with one shared observable contract. The +point of this chapter is not the codec; it's the two different ways to +measure it, side by side in the same binary. + +## The program + +``` +benchlab --op encode|decode|roundtrip --iters N [--warmup W] # the real thing +benchlab --lie [--op encode|decode|roundtrip] # the anti-pattern +``` + +`--op` selects what's timed against a fixed workload — one chatterd `DELIVER` +frame (`alice\0the quick brown fox jumps over the lazy dog, three times, for +benchlab`): + +- `encode` — build the frame from the payload bytes. +- `decode` — parse a pre-built frame back into `(type, payload)`. +- `roundtrip` — both, per iteration. + +### The real thing + +`--iters N --warmup W` (default `W=1000`) runs `W` untimed warmup iterations +— letting the allocator, branch predictor, and (on Go) the runtime reach +something like steady state — then times each of the next `N` iterations +individually with a monotonic clock, and reports: + +``` +$ ./demo.sh cpp run --op roundtrip --iters 100000 --warmup 5000 +benchlab: op=roundtrip iters=100000 warmup=5000 +benchlab: n=100000 min_ns=57 median_ns=87 p99_ns=124 max_ns=1012817 +benchlab: co_p99_ns=914420 expected_interval_ns=87 co_n=113003 +benchlab: checksum=000000000079c3e0 +``` + +- `min_ns`/`median_ns`/`p99_ns`/`max_ns` — the raw latency distribution in + nanoseconds, nearest-rank percentiles over the sorted sample. +- `co_p99_ns` — the **coordinated-omission-corrected** p99 (see below); + always `>= p99_ns`. +- `expected_interval_ns` — the interval the correction assumes, derived from + the run's own raw median (documented, not a hidden knob). +- `co_n` — how many samples the corrected distribution actually holds + (`>= n`, since correction only ever adds synthetic samples). +- `checksum` — a running sum of a value derived from every call's result + (the encoded frame's last byte, or the decoded payload's length). Two + purposes: it stops the compiler from proving the loop body's result is + unused and deleting it, and — because the workload is fixed — it is + **bit-for-bit identical every run and across all three languages** + (`encode` above: `00000000000fb388` in C++, Go, *and* Rust, for the exact + same input). If the codec is behaving identically everywhere, the + checksums must match; a diverging checksum is a bug, not noise. + +The `min`/`median`/`p99`/`max` numbers above are one machine's numbers — like +every other latency figure in this book, their magnitude is run- and +hardware-dependent. What's invariant, and what `verify.lua` actually checks, +is the *shape*: `min <= median <= p99 <= max`, `co_p99 >= p99`, `co_n >= n`, +and a stable checksum across repeated runs of the same workload. + +### Coordinated omission, and why a tight loop needs correcting for it + +This harness is a **closed loop**: iteration *N+1* never starts until +iteration *N* returns. That's also the classic setup for **coordinated +omission** (Gil Tene's term): if iteration *N* stalls — a page fault, a +context switch, a GC-adjacent pause, a neighbor process getting the core — +the *one* sample recorded for it is that stall's full latency, but a +fixed-rate caller (a real client sending one request every `E` nanoseconds) +would have had several requests queue up behind that stall, each observing a +delay close to the stall's length. Recording only one slow sample instead of +several **erases exactly the tail** a benchmark is supposed to report +accurately. + +The correction (HdrHistogram's `recordValueWithExpectedInterval`): for every +raw sample `v` above the expected interval `E`, backfill synthetic samples +`v - E, v - 2E, ...` down to (but not below) `E`, representing the virtual +requests that would have been queued behind the stall. `E` here is set to the +run's own raw median — i.e., "assume this loop could sustain its typical +per-op rate" — so no extra flag is needed and the two numbers come from one +run. A safety cap (`co_n <= 5,000,000`) stops a pathological single giant +stall from producing an unbounded backfill. + +`co_p99_ns` in the example above (`914420`) vs. the raw `p99_ns` (`124`) is +the whole lesson: one paused iteration during a 100k-iteration run is invisible +to the raw p99 (only 1000 of 100000 samples count toward it) but dominates the +corrected one, because that single stall represented what would have been +thousands of queued requests in a real fixed-rate system. + +### `--lie` — the benchmark this chapter argues against + +``` +$ ./demo.sh cpp run --lie --op roundtrip +benchlab: lie op=roundtrip (no warmup, single wall-clock sample, ignores variance) +benchlab: lie elapsed_ns=3069 sink=76 +``` + +No warmup, one call, one wall-clock delta, no percentiles, no variance — and +therefore no way to tell a JIT/allocator-cold first call, a scheduler +preemption, or a page fault apart from steady-state performance. This is not +a strawman: it is the shape of `time single_call()` benchmarks that get +pasted into commit messages. `--lie`'s output never contains `p99_ns=` or +`median_ns=` — there is nothing here to compute a percentile *from* — and +`verify.lua` asserts that absence directly. + +## Wire format (the codec under test, unchanged since ch21) + +``` + byte: 0 1 2 3 4 5 6 ............. 6+LEN-1 + +-----+-----+-------+------+-----+-----+---------------------------+ + | 'C' | 'H' | 0x01 | TYPE | LENGTH | PAYLOAD | + +-----+-----+-------+------+-----+-----+---------------------------+ + magic version u16 be (LEN) LEN bytes +``` + +`benchlab` re-implements only `encode_frame`/`decode_frame` against this +format (a `DELIVER`, type `0x03`) — not the daemon; see ch21/22/27 for the +networked chatterd that actually speaks this frame over a socket. `decode` +and `roundtrip` use `std::expected` (C++), +`Result<(u8, Vec), String>` with `?` (Rust), and a wrapped `error` via +`%w` (Go) to report a malformed frame — on this fixed, always-valid workload +that path is never taken, but it exists because a hand-rolled codec harness +should look exactly like calling code would, not a stripped-down happy path. + +## Layout + +``` +39-benchmarking-without-lies/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build, hand-rolled harness, demo.sh +├── go/ # go build, hand-rolled harness (testing.B-style loop), demo.sh +└── rust/ # cargo build, hand-rolled harness, demo.sh +``` + +Every language uses a small hand-rolled harness rather than a benchmarking +framework (Google Benchmark / `go test -bench` / criterion): `benchlab` is a +CLI that prints one shared, cross-language table shape, which a framework's +own report format would not give us for free, and there is no third-party +dependency to pin or fetch for any of the three languages — `rust/Cargo.lock` +carries no external crates. + +## The demo contract + +Every language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary + (e.g. `./demo.sh run --op decode --iters 100000 --warmup 5000`) +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh` instead of running locally. + +This is a host-only example — pure in-process computation, no sockets, no VM. +Exit codes: `0` a run completed and printed its table (or its lie), `1` the +codec broke on its own fixed workload (a bug, never expected here), `2` a +usage error (missing/unknown flags, or `--warmup >= --iters`). + +## Verification + +`verify.lua` runs from this directory with `LSP_LANG` set to one language at a +time and asserts, per language: + +1. for `encode`, `decode`, and `roundtrip` at `--iters 20000 --warmup 1000`: + the header line echoes `op`/`iters`/`warmup`; `n` equals the requested + `iters`; `min_ns <= median_ns <= p99_ns <= max_ns`; `co_p99_ns >= p99_ns`; + `co_n >= n`; `expected_interval_ns` matches the documented derivation + (the run's own raw median, floored at 1); and a 64-bit hex checksum is + printed; +2. running the identical fixed workload twice yields the identical checksum + both times — the codec's output is deterministic even though the ns + timings are not; +3. `--lie` exits 0, identifies itself as the lie, reports exactly one + `elapsed_ns` sample, and contains **no** `p99_ns=`/`median_ns=` anywhere + in its output; +4. no flags, an unknown `--op`, and `--warmup >= --iters` each exit 2 with a + usage line. + +Verified on the Fedora 44 reference host (kernel 7.1.3-200.fc44, GCC 16.1.1, +go1.26.5, rustc 1.97.1): all three languages PASS all 42 checks, repeatably +across multiple runs. diff --git a/examples/39-benchmarking-without-lies/cpp/.clang-format b/examples/39-benchmarking-without-lies/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/39-benchmarking-without-lies/cpp/.clang-tidy b/examples/39-benchmarking-without-lies/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/39-benchmarking-without-lies/cpp/CMakeLists.txt b/examples/39-benchmarking-without-lies/cpp/CMakeLists.txt new file mode 100644 index 0000000..357c950 --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(benchlab CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/39-benchmarking-without-lies/cpp/CMakePresets.json b/examples/39-benchmarking-without-lies/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/39-benchmarking-without-lies/cpp/demo.sh b/examples/39-benchmarking-without-lies/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/39-benchmarking-without-lies/cpp/src/main.cpp b/examples/39-benchmarking-without-lies/cpp/src/main.cpp new file mode 100644 index 0000000..c85d3f4 --- /dev/null +++ b/examples/39-benchmarking-without-lies/cpp/src/main.cpp @@ -0,0 +1,287 @@ +// benchlab (C++23) — proper benchmarking of the chatterd frame codec (ch21). +// +// Usage: +// app --op encode|decode|roundtrip --iters N [--warmup W] # the real thing +// app --lie [--op encode|decode|roundtrip] # the anti-pattern +// +// The "real thing" times each iteration individually with a monotonic clock, +// discards a warmup phase, and reports min/median/p99/max plus a +// coordinated-omission-corrected p99 (see co_correct below). `--lie` times a +// single unwarmed call with wall-clock and reports one number — exactly the +// benchmark the rest of this example exists to discredit. +// +// Wire format (canonical chatterd chat frame, introduced ch21): +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// This file re-implements only encode_frame/decode_frame — the codec under +// test — not the daemon; see ch21/22/27 for the networked version. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +// --------------------------------------------------------------------------- +// Frame codec under test — canonical chatterd chat frame (see file header). +// --------------------------------------------------------------------------- +constexpr std::size_t kHeaderLen = 6; +constexpr std::uint8_t kMagic0 = 0x43; +constexpr std::uint8_t kMagic1 = 0x48; +constexpr std::uint8_t kVersion = 0x01; +constexpr std::uint8_t kTypeDeliver = 3; + +struct DecodedFrame { + std::uint8_t type; + std::string body; +}; + +std::string encode_frame(std::uint8_t type, std::string_view body) { + std::string out; + out.reserve(kHeaderLen + body.size()); + out.push_back(static_cast(kMagic0)); + out.push_back(static_cast(kMagic1)); + out.push_back(static_cast(kVersion)); + out.push_back(static_cast(type)); + auto len = static_cast(body.size()); + out.push_back(static_cast((len >> 8) & 0xff)); + out.push_back(static_cast(len & 0xff)); + out.append(body); + return out; +} + +// Decodes exactly one frame that fills `buf` completely — no partial-read +// reassembly here, that lives in the networked chatterd (ch21/22/27); this +// harness only measures the codec's per-call cost. +[[nodiscard]] std::expected decode_frame(std::string_view buf) { + if (buf.size() < kHeaderLen) { + return std::unexpected("frame shorter than header"); + } + if (static_cast(buf[0]) != kMagic0 || static_cast(buf[1]) != kMagic1) { + return std::unexpected("bad magic"); + } + if (static_cast(buf[2]) != kVersion) { + return std::unexpected("bad version"); + } + std::uint8_t type = static_cast(buf[3]); + std::uint16_t len = static_cast((static_cast(buf[4]) << 8) | + static_cast(buf[5])); + if (buf.size() != kHeaderLen + len) { + return std::unexpected("length mismatch"); + } + return DecodedFrame{type, std::string(buf.substr(kHeaderLen))}; +} + +// --------------------------------------------------------------------------- +// The fixed workload: a DELIVER frame carrying a realistic chat line. Every +// language in this example encodes/decodes the identical bytes. +// --------------------------------------------------------------------------- +constexpr std::string_view kNick = "alice"; +constexpr std::string_view kText = "the quick brown fox jumps over the lazy dog, three times, for benchlab"; + +std::string delivery_body() { + std::string body; + body.reserve(kNick.size() + 1 + kText.size()); + body.append(kNick); + body.push_back('\0'); + body.append(kText); + return body; +} + +enum class Op { Encode, Decode, Roundtrip }; + +std::expected parse_op(std::string_view s) { + if (s == "encode") return Op::Encode; + if (s == "decode") return Op::Decode; + if (s == "roundtrip") return Op::Roundtrip; + return std::unexpected("unknown --op '" + std::string(s) + "'"); +} + +std::string_view op_name(Op op) { + switch (op) { + case Op::Encode: return "encode"; + case Op::Decode: return "decode"; + case Op::Roundtrip: return "roundtrip"; + } + return "?"; +} + +// One iteration of the requested op against the fixed workload. Returns a +// small scalar derived from the result so the caller can fold it into a +// checksum — this is what keeps the optimizer from proving the call's result +// is unused and deleting it. +std::uint64_t run_once(Op op, const std::string& body, const std::string& prebuilt_frame) { + switch (op) { + case Op::Encode: { + std::string frame = encode_frame(kTypeDeliver, body); + return frame.empty() ? 0 : static_cast(frame.back()); + } + case Op::Decode: { + auto decoded = decode_frame(prebuilt_frame); + if (!decoded) { + std::println(stderr, "benchlab: codec bug: decode failed: {}", decoded.error()); + std::exit(1); + } + return decoded->body.size(); + } + case Op::Roundtrip: { + std::string frame = encode_frame(kTypeDeliver, body); + auto decoded = decode_frame(frame); + if (!decoded) { + std::println(stderr, "benchlab: codec bug: decode failed: {}", decoded.error()); + std::exit(1); + } + return decoded->body.size(); + } + } + return 0; +} + +using Clock = std::chrono::steady_clock; + +std::int64_t ns_between(Clock::time_point a, Clock::time_point b) { + return std::chrono::duration_cast(b - a).count(); +} + +std::int64_t percentile(const std::vector& sorted, double p) { + if (sorted.empty()) return 0; + std::size_t idx = static_cast(p * static_cast(sorted.size() - 1)); + return sorted[idx]; +} + +// Coordinated-omission correction (HdrHistogram's recordValueWithExpectedInterval): +// this harness is a closed loop — iteration N+1 never starts until N returns — +// so a stall (a page fault, a context switch, a scheduling hiccup) is +// recorded as a single slow sample. A fixed-rate caller would instead have +// had many requests queue up behind that stall, each experiencing a similar +// multiple-of-the-target-interval delay. This backfills those missing +// virtual samples so the tail reflects what a real caller would have seen, +// not just what one lucky/unlucky iteration measured. +std::vector co_correct(const std::vector& raw, + std::int64_t expected_interval_ns, + std::size_t cap) { + std::vector out; + out.reserve(raw.size()); + for (std::int64_t v : raw) { + out.push_back(v); + if (expected_interval_ns <= 0 || v <= expected_interval_ns) continue; + std::int64_t missing = v - expected_interval_ns; + while (missing >= expected_interval_ns && out.size() < cap) { + out.push_back(missing); + missing -= expected_interval_ns; + } + if (out.size() >= cap) break; + } + return out; +} + +void usage_and_exit() { + std::println(stderr, "usage: benchlab --op encode|decode|roundtrip --iters N [--warmup W]"); + std::println(stderr, " benchlab --lie [--op encode|decode|roundtrip]"); + std::exit(2); +} + +std::expected parse_u64(std::string_view s) { + if (s.empty()) return std::unexpected("empty number"); + long long v = 0; + for (char c : s) { + if (c < '0' || c > '9') return std::unexpected("not a number: " + std::string(s)); + v = v * 10 + (c - '0'); + } + return v; +} + +} // namespace + +int main(int argc, char** argv) { + bool lie = false; + std::string op_str; + long long iters = -1; + long long warmup = 1000; + + for (int i = 1; i < argc; ++i) { + std::string_view a = argv[i]; + if (a == "--lie") { + lie = true; + } else if (a == "--op" && i + 1 < argc) { + op_str = argv[++i]; + } else if (a == "--iters" && i + 1 < argc) { + auto v = parse_u64(argv[++i]); + if (!v) usage_and_exit(); + iters = *v; + } else if (a == "--warmup" && i + 1 < argc) { + auto v = parse_u64(argv[++i]); + if (!v) usage_and_exit(); + warmup = *v; + } else { + usage_and_exit(); + } + } + + if (!lie && op_str.empty()) usage_and_exit(); + Op op = Op::Roundtrip; + if (!op_str.empty()) { + auto parsed = parse_op(op_str); + if (!parsed) usage_and_exit(); + op = *parsed; + } + + const std::string body = delivery_body(); + const std::string prebuilt = encode_frame(kTypeDeliver, body); + + if (lie) { + // The anti-pattern: no warmup, one call, wall-clock, done. This is + // exactly the benchmark this example's README argues never to trust. + auto t0 = Clock::now(); + std::uint64_t sink = run_once(op, body, prebuilt); + auto t1 = Clock::now(); + std::println("benchlab: lie op={} (no warmup, single wall-clock sample, ignores variance)", op_name(op)); + std::println("benchlab: lie elapsed_ns={} sink={}", ns_between(t0, t1), sink); + return 0; + } + + if (iters < 2 || warmup < 0 || iters <= warmup) usage_and_exit(); + + // Warmup: run the op without timing it, letting allocators/caches/branch + // predictors reach steady state before any sample is recorded. + std::uint64_t checksum = 0; + for (long long i = 0; i < warmup; ++i) { + checksum += run_once(op, body, prebuilt); + } + + std::vector raw; + raw.reserve(static_cast(iters)); + for (long long i = 0; i < iters; ++i) { + auto t0 = Clock::now(); + checksum += run_once(op, body, prebuilt); + auto t1 = Clock::now(); + raw.push_back(ns_between(t0, t1)); + } + + std::vector sorted = raw; + std::sort(sorted.begin(), sorted.end()); + std::int64_t min_ns = sorted.front(); + std::int64_t median_ns = percentile(sorted, 0.5); + std::int64_t p99_ns = percentile(sorted, 0.99); + std::int64_t max_ns = sorted.back(); + + std::int64_t expected_interval_ns = median_ns > 0 ? median_ns : 1; + constexpr std::size_t kCoCap = 5'000'000; + std::vector corrected = co_correct(raw, expected_interval_ns, kCoCap); + std::sort(corrected.begin(), corrected.end()); + std::int64_t co_p99_ns = percentile(corrected, 0.99); + + std::println("benchlab: op={} iters={} warmup={}", op_name(op), iters, warmup); + std::println("benchlab: n={} min_ns={} median_ns={} p99_ns={} max_ns={}", + sorted.size(), min_ns, median_ns, p99_ns, max_ns); + std::println("benchlab: co_p99_ns={} expected_interval_ns={} co_n={}", + co_p99_ns, expected_interval_ns, corrected.size()); + std::println("benchlab: checksum={:016x}", checksum); + return 0; +} diff --git a/examples/39-benchmarking-without-lies/demo.sh b/examples/39-benchmarking-without-lies/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/39-benchmarking-without-lies/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/39-benchmarking-without-lies/go/demo.sh b/examples/39-benchmarking-without-lies/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/39-benchmarking-without-lies/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/39-benchmarking-without-lies/go/go.mod b/examples/39-benchmarking-without-lies/go/go.mod new file mode 100644 index 0000000..55747fc --- /dev/null +++ b/examples/39-benchmarking-without-lies/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/39-benchmarking-without-lies/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/39-benchmarking-without-lies/go/main.go b/examples/39-benchmarking-without-lies/go/main.go new file mode 100644 index 0000000..7ba2436 --- /dev/null +++ b/examples/39-benchmarking-without-lies/go/main.go @@ -0,0 +1,347 @@ +// benchlab (Go) — proper benchmarking of the chatterd frame codec (ch21). +// +// Usage: +// +// app --op encode|decode|roundtrip --iters N [--warmup W] # the real thing +// app --lie [--op encode|decode|roundtrip] # the anti-pattern +// +// The "real thing" times each iteration individually with a monotonic clock +// (time.Now's monotonic reading), discards a warmup phase, and reports +// min/median/p99/max plus a coordinated-omission-corrected p99 (see +// coCorrect below). `--lie` times a single unwarmed call with wall-clock and +// reports one number — exactly the benchmark this example exists to +// discredit. +// +// Wire format (canonical chatterd chat frame, introduced ch21): +// +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// +// This file re-implements only encodeFrame/decodeFrame — the codec under +// test — not the daemon; see ch21/22/27 for the networked version. +package main + +import ( + "encoding/binary" + "errors" + "fmt" + "os" + "sort" + "strconv" + "time" +) + +// --------------------------------------------------------------------------- +// Frame codec under test — canonical chatterd chat frame (see file header). +// --------------------------------------------------------------------------- +const ( + headerLen = 6 + magic0 = 0x43 + magic1 = 0x48 + version = 0x01 + + typeDeliver = 3 +) + +func encodeFrame(typ byte, body []byte) []byte { + out := make([]byte, headerLen+len(body)) + out[0] = magic0 + out[1] = magic1 + out[2] = version + out[3] = typ + binary.BigEndian.PutUint16(out[4:6], uint16(len(body))) + copy(out[6:], body) + return out +} + +// decodeFrame decodes exactly one frame that fills buf completely — no +// partial-read reassembly here, that lives in the networked chatterd +// (ch21/22/27); this harness only measures the codec's per-call cost. +func decodeFrame(buf []byte) (byte, []byte, error) { + if len(buf) < headerLen { + return 0, nil, fmt.Errorf("decode frame: %w", errShortHeader) + } + if buf[0] != magic0 || buf[1] != magic1 { + return 0, nil, fmt.Errorf("decode frame: %w", errBadMagic) + } + if buf[2] != version { + return 0, nil, fmt.Errorf("decode frame: %w", errBadVersion) + } + typ := buf[3] + n := binary.BigEndian.Uint16(buf[4:6]) + if len(buf) != headerLen+int(n) { + return 0, nil, fmt.Errorf("decode frame: %w", errLengthMismatch) + } + return typ, buf[headerLen:], nil +} + +var ( + errShortHeader = errors.New("frame shorter than header") + errBadMagic = errors.New("bad magic") + errBadVersion = errors.New("bad version") + errLengthMismatch = errors.New("length mismatch") +) + +// --------------------------------------------------------------------------- +// The fixed workload: a DELIVER frame carrying a realistic chat line. Every +// language in this example encodes/decodes the identical bytes. +// --------------------------------------------------------------------------- +const ( + nick = "alice" + text = "the quick brown fox jumps over the lazy dog, three times, for benchlab" +) + +func deliveryBody() []byte { + body := make([]byte, 0, len(nick)+1+len(text)) + body = append(body, nick...) + body = append(body, 0) + body = append(body, text...) + return body +} + +type op int + +const ( + opEncode op = iota + opDecode + opRoundtrip +) + +func parseOp(s string) (op, error) { + switch s { + case "encode": + return opEncode, nil + case "decode": + return opDecode, nil + case "roundtrip": + return opRoundtrip, nil + default: + return 0, fmt.Errorf("unknown --op %q", s) + } +} + +func (o op) String() string { + switch o { + case opEncode: + return "encode" + case opDecode: + return "decode" + case opRoundtrip: + return "roundtrip" + default: + return "?" + } +} + +// runOnce performs one iteration of the requested op against the fixed +// workload, returning a small scalar derived from the result so the caller +// can fold it into a checksum — this is what keeps the compiler (and a human +// skimming the loop) from mistaking the call for dead code. A codec bug +// (never expected on this fixed workload) is wrapped and propagated. +func runOnce(o op, body, prebuiltFrame []byte) (uint64, error) { + switch o { + case opEncode: + frame := encodeFrame(typeDeliver, body) + if len(frame) == 0 { + return 0, nil + } + return uint64(frame[len(frame)-1]), nil + case opDecode: + _, payload, err := decodeFrame(prebuiltFrame) + if err != nil { + return 0, fmt.Errorf("runOnce decode: %w", err) + } + return uint64(len(payload)), nil + case opRoundtrip: + frame := encodeFrame(typeDeliver, body) + _, payload, err := decodeFrame(frame) + if err != nil { + return 0, fmt.Errorf("runOnce roundtrip: %w", err) + } + return uint64(len(payload)), nil + default: + return 0, fmt.Errorf("runOnce: unknown op %d", o) + } +} + +func percentile(sorted []int64, p float64) int64 { + if len(sorted) == 0 { + return 0 + } + idx := int(p * float64(len(sorted)-1)) + return sorted[idx] +} + +// coCorrect applies the coordinated-omission correction (HdrHistogram's +// recordValueWithExpectedInterval): this harness is a closed loop — +// iteration N+1 never starts until N returns — so a stall (a GC pause, a +// page fault, a scheduling hiccup) is recorded as a single slow sample. A +// fixed-rate caller would instead have had many requests queue up behind +// that stall, each experiencing a similar multiple-of-the-target-interval +// delay. This backfills those missing virtual samples so the tail reflects +// what a real caller would have seen, not just what one lucky/unlucky +// iteration measured. +func coCorrect(raw []int64, expectedIntervalNs int64, cap int) []int64 { + out := make([]int64, 0, len(raw)) + for _, v := range raw { + out = append(out, v) + if expectedIntervalNs <= 0 || v <= expectedIntervalNs { + continue + } + missing := v - expectedIntervalNs + for missing >= expectedIntervalNs { + if len(out) >= cap { + return out + } + out = append(out, missing) + missing -= expectedIntervalNs + } + } + return out +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: benchlab --op encode|decode|roundtrip --iters N [--warmup W]") + fmt.Fprintln(os.Stderr, " benchlab --lie [--op encode|decode|roundtrip]") + os.Exit(2) +} + +func parseUint(s string) (int64, bool) { + if s == "" { + return 0, false + } + for _, c := range s { + if c < '0' || c > '9' { + return 0, false + } + } + v, err := strconv.ParseInt(s, 10, 64) + if err != nil { + return 0, false + } + return v, true +} + +func main() { + var lie bool + var opStr string + var iters int64 = -1 + var warmup int64 = 1000 + + args := os.Args[1:] + for i := 0; i < len(args); i++ { + switch args[i] { + case "--lie": + lie = true + case "--op": + if i+1 >= len(args) { + usage() + } + i++ + opStr = args[i] + case "--iters": + if i+1 >= len(args) { + usage() + } + i++ + v, ok := parseUint(args[i]) + if !ok { + usage() + } + iters = v + case "--warmup": + if i+1 >= len(args) { + usage() + } + i++ + v, ok := parseUint(args[i]) + if !ok { + usage() + } + warmup = v + default: + usage() + } + } + + if !lie && opStr == "" { + usage() + } + selected := opRoundtrip + if opStr != "" { + parsed, err := parseOp(opStr) + if err != nil { + usage() + } + selected = parsed + } + + body := deliveryBody() + prebuilt := encodeFrame(typeDeliver, body) + + if lie { + // The anti-pattern: no warmup, one call, wall-clock, done. This is + // exactly the benchmark this example's README argues never to trust. + t0 := time.Now() + sink, err := runOnce(selected, body, prebuilt) + elapsed := time.Since(t0) + if err != nil { + fmt.Fprintln(os.Stderr, "benchlab: codec bug:", err) + os.Exit(1) + } + fmt.Printf("benchlab: lie op=%s (no warmup, single wall-clock sample, ignores variance)\n", selected) + fmt.Printf("benchlab: lie elapsed_ns=%d sink=%d\n", elapsed.Nanoseconds(), sink) + return + } + + if iters < 2 || warmup < 0 || iters <= warmup { + usage() + } + + // Warmup: run the op without timing it, letting the allocator/GC/branch + // predictors reach steady state before any sample is recorded. + var checksum uint64 + for i := int64(0); i < warmup; i++ { + v, err := runOnce(selected, body, prebuilt) + if err != nil { + fmt.Fprintln(os.Stderr, "benchlab: codec bug:", err) + os.Exit(1) + } + checksum += v + } + + raw := make([]int64, 0, iters) + for i := int64(0); i < iters; i++ { + t0 := time.Now() + v, err := runOnce(selected, body, prebuilt) + elapsed := time.Since(t0) + if err != nil { + fmt.Fprintln(os.Stderr, "benchlab: codec bug:", err) + os.Exit(1) + } + checksum += v + raw = append(raw, elapsed.Nanoseconds()) + } + + sorted := append([]int64(nil), raw...) + sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] }) + minNs := sorted[0] + medianNs := percentile(sorted, 0.5) + p99Ns := percentile(sorted, 0.99) + maxNs := sorted[len(sorted)-1] + + expectedIntervalNs := medianNs + if expectedIntervalNs <= 0 { + expectedIntervalNs = 1 + } + const coCap = 5_000_000 + corrected := coCorrect(raw, expectedIntervalNs, coCap) + sort.Slice(corrected, func(i, j int) bool { return corrected[i] < corrected[j] }) + coP99Ns := percentile(corrected, 0.99) + + fmt.Printf("benchlab: op=%s iters=%d warmup=%d\n", selected, iters, warmup) + fmt.Printf("benchlab: n=%d min_ns=%d median_ns=%d p99_ns=%d max_ns=%d\n", + len(sorted), minNs, medianNs, p99Ns, maxNs) + fmt.Printf("benchlab: co_p99_ns=%d expected_interval_ns=%d co_n=%d\n", + coP99Ns, expectedIntervalNs, len(corrected)) + fmt.Printf("benchlab: checksum=%016x\n", checksum) +} diff --git a/examples/39-benchmarking-without-lies/rust/Cargo.lock b/examples/39-benchmarking-without-lies/rust/Cargo.lock new file mode 100644 index 0000000..1f58d41 --- /dev/null +++ b/examples/39-benchmarking-without-lies/rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "benchlab" +version = "0.1.0" diff --git a/examples/39-benchmarking-without-lies/rust/Cargo.toml b/examples/39-benchmarking-without-lies/rust/Cargo.toml new file mode 100644 index 0000000..54d05fa --- /dev/null +++ b/examples/39-benchmarking-without-lies/rust/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "benchlab" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] + +[profile.release] +opt-level = 3 diff --git a/examples/39-benchmarking-without-lies/rust/demo.sh b/examples/39-benchmarking-without-lies/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/39-benchmarking-without-lies/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/39-benchmarking-without-lies/rust/rust-toolchain.toml b/examples/39-benchmarking-without-lies/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/39-benchmarking-without-lies/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/39-benchmarking-without-lies/rust/src/main.rs b/examples/39-benchmarking-without-lies/rust/src/main.rs new file mode 100644 index 0000000..a8cafb0 --- /dev/null +++ b/examples/39-benchmarking-without-lies/rust/src/main.rs @@ -0,0 +1,308 @@ +// benchlab (Rust) — proper benchmarking of the chatterd frame codec (ch21). +// +// Usage: +// app --op encode|decode|roundtrip --iters N [--warmup W] # the real thing +// app --lie [--op encode|decode|roundtrip] # the anti-pattern +// +// The "real thing" times each iteration individually with a monotonic clock, +// discards a warmup phase, and reports min/median/p99/max plus a +// coordinated-omission-corrected p99 (see co_correct below). `--lie` times a +// single unwarmed call with wall-clock and reports one number — exactly the +// benchmark the rest of this example exists to discredit. +// +// Wire format (canonical chatterd chat frame, introduced ch21): +// [ magic 0x43 0x48 ][ version 0x01 ][ type u8 ][ length u16 be ][ payload ] +// This file re-implements only encode_frame/decode_frame — the codec under +// test — not the daemon; see ch21/22/27 for the networked version. + +use std::process::ExitCode; +use std::time::Instant; + +// --------------------------------------------------------------------------- +// Frame codec under test — canonical chatterd chat frame (see file header). +// --------------------------------------------------------------------------- +const HEADER_LEN: usize = 6; +const MAGIC0: u8 = 0x43; +const MAGIC1: u8 = 0x48; +const VERSION: u8 = 0x01; +const TYPE_DELIVER: u8 = 3; + +fn encode_frame(typ: u8, body: &[u8]) -> Vec { + let len = body.len() as u16; + let mut out = Vec::with_capacity(HEADER_LEN + body.len()); + out.push(MAGIC0); + out.push(MAGIC1); + out.push(VERSION); + out.push(typ); + out.extend_from_slice(&len.to_be_bytes()); + out.extend_from_slice(body); + out +} + +// Decodes exactly one frame that fills `buf` completely — no partial-read +// reassembly here, that lives in the networked chatterd (ch21/22/27); this +// harness only measures the codec's per-call cost. +fn decode_frame(buf: &[u8]) -> Result<(u8, Vec), String> { + if buf.len() < HEADER_LEN { + return Err("frame shorter than header".to_string()); + } + if buf[0] != MAGIC0 || buf[1] != MAGIC1 { + return Err("bad magic".to_string()); + } + if buf[2] != VERSION { + return Err("bad version".to_string()); + } + let typ = buf[3]; + let len = u16::from_be_bytes([buf[4], buf[5]]) as usize; + if buf.len() != HEADER_LEN + len { + return Err("length mismatch".to_string()); + } + Ok((typ, buf[HEADER_LEN..].to_vec())) +} + +// --------------------------------------------------------------------------- +// The fixed workload: a DELIVER frame carrying a realistic chat line. Every +// language in this example encodes/decodes the identical bytes. +// --------------------------------------------------------------------------- +const NICK: &str = "alice"; +const TEXT: &str = "the quick brown fox jumps over the lazy dog, three times, for benchlab"; + +fn delivery_body() -> Vec { + let mut body = Vec::with_capacity(NICK.len() + 1 + TEXT.len()); + body.extend_from_slice(NICK.as_bytes()); + body.push(0); + body.extend_from_slice(TEXT.as_bytes()); + body +} + +#[derive(Clone, Copy)] +enum Op { + Encode, + Decode, + Roundtrip, +} + +impl Op { + fn parse(s: &str) -> Result { + match s { + "encode" => Ok(Op::Encode), + "decode" => Ok(Op::Decode), + "roundtrip" => Ok(Op::Roundtrip), + other => Err(format!("unknown --op '{other}'")), + } + } + + fn name(self) -> &'static str { + match self { + Op::Encode => "encode", + Op::Decode => "decode", + Op::Roundtrip => "roundtrip", + } + } +} + +// One iteration of the requested op against the fixed workload. Returns a +// small scalar derived from the result so the caller can fold it into a +// checksum — this is what keeps the optimizer from proving the call's result +// is unused and deleting it. Propagates a codec bug (should never trigger on +// this fixed workload) via `?` up to main. +fn run_once(op: Op, body: &[u8], prebuilt_frame: &[u8]) -> Result { + match op { + Op::Encode => { + let frame = encode_frame(TYPE_DELIVER, body); + Ok(*frame.last().unwrap_or(&0) as u64) + } + Op::Decode => { + let (_typ, payload) = decode_frame(prebuilt_frame)?; + Ok(payload.len() as u64) + } + Op::Roundtrip => { + let frame = encode_frame(TYPE_DELIVER, body); + let (_typ, payload) = decode_frame(&frame)?; + Ok(payload.len() as u64) + } + } +} + +fn percentile(sorted: &[i64], p: f64) -> i64 { + if sorted.is_empty() { + return 0; + } + let idx = (p * (sorted.len() - 1) as f64) as usize; + sorted[idx] +} + +// Coordinated-omission correction (HdrHistogram's recordValueWithExpectedInterval): +// this harness is a closed loop — iteration N+1 never starts until N returns — +// so a stall (a page fault, a context switch, a scheduling hiccup) is +// recorded as a single slow sample. A fixed-rate caller would instead have +// had many requests queue up behind that stall, each experiencing a similar +// multiple-of-the-target-interval delay. This backfills those missing +// virtual samples so the tail reflects what a real caller would have seen, +// not just what one lucky/unlucky iteration measured. +fn co_correct(raw: &[i64], expected_interval_ns: i64, cap: usize) -> Vec { + let mut out = Vec::with_capacity(raw.len()); + 'outer: for &v in raw { + out.push(v); + if expected_interval_ns <= 0 || v <= expected_interval_ns { + continue; + } + let mut missing = v - expected_interval_ns; + while missing >= expected_interval_ns { + if out.len() >= cap { + break 'outer; + } + out.push(missing); + missing -= expected_interval_ns; + } + } + out +} + +fn usage() -> ! { + eprintln!("usage: benchlab --op encode|decode|roundtrip --iters N [--warmup W]"); + eprintln!(" benchlab --lie [--op encode|decode|roundtrip]"); + std::process::exit(2); +} + +fn parse_u64(s: &str) -> Option { + if s.is_empty() || !s.bytes().all(|b| b.is_ascii_digit()) { + return None; + } + s.parse::().ok() +} + +struct Args { + lie: bool, + op: Option, + iters: i64, + warmup: i64, +} + +fn parse_args() -> Args { + let argv: Vec = std::env::args().collect(); + let mut lie = false; + let mut op_str: Option = None; + let mut iters: i64 = -1; + let mut warmup: i64 = 1000; + + let mut i = 1; + while i < argv.len() { + match argv[i].as_str() { + "--lie" => lie = true, + "--op" if i + 1 < argv.len() => { + op_str = Some(argv[i + 1].clone()); + i += 1; + } + "--iters" if i + 1 < argv.len() => { + iters = parse_u64(&argv[i + 1]).unwrap_or_else(|| usage()); + i += 1; + } + "--warmup" if i + 1 < argv.len() => { + warmup = parse_u64(&argv[i + 1]).unwrap_or_else(|| usage()); + i += 1; + } + _ => usage(), + } + i += 1; + } + + if !lie && op_str.is_none() { + usage(); + } + let op = op_str.map(|s| Op::parse(&s).unwrap_or_else(|_| usage())); + + Args { lie, op, iters, warmup } +} + +fn main() -> ExitCode { + let args = parse_args(); + let op = args.op.unwrap_or(Op::Roundtrip); + let body = delivery_body(); + let prebuilt = encode_frame(TYPE_DELIVER, &body); + + if args.lie { + // The anti-pattern: no warmup, one call, wall-clock, done. This is + // exactly the benchmark this example's README argues never to trust. + let t0 = Instant::now(); + let sink = match run_once(op, &body, &prebuilt) { + Ok(v) => v, + Err(e) => { + eprintln!("benchlab: codec bug: {e}"); + return ExitCode::from(1); + } + }; + let elapsed_ns = t0.elapsed().as_nanos(); + println!( + "benchlab: lie op={} (no warmup, single wall-clock sample, ignores variance)", + op.name() + ); + println!("benchlab: lie elapsed_ns={elapsed_ns} sink={sink}"); + return ExitCode::SUCCESS; + } + + if args.iters < 2 || args.warmup < 0 || args.iters <= args.warmup { + usage(); + } + let iters = args.iters; + let warmup = args.warmup; + + // Warmup: run the op without timing it, letting allocators/caches/branch + // predictors reach steady state before any sample is recorded. + let mut checksum: u64 = 0; + for _ in 0..warmup { + match run_once(op, &body, &prebuilt) { + Ok(v) => checksum = checksum.wrapping_add(v), + Err(e) => { + eprintln!("benchlab: codec bug: {e}"); + return ExitCode::from(1); + } + } + } + + let mut raw: Vec = Vec::with_capacity(iters as usize); + for _ in 0..iters { + let t0 = Instant::now(); + let r = run_once(op, &body, &prebuilt); + let elapsed_ns = t0.elapsed().as_nanos() as i64; + match r { + Ok(v) => checksum = checksum.wrapping_add(v), + Err(e) => { + eprintln!("benchlab: codec bug: {e}"); + return ExitCode::from(1); + } + } + raw.push(elapsed_ns); + } + + let mut sorted = raw.clone(); + sorted.sort_unstable(); + let min_ns = sorted[0]; + let median_ns = percentile(&sorted, 0.5); + let p99_ns = percentile(&sorted, 0.99); + let max_ns = *sorted.last().unwrap(); + + let expected_interval_ns = if median_ns > 0 { median_ns } else { 1 }; + const CO_CAP: usize = 5_000_000; + let mut corrected = co_correct(&raw, expected_interval_ns, CO_CAP); + corrected.sort_unstable(); + let co_p99_ns = percentile(&corrected, 0.99); + + println!("benchlab: op={} iters={} warmup={}", op.name(), iters, warmup); + println!( + "benchlab: n={} min_ns={} median_ns={} p99_ns={} max_ns={}", + sorted.len(), + min_ns, + median_ns, + p99_ns, + max_ns + ); + println!( + "benchlab: co_p99_ns={} expected_interval_ns={} co_n={}", + co_p99_ns, + expected_interval_ns, + corrected.len() + ); + println!("benchlab: checksum={checksum:016x}"); + ExitCode::SUCCESS +} diff --git a/examples/39-benchmarking-without-lies/verify.lua b/examples/39-benchmarking-without-lies/verify.lua new file mode 100644 index 0000000..f0076b8 --- /dev/null +++ b/examples/39-benchmarking-without-lies/verify.lua @@ -0,0 +1,144 @@ +-- Verify benchlab (39-benchmarking-without-lies) for LSP_LANG. +-- +-- Asserts observable behavior, identical in shape across cpp/go/rust: +-- 1. for each op (encode/decode/roundtrip), a real run (--iters/--warmup) +-- emits a well-formed percentile table with sane monotonic ordering +-- (min <= median <= p99 <= max), a coordinated-omission-corrected p99 +-- that is never below the raw p99, n == the requested iters, and a +-- checksum proving the codec actually ran (not skipped/optimized away) +-- 2. the same fixed workload run twice produces the same checksum both +-- times (the codec's output is deterministic even though the ns +-- timings are not) — the "repeatable-order" contract +-- 3. --lie runs, exits 0, and its output contains no percentile table at +-- all — that absence is the whole point of the contrast +-- 4. error shapes: no flags, an unknown --op, and --warmup >= --iters all +-- exit 2 with a usage line +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local demo = "./demo.sh " .. lang .. " run" + +-- Boolean assertion via the checks.lua pattern-match API: match "ok" against +-- "ok" when true, or force a mismatch (with the real detail in the label) +-- when false, so failures show up the same way expect_match failures do. +local function expect_true(cond, label, detail) + if cond then + checks.expect_match("ok", "ok", label) + else + local msg = label + if detail ~= nil then + msg = msg .. " (" .. tostring(detail) .. ")" + end + checks.expect_match("no", "yes", msg) + end +end + +local function parse_bench(out) + local op, iters, warmup = out:match("benchlab: op=(%a+) iters=(%d+) warmup=(%d+)") + local n, min_ns, median_ns, p99_ns, max_ns = out:match( + "benchlab: n=(%d+) min_ns=(%d+) median_ns=(%d+) p99_ns=(%d+) max_ns=(%d+)") + local co_p99_ns, expected_interval_ns, co_n = out:match( + "benchlab: co_p99_ns=(%d+) expected_interval_ns=(%d+) co_n=(%d+)") + local checksum = out:match("benchlab: checksum=(%x+)") + return { + op = op, + iters = tonumber(iters), warmup = tonumber(warmup), + n = tonumber(n), min_ns = tonumber(min_ns), median_ns = tonumber(median_ns), + p99_ns = tonumber(p99_ns), max_ns = tonumber(max_ns), + co_p99_ns = tonumber(co_p99_ns), expected_interval_ns = tonumber(expected_interval_ns), + co_n = tonumber(co_n), + checksum = checksum, + } +end + +local ITERS = 20000 +local WARMUP = 1000 + +-- --------------------------------------------------------------------------- +-- 1 & 2. Each op: shape, monotonicity, and repeatable checksum across two runs. +-- --------------------------------------------------------------------------- + +for _, op in ipairs({ "encode", "decode", "roundtrip" }) do + local cmd = demo .. " --op " .. op .. " --iters " .. ITERS .. " --warmup " .. WARMUP + local tag = lang .. " " .. op + + local r1 = checks.run(cmd) + checks.expect_exit(r1, 0, tag .. ": exits 0") + checks.expect_match(r1.out, + "benchlab: op=" .. op .. " iters=" .. ITERS .. " warmup=" .. WARMUP, + tag .. ": header line echoes op/iters/warmup") + checks.expect_match(r1.out, "benchlab: checksum=%x%x%x%x%x%x%x%x%x%x%x%x%x%x%x%x", + tag .. ": prints a 64-bit hex checksum (proves the codec actually ran)") + + local f1 = parse_bench(r1.out) + expect_true(f1.n == ITERS, tag .. ": n equals the requested iters", f1.n) + expect_true(f1.min_ns ~= nil and f1.median_ns ~= nil and f1.p99_ns ~= nil and f1.max_ns ~= nil, + tag .. ": min/median/p99/max all parse as numbers") + if f1.min_ns then + expect_true(f1.min_ns <= f1.median_ns and f1.median_ns <= f1.p99_ns and f1.p99_ns <= f1.max_ns, + tag .. ": percentiles are monotonic (min <= median <= p99 <= max)", + string.format("min=%d median=%d p99=%d max=%d", f1.min_ns, f1.median_ns, f1.p99_ns, f1.max_ns)) + expect_true(f1.co_p99_ns >= f1.p99_ns, + tag .. ": coordinated-omission-corrected p99 is never below the raw p99", + string.format("co_p99=%d p99=%d", f1.co_p99_ns, f1.p99_ns)) + expect_true(f1.co_n >= f1.n, + tag .. ": the CO-corrected sample count is at least the raw sample count", + string.format("co_n=%d n=%d", f1.co_n, f1.n)) + local want_interval = f1.median_ns > 0 and f1.median_ns or 1 + expect_true(f1.expected_interval_ns == want_interval, + tag .. ": expected_interval_ns matches the documented derivation (raw median, floor 1)", + string.format("expected_interval_ns=%d median_ns=%d", f1.expected_interval_ns, f1.median_ns)) + end + + -- Repeatable-order: a second run of the identical fixed workload must + -- produce the identical checksum (deterministic codec output), even though + -- the ns timings themselves are expected to vary run to run. + local r2 = checks.run(cmd) + checks.expect_exit(r2, 0, tag .. ": second run also exits 0") + local f2 = parse_bench(r2.out) + expect_true(f1.checksum ~= nil and f1.checksum == f2.checksum, + tag .. ": checksum is identical across two runs of the same fixed workload", + tostring(f1.checksum) .. " vs " .. tostring(f2.checksum)) +end + +-- --------------------------------------------------------------------------- +-- 3. --lie: runs, exits 0, and — the entire point — never emits a percentile +-- table, only a single untrusted wall-clock sample. +-- --------------------------------------------------------------------------- + +local lie = checks.run(demo .. " --lie --op roundtrip") +checks.expect_exit(lie, 0, lang .. ": --lie exits 0") +checks.expect_match(lie.out, "benchlab: lie op=roundtrip", + lang .. ": --lie header names itself as the lie") +checks.expect_match(lie.out, "benchlab: lie elapsed_ns=%d+ sink=%d+", + lang .. ": --lie reports exactly one elapsed_ns sample") +expect_true(not lie.out:match("p99_ns="), + lang .. ": --lie output has no percentile table at all") +expect_true(not lie.out:match("median_ns="), + lang .. ": --lie output has no median either — one sample, no warmup, no variance") + +-- --------------------------------------------------------------------------- +-- 4. Error shapes: usage, exit 2. +-- --------------------------------------------------------------------------- + +local noargs = checks.run(demo) +checks.expect_exit(noargs, 2, lang .. ": no flags exits 2") +checks.expect_match(noargs.out, "usage: benchlab", lang .. ": usage line is printed") + +local badop = checks.run(demo .. " --op bogus --iters 1000") +checks.expect_exit(badop, 2, lang .. ": unknown --op exits 2") + +local badrange = checks.run(demo .. " --op encode --iters 100 --warmup 500") +checks.expect_exit(badrange, 2, lang .. ": --warmup >= --iters exits 2") + +checks.finish() diff --git a/examples/40-low-latency-fastpath/README.md b/examples/40-low-latency-fastpath/README.md new file mode 100644 index 0000000..39b523c --- /dev/null +++ b/examples/40-low-latency-fastpath/README.md @@ -0,0 +1,51 @@ +# _template — hello-syscall + +The seed for every runnable example in the book. It implements the same tiny +program three times — C++23, Go, and Rust — each printing exactly one line: + +``` +pid on () +``` + +via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error +handling (`std::expected` / wrapped errors via `%w` / `rustix`). + +## Layout + +``` +_template/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build, demo.sh +├── go/ # go build, demo.sh +└── rust/ # cargo build, demo.sh +``` + +## The demo contract + +Every language directory has a `demo.sh` with the same interface, identical +across the whole book: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of + running locally. + +The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs +`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. + +## Verification + +`verify.lua` is run from this directory with `LSP_LANG` set to one language at +a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and +asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, +77 = skip. + +## Mapping to a chapter + +Each chapter's example is a copy of this template (`scripts/new-example.sh +NN-slug`) with the program body replaced by that chapter's code. Keep the +demo contract and `verify.lua` shape intact so CI and the manifest +(`examples/manifest.yaml`) keep working. diff --git a/examples/40-low-latency-fastpath/cpp/.clang-format b/examples/40-low-latency-fastpath/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/40-low-latency-fastpath/cpp/.clang-tidy b/examples/40-low-latency-fastpath/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/40-low-latency-fastpath/cpp/CMakeLists.txt b/examples/40-low-latency-fastpath/cpp/CMakeLists.txt new file mode 100644 index 0000000..0659a26 --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(chatterd_fastpath CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/40-low-latency-fastpath/cpp/CMakePresets.json b/examples/40-low-latency-fastpath/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/40-low-latency-fastpath/cpp/demo.sh b/examples/40-low-latency-fastpath/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/40-low-latency-fastpath/cpp/src/main.cpp b/examples/40-low-latency-fastpath/cpp/src/main.cpp new file mode 100644 index 0000000..7e76796 --- /dev/null +++ b/examples/40-low-latency-fastpath/cpp/src/main.cpp @@ -0,0 +1,577 @@ +// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd echo/pub-sub hot path (chapter 21) built purely to answer one +// question: what does a low-latency network hot path actually cost, and what +// does removing that cost look like at the syscall level? +// +// Three subcommands, one 64-byte wire frame: +// +// app fastpath --port P --pin CPU [--busy-poll] +// Low-latency echo server. sched_setaffinity(2) pins this (single, +// unthreaded) process to CPU; one stack-allocated 64-byte buffer is +// reused for every message (zero heap allocation in the loop); +// --busy-poll spins a non-blocking recv(2) instead of blocking read(2), +// trading 100% of the pinned core for the removal of the sleep/wake +// round trip through the scheduler. +// +// app naive --port P +// Ordinary echo server: default scheduling (no affinity), a fresh heap +// buffer allocated per message, blocking read(2). The baseline chatterd +// hot loop would do exactly this. +// +// app measure --target HOST:PORT --n N [--warmup W] +// Connect, then run N synchronous ping-pong round trips (send frame, +// block for the echo, repeat — never more than one frame in flight) and +// print a p50/p90/p99/p99.9 latency table in nanoseconds. W warmup +// round trips are discarded before recording starts (they pay for TCP +// slow-start / first-touch page faults / connection setup, none of +// which are what this is measuring). +// +// THE FASTPATH FRAME (fixed 64 bytes; this example's own wire format, a +// deliberately smaller cousin of the chatterd CHAT FRAME from ch21 — no +// variable-length payload, so nothing here ever needs to size a buffer at +// runtime): +// +// +-------+---------+------+-----------+-----------------+---------+ +// | magic | version | type | seq (u64) | send_ns (u64) | pad | +// | 2B | 1B | 1B | 8B BE | 8B BE | 44B | +// +-------+---------+------+-----------+-----------------+---------+ +// +// magic is "CF" (chatterd-fastpath); the server never inspects seq/send_ns — +// it reads 64 bytes and writes the identical 64 bytes back. All timing is +// done by the client against its own clock; seq/send_ns are echoed back only +// so the client can confirm it got its own frame back rather than another +// client's or garbage (a correctness check, not a timing mechanism). +// +// C++23: RAII Socket owns every fd; std::expected carries fallible setup +// steps; std::atomic + a non-restarting SIGINT/SIGTERM handler unblock +// the accept wait and the read in progress, mirroring chatterd's +// stop_callback-via-shutdown(2) pattern without needing a second thread — the +// whole point of this example is that the hot loop runs on exactly one +// thread, so a hand-off to a jthread is deliberately not in the loop. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr std::size_t kFrameSize = 64; +constexpr std::uint8_t kMagic0 = 0x43; // 'C' +constexpr std::uint8_t kMagic1 = 0x46; // 'F' +constexpr std::uint8_t kWireVersion = 0x01; +constexpr std::uint8_t kTypeEcho = 0x01; + +using Frame = std::array; + +[[nodiscard]] std::string errno_text() { + return std::error_code{errno, std::system_category()}.message(); +} + +template +using Result = std::expected; + +void put_u64be(std::uint8_t* p, std::uint64_t v) { + for (int i = 7; i >= 0; --i) { + p[7 - i] = static_cast((v >> (i * 8)) & 0xff); + } +} + +std::uint64_t get_u64be(const std::uint8_t* p) { + std::uint64_t v = 0; + for (int i = 0; i < 8; ++i) { + v = (v << 8) | p[i]; + } + return v; +} + +Frame build_frame(std::uint64_t seq, std::uint64_t send_ns) { + Frame f{}; + f[0] = kMagic0; + f[1] = kMagic1; + f[2] = kWireVersion; + f[3] = kTypeEcho; + put_u64be(f.data() + 4, seq); + put_u64be(f.data() + 12, send_ns); + // bytes 20..63 stay zero (aggregate init already zeroed them) + return f; +} + +bool frame_header_ok(const Frame& f) { + return f[0] == kMagic0 && f[1] == kMagic1 && f[2] == kWireVersion && f[3] == kTypeEcho; +} + +std::uint64_t now_ns() { + timespec ts{}; + ::clock_gettime(CLOCK_MONOTONIC, &ts); + return static_cast(ts.tv_sec) * 1'000'000'000ull + + static_cast(ts.tv_nsec); +} + +// --------------------------------------------------------------------------- +// RAII Socket (same shape as chapter 21's). +// --------------------------------------------------------------------------- + +class Socket { +public: + Socket() = default; + explicit Socket(int fd) : fd_{fd} {} + Socket(const Socket&) = delete; + Socket& operator=(const Socket&) = delete; + Socket(Socket&& o) noexcept : fd_{std::exchange(o.fd_, -1)} {} + Socket& operator=(Socket&& o) noexcept { + if (this != &o) { + reset(); + fd_ = std::exchange(o.fd_, -1); + } + return *this; + } + ~Socket() { reset(); } + + void reset() noexcept { + if (fd_ >= 0) { + ::close(fd_); + } + fd_ = -1; + } + [[nodiscard]] int get() const noexcept { return fd_; } + [[nodiscard]] bool valid() const noexcept { return fd_ >= 0; } + +private: + int fd_ = -1; +}; + +Result listen_tcp(std::uint16_t port) { + Socket sock{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!sock.valid()) { + return std::unexpected(std::format("socket: {}", errno_text())); + } + int one = 1; + ::setsockopt(sock.get(), SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + addr.sin_addr.s_addr = INADDR_ANY; + if (::bind(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected(std::format("bind 0.0.0.0:{}: {}", port, errno_text())); + } + if (::listen(sock.get(), 16) < 0) { + return std::unexpected(std::format("listen: {}", errno_text())); + } + return sock; +} + +Result connect_tcp(const std::string& host, std::uint16_t port) { + Socket sock{::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)}; + if (!sock.valid()) { + return std::unexpected(std::format("socket: {}", errno_text())); + } + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(port); + if (::inet_pton(AF_INET, host.c_str(), &addr.sin_addr) != 1) { + return std::unexpected(std::format("{}: not an IPv4 address", host)); + } + if (::connect(sock.get(), reinterpret_cast(&addr), sizeof(addr)) < 0) { + return std::unexpected(std::format("connect {}:{}: {}", host, port, errno_text())); + } + return sock; +} + +// Blocking read of exactly n bytes; false on EOF, hard error, or a signal +// landing (EINTR) — the caller decides whether that means "stop". +bool read_full_blocking(int fd, void* buf, std::size_t n) { + auto* p = static_cast(buf); + std::size_t off = 0; + while (off < n) { + ssize_t r = ::read(fd, p + off, n - off); + if (r < 0) { + return false; + } + if (r == 0) { + return false; // peer closed + } + off += static_cast(r); + } + return true; +} + +// Non-blocking spin read: the socket must already be O_NONBLOCK. Spins on +// EAGAIN with no syscall other than recv(2) itself — no sched_yield, no +// nanosleep — which is the entire point of "busy-poll". +std::atomic* g_stop_for_busy_read = nullptr; + +bool read_full_busy(int fd, void* buf, std::size_t n) { + auto* p = static_cast(buf); + std::size_t off = 0; + while (off < n) { + ssize_t r = ::recv(fd, p + off, n - off, 0); + if (r < 0) { + if (errno == EAGAIN || errno == EWOULDBLOCK) { + if (g_stop_for_busy_read && g_stop_for_busy_read->load(std::memory_order_relaxed)) { + return false; + } + continue; // spin + } + return false; + } + if (r == 0) { + return false; // peer closed + } + off += static_cast(r); + } + return true; +} + +void write_all(int fd, const void* buf, std::size_t n) { + const auto* p = static_cast(buf); + std::size_t off = 0; + while (off < n) { + ssize_t w = ::write(fd, p + off, n - off); + if (w < 0) { + if (errno == EINTR) { + continue; + } + return; // peer gone; nothing sensible to do + } + off += static_cast(w); + } +} + +// --------------------------------------------------------------------------- +// Shutdown plumbing: a plain (non-SA_RESTART) signal handler so a blocking +// read(2)/accept-wait poll(2) in progress is interrupted with EINTR rather +// than transparently resumed. +// --------------------------------------------------------------------------- + +std::atomic g_stop{false}; + +extern "C" void on_signal(int) { g_stop.store(true, std::memory_order_relaxed); } + +void install_signal_handlers() { + struct sigaction sa{}; + sa.sa_handler = on_signal; + sa.sa_flags = 0; // deliberately NOT SA_RESTART + ::sigemptyset(&sa.sa_mask); + ::sigaction(SIGINT, &sa, nullptr); + ::sigaction(SIGTERM, &sa, nullptr); +} + +Result pin_to_cpu(int cpu) { + long nproc = ::sysconf(_SC_NPROCESSORS_ONLN); + if (cpu < 0 || (nproc > 0 && cpu >= nproc)) { + return std::unexpected( + std::format("cpu {} out of range (0..{})", cpu, nproc > 0 ? nproc - 1 : 0)); + } + cpu_set_t set; + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (::sched_setaffinity(0, sizeof(set), &set) != 0) { + return std::unexpected(std::format("sched_setaffinity({}): {}", cpu, errno_text())); + } + return {}; +} + +// --------------------------------------------------------------------------- +// fastpath / naive servers +// --------------------------------------------------------------------------- + +int cmd_fastpath(std::uint16_t port, int pin_cpu, bool busy_poll) { + if (auto pinned = pin_to_cpu(pin_cpu); !pinned) { + std::println(stderr, "app: error: {}", pinned.error()); + return 1; + } + auto lsock = listen_tcp(port); + if (!lsock) { + std::println(stderr, "app: error: {}", lsock.error()); + return 1; + } + install_signal_handlers(); + g_stop_for_busy_read = &g_stop; + std::println(stderr, "app: fastpath listening on 0.0.0.0:{} pinned-cpu={} busy-poll={}", + port, pin_cpu, busy_poll ? "on" : "off"); + + while (!g_stop.load(std::memory_order_relaxed)) { + pollfd pfd{lsock->get(), POLLIN, 0}; + int pr = ::poll(&pfd, 1, 200); + if (pr < 0) { + if (errno == EINTR) continue; + std::println(stderr, "app: error: poll: {}", errno_text()); + return 1; + } + if (pr == 0 || !(pfd.revents & POLLIN)) { + continue; + } + Socket conn{::accept4(lsock->get(), nullptr, nullptr, SOCK_CLOEXEC)}; + if (!conn.valid()) { + if (errno == EINTR || errno == EAGAIN) continue; + std::println(stderr, "app: error: accept: {}", errno_text()); + return 1; + } + int one = 1; + ::setsockopt(conn.get(), IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); + if (busy_poll) { + int flags = ::fcntl(conn.get(), F_GETFL, 0); + ::fcntl(conn.get(), F_SETFL, flags | O_NONBLOCK); + } + + // The ONE buffer for the whole life of this connection: no malloc, + // no new, no make_unique anywhere in the loop below. + Frame buf{}; + for (;;) { + bool ok = busy_poll ? read_full_busy(conn.get(), buf.data(), kFrameSize) + : read_full_blocking(conn.get(), buf.data(), kFrameSize); + if (!ok) break; + write_all(conn.get(), buf.data(), kFrameSize); + } + } + std::println(stderr, "app: fastpath shutting down"); + return 0; +} + +int cmd_naive(std::uint16_t port) { + auto lsock = listen_tcp(port); + if (!lsock) { + std::println(stderr, "app: error: {}", lsock.error()); + return 1; + } + install_signal_handlers(); + std::println(stderr, "app: naive listening on 0.0.0.0:{}", port); + + while (!g_stop.load(std::memory_order_relaxed)) { + pollfd pfd{lsock->get(), POLLIN, 0}; + int pr = ::poll(&pfd, 1, 200); + if (pr < 0) { + if (errno == EINTR) continue; + std::println(stderr, "app: error: poll: {}", errno_text()); + return 1; + } + if (pr == 0 || !(pfd.revents & POLLIN)) { + continue; + } + Socket conn{::accept4(lsock->get(), nullptr, nullptr, SOCK_CLOEXEC)}; + if (!conn.valid()) { + if (errno == EINTR || errno == EAGAIN) continue; + std::println(stderr, "app: error: accept: {}", errno_text()); + return 1; + } + for (;;) { + // The "naive" part: a fresh heap buffer, every single message. + auto heap_buf = std::make_unique(kFrameSize); + if (!read_full_blocking(conn.get(), heap_buf.get(), kFrameSize)) break; + write_all(conn.get(), heap_buf.get(), kFrameSize); + // heap_buf frees here, every iteration. + } + } + std::println(stderr, "app: naive shutting down"); + return 0; +} + +// --------------------------------------------------------------------------- +// measure +// --------------------------------------------------------------------------- + +std::uint64_t percentile_ns(const std::vector& sorted, double p) { + if (sorted.empty()) return 0; + std::size_t n = sorted.size(); + std::size_t idx = static_cast(std::ceil(p / 100.0 * static_cast(n))); + if (idx == 0) idx = 1; + if (idx > n) idx = n; + return sorted[idx - 1]; +} + +int cmd_measure(const std::string& host, std::uint16_t port, std::uint64_t n, std::uint64_t warmup, + const std::string& tag) { + auto conn = connect_tcp(host, port); + if (!conn) { + std::println(stderr, "app: error: {}", conn.error()); + return 1; + } + int one = 1; + ::setsockopt(conn->get(), IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); + + std::println("app: measure target={}:{} n={} warmup={}", host, port, n, warmup); + + std::vector samples; + samples.reserve(n); + const std::uint64_t total = n + warmup; + for (std::uint64_t i = 0; i < total; ++i) { + const Frame out = build_frame(i, 0); + const std::uint64_t t_send = now_ns(); + write_all(conn->get(), out.data(), kFrameSize); + Frame in{}; + if (!read_full_blocking(conn->get(), in.data(), kFrameSize)) { + std::println(stderr, "app: error: measure: connection closed at iteration {}", i); + return 1; + } + const std::uint64_t t_recv = now_ns(); + if (!frame_header_ok(in) || get_u64be(in.data() + 4) != i) { + std::println(stderr, "app: error: measure: malformed/mismatched echo at iteration {}", + i); + return 1; + } + if (i >= warmup) { + samples.push_back(t_recv - t_send); + } + } + + std::ranges::sort(samples); + const std::uint64_t p50 = percentile_ns(samples, 50.0); + const std::uint64_t p90 = percentile_ns(samples, 90.0); + const std::uint64_t p99 = percentile_ns(samples, 99.0); + const std::uint64_t p999 = percentile_ns(samples, 99.9); + const std::uint64_t mn = samples.front(); + const std::uint64_t mx = samples.back(); + double sum = 0; + for (auto v : samples) sum += static_cast(v); + const double mean = sum / static_cast(samples.size()); + + std::println( + "app: percentiles_ns tag={} p50={} p90={} p99={} p99.9={} min={} max={} mean={:.2f} n={}", + tag.empty() ? "-" : tag, p50, p90, p99, p999, mn, mx, mean, samples.size()); + std::println("app: table"); + std::println(" p50 {} ns", p50); + std::println(" p90 {} ns", p90); + std::println(" p99 {} ns", p99); + std::println(" p99.9 {} ns", p999); + std::println(" min {} ns", mn); + std::println(" max {} ns", mx); + std::println(" mean {:.2f} ns", mean); + return 0; +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +int usage() { + std::println(stderr, "usage: app fastpath --port P --pin CPU [--busy-poll]"); + std::println(stderr, " app naive --port P"); + std::println(stderr, " app measure --target HOST:PORT --n N [--warmup W] [--tag TAG]"); + return 2; +} + +std::optional parse_port(std::string_view s) { + std::uint64_t v = 0; + auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || ptr != s.data() + s.size() || v == 0 || v > 65535) { + return std::nullopt; + } + return static_cast(v); +} + +std::optional parse_u64(std::string_view s) { + std::uint64_t v = 0; + auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || ptr != s.data() + s.size()) { + return std::nullopt; + } + return v; +} + +} // namespace + +int main(int argc, char** argv) { + std::vector args(argv + 1, argv + argc); + if (args.empty()) { + return usage(); + } + const std::string sub = args[0]; + + if (sub == "fastpath") { + std::string port_str; + int pin_cpu = -1; + bool busy_poll = false; + for (std::size_t i = 1; i < args.size(); ++i) { + if (args[i] == "--port" && i + 1 < args.size()) { + port_str = args[++i]; + } else if (args[i] == "--pin" && i + 1 < args.size()) { + auto v = parse_u64(args[++i]); + if (!v) return usage(); + pin_cpu = static_cast(*v); + } else if (args[i] == "--busy-poll") { + busy_poll = true; + } else { + return usage(); + } + } + auto port = parse_port(port_str); + if (!port || pin_cpu < 0) return usage(); + return cmd_fastpath(*port, pin_cpu, busy_poll); + } + + if (sub == "naive") { + std::string port_str; + for (std::size_t i = 1; i < args.size(); ++i) { + if (args[i] == "--port" && i + 1 < args.size()) { + port_str = args[++i]; + } else { + return usage(); + } + } + auto port = parse_port(port_str); + if (!port) return usage(); + return cmd_naive(*port); + } + + if (sub == "measure") { + std::string target; + std::uint64_t n = 0; + std::uint64_t warmup = 200; + std::string tag; + bool have_n = false; + for (std::size_t i = 1; i < args.size(); ++i) { + if (args[i] == "--target" && i + 1 < args.size()) { + target = args[++i]; + } else if (args[i] == "--n" && i + 1 < args.size()) { + auto v = parse_u64(args[++i]); + if (!v) return usage(); + n = *v; + have_n = true; + } else if (args[i] == "--warmup" && i + 1 < args.size()) { + auto v = parse_u64(args[++i]); + if (!v) return usage(); + warmup = *v; + } else if (args[i] == "--tag" && i + 1 < args.size()) { + tag = args[++i]; + } else { + return usage(); + } + } + auto colon = target.rfind(':'); + if (!have_n || n == 0 || colon == std::string::npos) return usage(); + auto port = parse_port(std::string_view(target).substr(colon + 1)); + if (!port) return usage(); + std::string host = target.substr(0, colon); + return cmd_measure(host, *port, n, warmup, tag); + } + + return usage(); +} diff --git a/examples/40-low-latency-fastpath/demo.sh b/examples/40-low-latency-fastpath/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/40-low-latency-fastpath/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/40-low-latency-fastpath/go/demo.sh b/examples/40-low-latency-fastpath/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/40-low-latency-fastpath/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/40-low-latency-fastpath/go/go.mod b/examples/40-low-latency-fastpath/go/go.mod new file mode 100644 index 0000000..b3e973b --- /dev/null +++ b/examples/40-low-latency-fastpath/go/go.mod @@ -0,0 +1,7 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/40-low-latency-fastpath/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 diff --git a/examples/40-low-latency-fastpath/go/go.sum b/examples/40-low-latency-fastpath/go/go.sum new file mode 100644 index 0000000..37ee2d4 --- /dev/null +++ b/examples/40-low-latency-fastpath/go/go.sum @@ -0,0 +1,2 @@ +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= diff --git a/examples/40-low-latency-fastpath/go/main.go b/examples/40-low-latency-fastpath/go/main.go new file mode 100644 index 0000000..3440284 --- /dev/null +++ b/examples/40-low-latency-fastpath/go/main.go @@ -0,0 +1,699 @@ +// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd echo/pub-sub hot path (chapter 21) built purely to answer one +// question: what does a low-latency network hot path actually cost, and what +// does removing that cost look like at the syscall level? +// +// Three subcommands, one 64-byte wire frame; observable behavior is identical +// to the C++ and Rust ports (same usage text, same "app: ..." lines, same +// percentiles_ns format), because verify.lua asserts against all three. +// +// app fastpath --port P --pin CPU [--busy-poll] +// app naive --port P +// app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] +// +// THE FASTPATH FRAME (fixed 64 bytes): +// +// +-------+---------+------+-----------+-----------------+---------+ +// | magic | version | type | seq (u64) | send_ns (u64) | pad | +// | 2B | 1B | 1B | 8B BE | 8B BE | 44B | +// +-------+---------+------+-----------+-----------------+---------+ +// +// WHERE THE GO PORT DIVERGES FROM THE C++ ONE, AND WHY: +// +// The C++ version installs a SIGINT handler with sa_flags = 0 — deliberately +// NOT SA_RESTART — so a blocking read(2) or poll(2) already in progress fails +// with EINTR and the loop notices the stop flag. That technique is not +// available here. The Go runtime installs its own signal handlers with +// SA_RESTART, and a "blocking" read on a net.Conn does not sit in a syscall +// at all: the netpoller parks the goroutine on an epoll registration. No +// amount of signal handling will make that read return EINTR. +// +// So Go reaches the same observable behavior by a different mechanism: a +// signal.Notify goroutine sets the stop flag and then CLOSES the listener and +// the live connection. Closing is what unblocks a parked netpoller read — +// the same shutdown(2)-driven wakeup that chapter 21's chatterd uses via +// stop_callback, and the Go-idiomatic answer to "interrupt a blocked I/O +// wait". The accept loop additionally carries a 200ms deadline, mirroring the +// C++ poll(2) timeout, so shutdown stays bounded even if a close races an +// accept already in flight. +// +// The second divergence is pinning. sched_setaffinity(2) acts on a THREAD, +// and a goroutine is not a thread — it migrates between Ms at the scheduler's +// discretion. Pinning here therefore needs runtime.LockOSThread to nail the +// hot loop to one M, plus GOMAXPROCS(1) so the runtime does not keep other +// worker threads on other CPUs. See pinToCPU for why the affinity call is +// made twice. +package main + +import ( + "context" + "encoding/binary" + "errors" + "fmt" + "io" + "math" + "net" + "os" + "os/signal" + "runtime" + "sort" + "strconv" + "strings" + "sync" + "sync/atomic" + "syscall" + "time" + + "golang.org/x/sys/unix" +) + +const ( + frameSize = 64 + magic0 = 0x43 // 'C' + magic1 = 0x46 // 'F' + wireVersion = 0x01 + typeEcho = 0x01 + + // Mirrors the C++ poll(2) timeout on the accept wait. + acceptPollInterval = 200 * time.Millisecond +) + +// --------------------------------------------------------------------------- +// Frame +// --------------------------------------------------------------------------- + +func buildFrame(seq, sendNs uint64) []byte { + f := make([]byte, frameSize) + f[0] = magic0 + f[1] = magic1 + f[2] = wireVersion + f[3] = typeEcho + binary.BigEndian.PutUint64(f[4:12], seq) + binary.BigEndian.PutUint64(f[12:20], sendNs) + // bytes 20..63 stay zero + return f +} + +func frameHeaderOK(f []byte) bool { + return f[0] == magic0 && f[1] == magic1 && f[2] == wireVersion && f[3] == typeEcho +} + +func nowNs() uint64 { + var ts unix.Timespec + _ = unix.ClockGettime(unix.CLOCK_MONOTONIC, &ts) + return uint64(ts.Sec)*1_000_000_000 + uint64(ts.Nsec) +} + +// --------------------------------------------------------------------------- +// Shutdown plumbing +// --------------------------------------------------------------------------- + +// stopper carries the stop flag plus the closers that unblock whatever I/O is +// currently parked. Closing is the Go substitute for the C++ port's EINTR. +type stopper struct { + stop atomic.Bool + + mu sync.Mutex + ln net.Listener + conn net.Conn + // rawFD is the busy-poll path's dup'd descriptor. It is not registered + // with the netpoller, so it is shut down rather than closed out from + // under a spinning read. + rawFD int +} + +func newStopper() *stopper { return &stopper{rawFD: -1} } + +func (s *stopper) setListener(ln net.Listener) { + s.mu.Lock() + defer s.mu.Unlock() + s.ln = ln +} + +func (s *stopper) setConn(c net.Conn) { + s.mu.Lock() + defer s.mu.Unlock() + s.conn = c +} + +func (s *stopper) setRawFD(fd int) { + s.mu.Lock() + defer s.mu.Unlock() + s.rawFD = fd +} + +func (s *stopper) trigger() { + s.stop.Store(true) + s.mu.Lock() + defer s.mu.Unlock() + if s.ln != nil { + _ = s.ln.Close() + } + if s.conn != nil { + _ = s.conn.Close() + } + if s.rawFD >= 0 { + // SHUT_RDWR makes an in-flight recv(2) return 0 immediately; the fd + // itself is closed by the loop that owns it. + _ = unix.Shutdown(s.rawFD, unix.SHUT_RDWR) + } +} + +// installSignalHandlers wires SIGINT/SIGTERM to the stopper. Unlike the C++ +// port there is no sa_flags choice to make: the Go runtime owns the signal +// disposition and hands us a channel, so unblocking is done by closing fds. +func installSignalHandlers(s *stopper) { + ch := make(chan os.Signal, 1) + signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM) + go func() { + <-ch + s.trigger() + }() +} + +// --------------------------------------------------------------------------- +// Pinning +// --------------------------------------------------------------------------- + +// onlineCPUs reports the number of CPUs online system-wide, matching the C++ +// port's sysconf(_SC_NPROCESSORS_ONLN). runtime.NumCPU() is deliberately NOT +// used: it returns the size of this process's CPU affinity MASK, so under the +// bench driver's `taskset -c 1` it would report 1 and reject a pin to CPU 1 as +// "out of range" — the classic case of a count that lies about its container, +// hit here via taskset rather than a cgroup. Reading the kernel's own online +// list sidesteps the mask. Falls back to runtime.NumCPU() only if the sysfs +// node is unreadable. +func onlineCPUs() int { + data, err := os.ReadFile("/sys/devices/system/cpu/online") + if err != nil { + return runtime.NumCPU() + } + count := 0 + for _, part := range strings.Split(strings.TrimSpace(string(data)), ",") { + if part == "" { + continue + } + if lo, hi, ok := strings.Cut(part, "-"); ok { + a, err1 := strconv.Atoi(lo) + b, err2 := strconv.Atoi(hi) + if err1 != nil || err2 != nil || b < a { + return runtime.NumCPU() + } + count += b - a + 1 + } else { + if _, err := strconv.Atoi(part); err != nil { + return runtime.NumCPU() + } + count++ + } + } + if count == 0 { + return runtime.NumCPU() + } + return count +} + +// pinToCPU restricts this process to a single CPU, and — critically — makes +// that restriction visible in /proc//status, which is where verify.lua +// reads Cpus_allowed_list to prove the pinning happened at the kernel level +// rather than trusting the line we print. +// +// The affinity call is made twice on purpose. sched_setaffinity(0, ...) acts +// on the CALLING thread, which after LockOSThread is the one running the hot +// loop — that call is what makes the pinning real. But /proc//status +// reports the mask of the THREAD GROUP LEADER (the thread whose tid == pid), +// and the main goroutine is not guaranteed to be running on it. Passing +// os.Getpid() explicitly targets the leader, so the observable proof matches +// the actual behavior. When both are the same thread the second call is a +// harmless no-op. +func pinToCPU(cpu int) error { + nproc := onlineCPUs() + if cpu < 0 || (nproc > 0 && cpu >= nproc) { + last := 0 + if nproc > 0 { + last = nproc - 1 + } + return fmt.Errorf("cpu %d out of range (0..%d)", cpu, last) + } + + // One P, one hot thread: without this the runtime is free to schedule the + // loop's goroutine onto another M running on another CPU. + runtime.GOMAXPROCS(1) + runtime.LockOSThread() + + var set unix.CPUSet + set.Zero() + set.Set(cpu) + if err := unix.SchedSetaffinity(0, &set); err != nil { + return fmt.Errorf("sched_setaffinity(%d): %v", cpu, err) + } + if err := unix.SchedSetaffinity(os.Getpid(), &set); err != nil { + return fmt.Errorf("sched_setaffinity(%d): %v", cpu, err) + } + return nil +} + +// --------------------------------------------------------------------------- +// Sockets +// --------------------------------------------------------------------------- + +func listenTCP(port uint16) (net.Listener, error) { + lc := net.ListenConfig{ + Control: func(_, _ string, c syscall.RawConn) error { + var serr error + if err := c.Control(func(fd uintptr) { + serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1) + }); err != nil { + return err + } + return serr + }, + } + ln, err := lc.Listen(context.Background(), "tcp4", fmt.Sprintf("0.0.0.0:%d", port)) + if err != nil { + return nil, fmt.Errorf("bind 0.0.0.0:%d: %v", port, unwrapSyscallErr(err)) + } + return ln, nil +} + +func connectTCP(host string, port uint16) (net.Conn, error) { + // Deliberately no DNS: the C++ port uses inet_pton(3), so a non-literal + // host is an error here too rather than a lookup. + ip := net.ParseIP(host) + if ip == nil || ip.To4() == nil { + return nil, fmt.Errorf("%s: not an IPv4 address", host) + } + conn, err := net.DialTCP("tcp4", nil, &net.TCPAddr{IP: ip.To4(), Port: int(port)}) + if err != nil { + return nil, fmt.Errorf("connect %s:%d: %v", host, port, unwrapSyscallErr(err)) + } + return conn, nil +} + +// unwrapSyscallErr digs the bare errno out of Go's layered net errors so the +// printed text reads like the C++ port's strerror output rather than +// "dial tcp4 127.0.0.1:1: connect: connection refused". +func unwrapSyscallErr(err error) error { + var errno syscall.Errno + if errors.As(err, &errno) { + return errno + } + return err +} + +func setNoDelay(c net.Conn) { + if tc, ok := c.(*net.TCPConn); ok { + _ = tc.SetNoDelay(true) + } +} + +// readFullBusy spins a non-blocking recv(2) with no other syscall in the loop +// — no sched_yield, no nanosleep — which is the entire point of "busy-poll". +func readFullBusy(fd int, buf []byte, s *stopper) bool { + off := 0 + for off < len(buf) { + n, err := unix.Read(fd, buf[off:]) + if err != nil { + if errors.Is(err, unix.EAGAIN) || errors.Is(err, unix.EWOULDBLOCK) { + if s.stop.Load() { + return false + } + continue // spin + } + if errors.Is(err, unix.EINTR) { + continue + } + return false + } + if n == 0 { + return false // peer closed + } + off += n + } + return true +} + +func writeAllFD(fd int, buf []byte) { + off := 0 + for off < len(buf) { + n, err := unix.Write(fd, buf[off:]) + if err != nil { + if errors.Is(err, unix.EINTR) || errors.Is(err, unix.EAGAIN) { + continue + } + return // peer gone; nothing sensible to do + } + off += n + } +} + +// acceptLoop runs the accept/serve skeleton shared by both servers: a +// deadline-bounded Accept so the stop flag is noticed within one interval, +// then handOff for the per-connection loop. +func acceptLoop(ln net.Listener, s *stopper, handOff func(net.Conn)) error { + for !s.stop.Load() { + if tl, ok := ln.(*net.TCPListener); ok { + _ = tl.SetDeadline(time.Now().Add(acceptPollInterval)) + } + conn, err := ln.Accept() + if err != nil { + var ne net.Error + if errors.As(err, &ne) && ne.Timeout() { + continue + } + if s.stop.Load() || errors.Is(err, net.ErrClosed) { + break + } + return fmt.Errorf("accept: %v", unwrapSyscallErr(err)) + } + setNoDelay(conn) + s.setConn(conn) + handOff(conn) + s.setConn(nil) + _ = conn.Close() + } + return nil +} + +// --------------------------------------------------------------------------- +// fastpath / naive servers +// --------------------------------------------------------------------------- + +func cmdFastpath(port uint16, pinCPU int, busyPoll bool) int { + if err := pinToCPU(pinCPU); err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + ln, err := listenTCP(port) + if err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + defer ln.Close() + + s := newStopper() + s.setListener(ln) + installSignalHandlers(s) + + busy := "off" + if busyPoll { + busy = "on" + } + fmt.Fprintf(os.Stderr, "app: fastpath listening on 0.0.0.0:%d pinned-cpu=%d busy-poll=%s\n", + port, pinCPU, busy) + + // The ONE buffer for the whole life of the server: no allocation anywhere + // in the loops below. + buf := make([]byte, frameSize) + + err = acceptLoop(ln, s, func(conn net.Conn) { + tc, ok := conn.(*net.TCPConn) + if busyPoll && ok { + // Take a dup'd descriptor out of the netpoller entirely and drive + // it non-blocking by hand. TCPConn.File() hands back a blocking + // dup, so O_NONBLOCK is set explicitly. + f, ferr := tc.File() + if ferr != nil { + return + } + defer f.Close() + fd := int(f.Fd()) + if serr := unix.SetNonblock(fd, true); serr != nil { + return + } + s.setRawFD(fd) + defer s.setRawFD(-1) + for { + if !readFullBusy(fd, buf, s) { + break + } + writeAllFD(fd, buf) + } + return + } + for { + if _, rerr := io.ReadFull(conn, buf); rerr != nil { + break + } + if _, werr := conn.Write(buf); werr != nil { + break + } + } + }) + if err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + + fmt.Fprintln(os.Stderr, "app: fastpath shutting down") + return 0 +} + +func cmdNaive(port uint16) int { + ln, err := listenTCP(port) + if err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + defer ln.Close() + + s := newStopper() + s.setListener(ln) + installSignalHandlers(s) + + fmt.Fprintf(os.Stderr, "app: naive listening on 0.0.0.0:%d\n", port) + + err = acceptLoop(ln, s, func(conn net.Conn) { + for { + // The "naive" part: a fresh heap buffer, every single message. + heapBuf := make([]byte, frameSize) + if _, rerr := io.ReadFull(conn, heapBuf); rerr != nil { + break + } + if _, werr := conn.Write(heapBuf); werr != nil { + break + } + // heapBuf becomes garbage here, every iteration. + } + }) + if err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + + fmt.Fprintln(os.Stderr, "app: naive shutting down") + return 0 +} + +// --------------------------------------------------------------------------- +// measure +// --------------------------------------------------------------------------- + +func percentileNs(sorted []uint64, p float64) uint64 { + if len(sorted) == 0 { + return 0 + } + n := len(sorted) + idx := int(math.Ceil(p / 100.0 * float64(n))) + if idx == 0 { + idx = 1 + } + if idx > n { + idx = n + } + return sorted[idx-1] +} + +func cmdMeasure(host string, port uint16, n, warmup uint64, tag string) int { + conn, err := connectTCP(host, port) + if err != nil { + fmt.Fprintf(os.Stderr, "app: error: %v\n", err) + return 1 + } + defer conn.Close() + setNoDelay(conn) + + fmt.Printf("app: measure target=%s:%d n=%d warmup=%d\n", host, port, n, warmup) + + samples := make([]uint64, 0, n) + in := make([]byte, frameSize) + total := n + warmup + for i := uint64(0); i < total; i++ { + out := buildFrame(i, 0) + tSend := nowNs() + if _, werr := conn.Write(out); werr != nil { + fmt.Fprintf(os.Stderr, "app: error: measure: connection closed at iteration %d\n", i) + return 1 + } + if _, rerr := io.ReadFull(conn, in); rerr != nil { + fmt.Fprintf(os.Stderr, "app: error: measure: connection closed at iteration %d\n", i) + return 1 + } + tRecv := nowNs() + if !frameHeaderOK(in) || binary.BigEndian.Uint64(in[4:12]) != i { + fmt.Fprintf(os.Stderr, + "app: error: measure: malformed/mismatched echo at iteration %d\n", i) + return 1 + } + if i >= warmup { + samples = append(samples, tRecv-tSend) + } + } + + sort.Slice(samples, func(a, b int) bool { return samples[a] < samples[b] }) + p50 := percentileNs(samples, 50.0) + p90 := percentileNs(samples, 90.0) + p99 := percentileNs(samples, 99.0) + p999 := percentileNs(samples, 99.9) + mn := samples[0] + mx := samples[len(samples)-1] + sum := 0.0 + for _, v := range samples { + sum += float64(v) + } + mean := sum / float64(len(samples)) + + if tag == "" { + tag = "-" + } + fmt.Printf( + "app: percentiles_ns tag=%s p50=%d p90=%d p99=%d p99.9=%d min=%d max=%d mean=%.2f n=%d\n", + tag, p50, p90, p99, p999, mn, mx, mean, len(samples)) + fmt.Println("app: table") + fmt.Printf(" p50 %d ns\n", p50) + fmt.Printf(" p90 %d ns\n", p90) + fmt.Printf(" p99 %d ns\n", p99) + fmt.Printf(" p99.9 %d ns\n", p999) + fmt.Printf(" min %d ns\n", mn) + fmt.Printf(" max %d ns\n", mx) + fmt.Printf(" mean %.2f ns\n", mean) + return 0 +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +func usage() int { + fmt.Fprintln(os.Stderr, "usage: app fastpath --port P --pin CPU [--busy-poll]") + fmt.Fprintln(os.Stderr, " app naive --port P") + fmt.Fprintln(os.Stderr, " app measure --target HOST:PORT --n N [--warmup W] [--tag TAG]") + return 2 +} + +func parsePort(s string) (uint16, bool) { + v, err := strconv.ParseUint(s, 10, 64) + if err != nil || v == 0 || v > 65535 { + return 0, false + } + return uint16(v), true +} + +func parseU64(s string) (uint64, bool) { + v, err := strconv.ParseUint(s, 10, 64) + if err != nil { + return 0, false + } + return v, true +} + +func main() { + os.Exit(run(os.Args[1:])) +} + +func run(args []string) int { + if len(args) == 0 { + return usage() + } + + switch args[0] { + case "fastpath": + portStr := "" + pinCPU := -1 + busyPoll := false + for i := 1; i < len(args); i++ { + switch { + case args[i] == "--port" && i+1 < len(args): + i++ + portStr = args[i] + case args[i] == "--pin" && i+1 < len(args): + i++ + v, ok := parseU64(args[i]) + if !ok { + return usage() + } + pinCPU = int(v) + case args[i] == "--busy-poll": + busyPoll = true + default: + return usage() + } + } + port, ok := parsePort(portStr) + if !ok || pinCPU < 0 { + return usage() + } + return cmdFastpath(port, pinCPU, busyPoll) + + case "naive": + portStr := "" + for i := 1; i < len(args); i++ { + if args[i] == "--port" && i+1 < len(args) { + i++ + portStr = args[i] + } else { + return usage() + } + } + port, ok := parsePort(portStr) + if !ok { + return usage() + } + return cmdNaive(port) + + case "measure": + target := "" + var n uint64 + warmup := uint64(200) + tag := "" + haveN := false + for i := 1; i < len(args); i++ { + switch { + case args[i] == "--target" && i+1 < len(args): + i++ + target = args[i] + case args[i] == "--n" && i+1 < len(args): + i++ + v, ok := parseU64(args[i]) + if !ok { + return usage() + } + n = v + haveN = true + case args[i] == "--warmup" && i+1 < len(args): + i++ + v, ok := parseU64(args[i]) + if !ok { + return usage() + } + warmup = v + case args[i] == "--tag" && i+1 < len(args): + i++ + tag = args[i] + default: + return usage() + } + } + colon := strings.LastIndex(target, ":") + if !haveN || n == 0 || colon < 0 { + return usage() + } + port, ok := parsePort(target[colon+1:]) + if !ok { + return usage() + } + return cmdMeasure(target[:colon], port, n, warmup, tag) + } + + return usage() +} diff --git a/examples/40-low-latency-fastpath/run-latency-bench.sh b/examples/40-low-latency-fastpath/run-latency-bench.sh new file mode 100755 index 0000000..a6a33eb --- /dev/null +++ b/examples/40-low-latency-fastpath/run-latency-bench.sh @@ -0,0 +1,137 @@ +#!/usr/bin/env bash +# run-latency-bench.sh — staged onto the lab guest by verify.lua and run there +# (never on the host: this book's vm-mode examples never ssh from the local +# machine into anything but the lab guest itself, and this script IS what runs +# on that guest). +# +# ./run-latency-bench.sh /path/to/app [N] [WARMUP] +# +# Two independent phases, both against a fresh pair of server processes so +# neither leaves state the other depends on: +# +# Phase A — kernel-level proof that --pin actually restricts scheduling. +# Neither server is touched by an external `taskset`; naive keeps whatever +# affinity mask the shell handed it (every online CPU) while fastpath's own +# sched_setaffinity(2) call (inside `app fastpath --pin`) narrows its own +# mask down to exactly one CPU. Reading /proc//status confirms this at +# the kernel level rather than trusting the program's own printed line. +# +# Phase B — the timing comparison. A naive-vs-fastpath latency race is only +# meaningful if everything about connection placement is identical between +# the two runs; on this lab's 2-vCPU guest, whether the client and server +# land on the SAME virtual CPU or DIFFERENT ones dominates the result (a +# same-vCPU handoff needs no cross-vCPU wakeup; a cross-vCPU one does, and +# under nested KVM that IPI is expensive). So here BOTH servers are +# externally `taskset`-pinned to the same CPU fastpath will end up on +# internally, and the client is pinned to the other CPU for both +# measurements — holding placement constant so the only thing that differs +# is the allocation/scheduling discipline inside each server. +set -uo pipefail + +APP="${1:?usage: run-latency-bench.sh /path/to/app [N] [WARMUP]}" +N="${2:-20000}" +WARMUP="${3:-500}" + +NPROC=$(nproc) +if (( NPROC >= 2 )); then + PIN_CPU=1 + CLIENT_CPU=0 +else + PIN_CPU=0 + CLIENT_CPU=0 +fi + +PORT_NAIVE_A=19501 +PORT_FAST_A=19502 +PORT_NAIVE_B=19503 +PORT_FAST_B=19504 + +echo "app: nproc=$NPROC pin_cpu=$PIN_CPU client_cpu=$CLIENT_CPU" + +wait_for_port() { + local host="$1" port="$2" + for _ in $(seq 1 50); do + if (exec 3<>"/dev/tcp/$host/$port") 2>/dev/null; then + exec 3<&- 2>/dev/null + exec 3>&- 2>/dev/null + return 0 + fi + sleep 0.1 + done + return 1 +} + +cpus_allowed() { + awk '/Cpus_allowed_list/ {print $2}' "/proc/$1/status" 2>/dev/null +} + +# --------------------------------------------------------------------------- +# Phase A: kernel-level pinning proof. +# --------------------------------------------------------------------------- +"$APP" naive --port "$PORT_NAIVE_A" >/tmp/lsp40-naiveA.log 2>&1 & +naive_a_pid=$! +"$APP" fastpath --port "$PORT_FAST_A" --pin "$PIN_CPU" >/tmp/lsp40-fastA.log 2>&1 & +fast_a_pid=$! + +if ! wait_for_port 127.0.0.1 "$PORT_NAIVE_A" || ! wait_for_port 127.0.0.1 "$PORT_FAST_A"; then + echo "app: error: phase A servers did not come up" + cat /tmp/lsp40-naiveA.log /tmp/lsp40-fastA.log 2>/dev/null + kill "$naive_a_pid" "$fast_a_pid" 2>/dev/null + exit 1 +fi + +echo "app: naive-cpus-allowed $(cpus_allowed "$naive_a_pid")" +echo "app: fastpath-cpus-allowed $(cpus_allowed "$fast_a_pid")" + +kill "$naive_a_pid" "$fast_a_pid" 2>/dev/null +wait "$naive_a_pid" 2>/dev/null +wait "$fast_a_pid" 2>/dev/null + +# --------------------------------------------------------------------------- +# Phase B: confound-controlled timing comparison. +# --------------------------------------------------------------------------- +taskset -c "$PIN_CPU" "$APP" naive --port "$PORT_NAIVE_B" >/tmp/lsp40-naiveB.log 2>&1 & +naive_b_pid=$! +taskset -c "$PIN_CPU" "$APP" fastpath --port "$PORT_FAST_B" --pin "$PIN_CPU" --busy-poll \ + >/tmp/lsp40-fastB.log 2>&1 & +fast_b_pid=$! + +if ! wait_for_port 127.0.0.1 "$PORT_NAIVE_B" || ! wait_for_port 127.0.0.1 "$PORT_FAST_B"; then + echo "app: error: phase B servers did not come up" + cat /tmp/lsp40-naiveB.log /tmp/lsp40-fastB.log 2>/dev/null + kill "$naive_b_pid" "$fast_b_pid" 2>/dev/null + exit 1 +fi + +# Three interleaved trials per variant (naive1, fastpath1, naive2, ...) rather +# than three-then-three: interleaving keeps any slow drift in host load from +# landing entirely on one side of the comparison. Each trial is tagged so +# verify.lua can group them by variant regardless of order; the pass/fail +# gate downstream compares the MEDIAN of the three trials per variant, not a +# single sample — this lab guest is a shared, nested-KVM host, and a lone +# run occasionally hits a multi-hundred-microsecond scheduling stall +# unrelated to which variant is being measured (see README). +status=0 +for trial in 1 2 3; do + echo "app: === naive measure trial $trial/3 ===" + taskset -c "$CLIENT_CPU" "$APP" measure --target "127.0.0.1:$PORT_NAIVE_B" --n "$N" \ + --warmup "$WARMUP" --tag "naive-$trial" + (( $? == 0 )) || status=1 + + echo "app: === fastpath measure trial $trial/3 ===" + taskset -c "$CLIENT_CPU" "$APP" measure --target "127.0.0.1:$PORT_FAST_B" --n "$N" \ + --warmup "$WARMUP" --tag "fastpath-$trial" + (( $? == 0 )) || status=1 +done +naive_status=$status +fast_status=$status + +kill "$naive_b_pid" "$fast_b_pid" 2>/dev/null +wait "$naive_b_pid" 2>/dev/null +wait "$fast_b_pid" 2>/dev/null + +if [[ "$naive_status" -ne 0 || "$fast_status" -ne 0 ]]; then + echo "app: error: measure exited nonzero (naive=$naive_status fastpath=$fast_status)" + exit 1 +fi +exit 0 diff --git a/examples/40-low-latency-fastpath/rust/Cargo.lock b/examples/40-low-latency-fastpath/rust/Cargo.lock new file mode 100644 index 0000000..1fe190f --- /dev/null +++ b/examples/40-low-latency-fastpath/rust/Cargo.lock @@ -0,0 +1,62 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cfg_aliases" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" + +[[package]] +name = "chatterd-fastpath" +version = "0.1.0" +dependencies = [ + "nix", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "memoffset" +version = "0.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "nix" +version = "0.31.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d" +dependencies = [ + "bitflags", + "cfg-if", + "cfg_aliases", + "libc", + "memoffset", +] diff --git a/examples/40-low-latency-fastpath/rust/Cargo.toml b/examples/40-low-latency-fastpath/rust/Cargo.toml new file mode 100644 index 0000000..beb15e5 --- /dev/null +++ b/examples/40-low-latency-fastpath/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "chatterd-fastpath" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +nix = { version = "0.31", features = ["socket", "net", "poll", "sched", "signal", "fs"] } diff --git a/examples/40-low-latency-fastpath/rust/demo.sh b/examples/40-low-latency-fastpath/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/40-low-latency-fastpath/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/40-low-latency-fastpath/rust/rust-toolchain.toml b/examples/40-low-latency-fastpath/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/40-low-latency-fastpath/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/40-low-latency-fastpath/rust/src/main.rs b/examples/40-low-latency-fastpath/rust/src/main.rs new file mode 100644 index 0000000..865971f --- /dev/null +++ b/examples/40-low-latency-fastpath/rust/src/main.rs @@ -0,0 +1,580 @@ +// chatterd-fastpath — a stripped, pinned, allocation-free derivative of the +// chatterd echo/pub-sub hot path (chapter 21) built purely to answer one +// question: what does a low-latency network hot path actually cost, and what +// does removing that cost look like at the syscall level? +// +// Three subcommands, one 64-byte wire frame; observable behavior is identical +// to the C++ and Go ports (same usage text, same "app: ..." lines, same +// percentiles_ns format), because verify.lua asserts against all three. +// +// app fastpath --port P --pin CPU [--busy-poll] +// app naive --port P +// app measure --target HOST:PORT --n N [--warmup W] [--tag TAG] +// +// THE FASTPATH FRAME (fixed 64 bytes): +// +// +-------+---------+------+-----------+-----------------+---------+ +// | magic | version | type | seq (u64) | send_ns (u64) | pad | +// | 2B | 1B | 1B | 8B BE | 8B BE | 44B | +// +-------+---------+------+-----------+-----------------+---------+ +// +// WHY THIS PORT USES RAW FDS INSTEAD OF std::net: +// +// This is the port that follows the C++ original most literally, and it can: +// Rust's std makes real blocking syscalls (there is no M:N runtime parking +// tasks off the syscall the way Go's netpoller does), so the C++ technique — a +// signal handler installed WITHOUT SA_RESTART, so a blocking read(2)/poll(2) +// already in progress returns EINTR and the loop notices the stop flag — works +// verbatim here. The Go port had to substitute a close-the-fd wakeup for +// exactly this reason; Rust does not. +// +// The catch is that std's higher-level wrappers hide EINTR: TcpListener::accept +// retries it internally (cvt_r), and Read::read_exact treats +// ErrorKind::Interrupted as "try again". Both would swallow the very signal +// this example is about. So the servers are built on raw fds via nix, where +// every EINTR is handled explicitly — which also makes OwnedFd the natural +// equivalent of the C++ RAII Socket: it closes its descriptor on drop, is +// move-only, and needs no hand-written destructor. +// +// Pinning is simpler than in Go. sched_setaffinity(2) acts on a thread, and +// this program is single-threaded — the main thread's tid equals the pid — so +// one sched_setaffinity(0, ...) call both pins the hot loop and shows up in +// /proc//status, where verify.lua reads Cpus_allowed_list to prove the +// pinning happened at the kernel level. No LockOSThread / GOMAXPROCS dance is +// needed because there is no runtime scheduler moving the work between CPUs. + +use std::net::{Ipv4Addr, SocketAddrV4}; +use std::os::fd::{AsFd, AsRawFd, BorrowedFd, FromRawFd, OwnedFd, RawFd}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Instant; + +use nix::errno::Errno; +use nix::fcntl::{FcntlArg, OFlag, fcntl}; +use nix::libc; +use nix::poll::{PollFd, PollFlags, PollTimeout, poll}; +use nix::sched::{CpuSet, sched_setaffinity}; +use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction}; +use nix::sys::socket::{ + AddressFamily, Backlog, MsgFlags, SockFlag, SockType, SockaddrIn, accept4, bind, connect, + listen, recv, setsockopt, socket, sockopt, +}; +use nix::unistd::{Pid, read, write}; + +const FRAME_SIZE: usize = 64; +const MAGIC0: u8 = 0x43; // 'C' +const MAGIC1: u8 = 0x46; // 'F' +const WIRE_VERSION: u8 = 0x01; +const TYPE_ECHO: u8 = 0x01; + +type Frame = [u8; FRAME_SIZE]; + +// A fallible setup step carrying an already-formatted, user-facing message — +// the Rust stand-in for the C++ port's std::expected. +type StepResult = Result; + +// --------------------------------------------------------------------------- +// Frame +// --------------------------------------------------------------------------- + +fn build_frame(seq: u64, send_ns: u64) -> Frame { + let mut f = [0u8; FRAME_SIZE]; + f[0] = MAGIC0; + f[1] = MAGIC1; + f[2] = WIRE_VERSION; + f[3] = TYPE_ECHO; + f[4..12].copy_from_slice(&seq.to_be_bytes()); + f[12..20].copy_from_slice(&send_ns.to_be_bytes()); + // bytes 20..63 stay zero + f +} + +fn frame_header_ok(f: &Frame) -> bool { + f[0] == MAGIC0 && f[1] == MAGIC1 && f[2] == WIRE_VERSION && f[3] == TYPE_ECHO +} + +fn seq_of(f: &Frame) -> u64 { + u64::from_be_bytes(f[4..12].try_into().unwrap()) +} + +// --------------------------------------------------------------------------- +// Shutdown plumbing: a plain (non-SA_RESTART) signal handler so a blocking +// read(2)/accept-wait poll(2) in progress is interrupted with EINTR rather +// than transparently resumed — exactly the C++ port's technique. +// --------------------------------------------------------------------------- + +static STOP: AtomicBool = AtomicBool::new(false); + +extern "C" fn on_signal(_: libc::c_int) { + STOP.store(true, Ordering::Relaxed); +} + +fn install_signal_handlers() { + let sa = SigAction::new( + SigHandler::Handler(on_signal), + SaFlags::empty(), // deliberately NOT SA_RESTART + SigSet::empty(), + ); + // SAFETY: on_signal only performs a relaxed atomic store, which is + // async-signal-safe. + unsafe { + let _ = sigaction(Signal::SIGINT, &sa); + let _ = sigaction(Signal::SIGTERM, &sa); + } +} + +fn stop_requested() -> bool { + STOP.load(Ordering::Relaxed) +} + +// --------------------------------------------------------------------------- +// Pinning +// --------------------------------------------------------------------------- + +fn pin_to_cpu(cpu: i32) -> StepResult<()> { + // sysconf(_SC_NPROCESSORS_ONLN) reports CPUs online SYSTEM-WIDE and ignores + // this process's affinity mask — the same call the C++ port uses. That + // matters under the bench driver's `taskset -c 1`: a mask-aware count + // (like Go's runtime.NumCPU) would report 1 and wrongly reject a pin to + // CPU 1. sysconf sidesteps that. + let nproc = unsafe { libc::sysconf(libc::_SC_NPROCESSORS_ONLN) }; + if cpu < 0 || (nproc > 0 && i64::from(cpu) >= nproc) { + let last = if nproc > 0 { nproc - 1 } else { 0 }; + return Err(format!("cpu {cpu} out of range (0..{last})")); + } + let mut set = CpuSet::new(); + set.set(cpu as usize) + .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; + // Pid 0 == the calling thread, which is the main (and only) thread, whose + // tid == pid — so this is also what /proc//status reports. + sched_setaffinity(Pid::from_raw(0), &set) + .map_err(|e| format!("sched_setaffinity({cpu}): {e}"))?; + Ok(()) +} + +// --------------------------------------------------------------------------- +// Sockets — OwnedFd is the RAII Socket: closes on drop, move-only. +// --------------------------------------------------------------------------- + +fn listen_tcp(port: u16) -> StepResult { + let sock = socket( + AddressFamily::Inet, + SockType::Stream, + SockFlag::SOCK_CLOEXEC, + None, + ) + .map_err(|e| format!("socket: {e}"))?; + let _ = setsockopt(&sock, sockopt::ReuseAddr, &true); + + let addr = SockaddrIn::new(0, 0, 0, 0, port); // 0.0.0.0:port + bind(sock.as_raw_fd(), &addr).map_err(|e| format!("bind 0.0.0.0:{port}: {e}"))?; + listen(&sock, Backlog::new(16).unwrap()).map_err(|e| format!("listen: {e}"))?; + Ok(sock) +} + +fn connect_tcp(host: &str, port: u16) -> StepResult { + // Deliberately no DNS: the C++ port uses inet_pton(3), so a non-literal + // host is an error here too rather than a lookup. + let ip: Ipv4Addr = host + .parse() + .map_err(|_| format!("{host}: not an IPv4 address"))?; + let sock = socket( + AddressFamily::Inet, + SockType::Stream, + SockFlag::SOCK_CLOEXEC, + None, + ) + .map_err(|e| format!("socket: {e}"))?; + let addr = SockaddrIn::from(SocketAddrV4::new(ip, port)); + connect(sock.as_raw_fd(), &addr).map_err(|e| format!("connect {host}:{port}: {e}"))?; + Ok(sock) +} + +// Blocking read of exactly n bytes; false on EOF, hard error, or a signal +// landing (EINTR) — the caller decides whether that means "stop". +fn read_full_blocking(fd: RawFd, buf: &mut [u8]) -> bool { + let bfd = unsafe { BorrowedFd::borrow_raw(fd) }; + let mut off = 0; + while off < buf.len() { + match read(bfd, &mut buf[off..]) { + Ok(0) => return false, // peer closed + Ok(n) => off += n, + Err(_) => return false, + } + } + true +} + +// Non-blocking spin read: the socket must already be O_NONBLOCK. Spins on +// EAGAIN with no syscall other than recv(2) itself — no sched_yield, no +// nanosleep — which is the entire point of "busy-poll". +fn read_full_busy(fd: RawFd, buf: &mut [u8]) -> bool { + let mut off = 0; + while off < buf.len() { + match recv(fd, &mut buf[off..], MsgFlags::empty()) { + Ok(0) => return false, // peer closed + Ok(n) => off += n, + Err(Errno::EAGAIN) => { + if stop_requested() { + return false; + } + // spin + } + Err(_) => return false, + } + } + true +} + +fn write_all(fd: RawFd, buf: &[u8]) { + let bfd = unsafe { BorrowedFd::borrow_raw(fd) }; + let mut off = 0; + while off < buf.len() { + match write(bfd, &buf[off..]) { + Ok(n) => off += n, + Err(Errno::EINTR) => {} // retry + Err(_) => return, // peer gone; nothing sensible to do + } + } +} + +// Poll the listener with a 200ms timeout, matching the C++ poll(2) wait, so +// the stop flag is checked periodically even absent a signal. Returns +// Some(conn) on a ready connection, None to loop again (timeout / EINTR). +fn accept_one(lfd: RawFd) -> Result, String> { + let borrowed = unsafe { BorrowedFd::borrow_raw(lfd) }; + let mut fds = [PollFd::new(borrowed, PollFlags::POLLIN)]; + match poll(&mut fds, PollTimeout::from(200u16)) { + Ok(0) => return Ok(None), + Ok(_) => {} + Err(Errno::EINTR) => return Ok(None), + Err(e) => return Err(format!("poll: {e}")), + } + if !fds[0] + .revents() + .map(|r| r.contains(PollFlags::POLLIN)) + .unwrap_or(false) + { + return Ok(None); + } + match accept4(lfd, SockFlag::SOCK_CLOEXEC) { + Ok(cfd) => Ok(Some(unsafe { OwnedFd::from_raw_fd(cfd) })), + Err(Errno::EINTR) | Err(Errno::EAGAIN) => Ok(None), + Err(e) => Err(format!("accept: {e}")), + } +} + +// --------------------------------------------------------------------------- +// fastpath / naive servers +// --------------------------------------------------------------------------- + +fn cmd_fastpath(port: u16, pin_cpu: i32, busy_poll: bool) -> i32 { + if let Err(e) = pin_to_cpu(pin_cpu) { + eprintln!("app: error: {e}"); + return 1; + } + let lsock = match listen_tcp(port) { + Ok(s) => s, + Err(e) => { + eprintln!("app: error: {e}"); + return 1; + } + }; + install_signal_handlers(); + eprintln!( + "app: fastpath listening on 0.0.0.0:{port} pinned-cpu={pin_cpu} busy-poll={}", + if busy_poll { "on" } else { "off" } + ); + + // The ONE buffer for the whole life of the server: no allocation anywhere + // in the loops below. + let mut buf: Frame = [0u8; FRAME_SIZE]; + + while !stop_requested() { + let conn = match accept_one(lsock.as_raw_fd()) { + Ok(Some(c)) => c, + Ok(None) => continue, + Err(e) => { + eprintln!("app: error: {e}"); + return 1; + } + }; + let _ = setsockopt(&conn, sockopt::TcpNoDelay, &true); + if busy_poll { + let flags = fcntl(conn.as_fd(), FcntlArg::F_GETFL).unwrap_or(0); + let mut oflag = OFlag::from_bits_truncate(flags); + oflag.insert(OFlag::O_NONBLOCK); + let _ = fcntl(conn.as_fd(), FcntlArg::F_SETFL(oflag)); + } + let cfd = conn.as_raw_fd(); + loop { + let ok = if busy_poll { + read_full_busy(cfd, &mut buf) + } else { + read_full_blocking(cfd, &mut buf) + }; + if !ok { + break; + } + write_all(cfd, &buf); + } + } + eprintln!("app: fastpath shutting down"); + 0 +} + +fn cmd_naive(port: u16) -> i32 { + let lsock = match listen_tcp(port) { + Ok(s) => s, + Err(e) => { + eprintln!("app: error: {e}"); + return 1; + } + }; + install_signal_handlers(); + eprintln!("app: naive listening on 0.0.0.0:{port}"); + + while !stop_requested() { + let conn = match accept_one(lsock.as_raw_fd()) { + Ok(Some(c)) => c, + Ok(None) => continue, + Err(e) => { + eprintln!("app: error: {e}"); + return 1; + } + }; + let cfd = conn.as_raw_fd(); + loop { + // The "naive" part: a fresh heap buffer, every single message. + let mut heap_buf = vec![0u8; FRAME_SIZE]; + if !read_full_blocking(cfd, &mut heap_buf) { + break; + } + write_all(cfd, &heap_buf); + // heap_buf frees here, every iteration. + } + } + eprintln!("app: naive shutting down"); + 0 +} + +// --------------------------------------------------------------------------- +// measure +// --------------------------------------------------------------------------- + +fn percentile_ns(sorted: &[u64], p: f64) -> u64 { + if sorted.is_empty() { + return 0; + } + let n = sorted.len(); + let mut idx = (p / 100.0 * n as f64).ceil() as usize; + if idx == 0 { + idx = 1; + } + if idx > n { + idx = n; + } + sorted[idx - 1] +} + +fn cmd_measure(host: &str, port: u16, n: u64, warmup: u64, tag: &str) -> i32 { + let conn = match connect_tcp(host, port) { + Ok(c) => c, + Err(e) => { + eprintln!("app: error: {e}"); + return 1; + } + }; + let _ = setsockopt(&conn, sockopt::TcpNoDelay, &true); + let cfd = conn.as_raw_fd(); + + println!("app: measure target={host}:{port} n={n} warmup={warmup}"); + + let mut samples: Vec = Vec::with_capacity(n as usize); + let total = n + warmup; + for i in 0..total { + let out = build_frame(i, 0); + let t_send = Instant::now(); + write_all(cfd, &out); + let mut inbuf: Frame = [0u8; FRAME_SIZE]; + if !read_full_blocking(cfd, &mut inbuf) { + eprintln!("app: error: measure: connection closed at iteration {i}"); + return 1; + } + let dt = t_send.elapsed().as_nanos() as u64; + if !frame_header_ok(&inbuf) || seq_of(&inbuf) != i { + eprintln!("app: error: measure: malformed/mismatched echo at iteration {i}"); + return 1; + } + if i >= warmup { + samples.push(dt); + } + } + + samples.sort_unstable(); + let p50 = percentile_ns(&samples, 50.0); + let p90 = percentile_ns(&samples, 90.0); + let p99 = percentile_ns(&samples, 99.0); + let p999 = percentile_ns(&samples, 99.9); + let mn = *samples.first().unwrap(); + let mx = *samples.last().unwrap(); + let sum: f64 = samples.iter().map(|&v| v as f64).sum(); + let mean = sum / samples.len() as f64; + + let tag = if tag.is_empty() { "-" } else { tag }; + println!( + "app: percentiles_ns tag={tag} p50={p50} p90={p90} p99={p99} p99.9={p999} \ + min={mn} max={mx} mean={mean:.2} n={}", + samples.len() + ); + println!("app: table"); + println!(" p50 {p50} ns"); + println!(" p90 {p90} ns"); + println!(" p99 {p99} ns"); + println!(" p99.9 {p999} ns"); + println!(" min {mn} ns"); + println!(" max {mx} ns"); + println!(" mean {mean:.2} ns"); + 0 +} + +// --------------------------------------------------------------------------- +// CLI +// --------------------------------------------------------------------------- + +fn usage() -> i32 { + eprintln!("usage: app fastpath --port P --pin CPU [--busy-poll]"); + eprintln!(" app naive --port P"); + eprintln!(" app measure --target HOST:PORT --n N [--warmup W] [--tag TAG]"); + 2 +} + +fn parse_port(s: &str) -> Option { + match s.parse::() { + Ok(v) if (1..=65535).contains(&v) => Some(v as u16), + _ => None, + } +} + +fn parse_u64(s: &str) -> Option { + s.parse::().ok() +} + +fn run(args: &[String]) -> i32 { + if args.is_empty() { + return usage(); + } + + match args[0].as_str() { + "fastpath" => { + let mut port_str = String::new(); + let mut pin_cpu: i32 = -1; + let mut busy_poll = false; + let mut i = 1; + while i < args.len() { + match args[i].as_str() { + "--port" if i + 1 < args.len() => { + i += 1; + port_str = args[i].clone(); + } + "--pin" if i + 1 < args.len() => { + i += 1; + match parse_u64(&args[i]) { + Some(v) => pin_cpu = v as i32, + None => return usage(), + } + } + "--busy-poll" => busy_poll = true, + _ => return usage(), + } + i += 1; + } + let port = match parse_port(&port_str) { + Some(p) => p, + None => return usage(), + }; + if pin_cpu < 0 { + return usage(); + } + cmd_fastpath(port, pin_cpu, busy_poll) + } + + "naive" => { + let mut port_str = String::new(); + let mut i = 1; + while i < args.len() { + match args[i].as_str() { + "--port" if i + 1 < args.len() => { + i += 1; + port_str = args[i].clone(); + } + _ => return usage(), + } + i += 1; + } + let port = match parse_port(&port_str) { + Some(p) => p, + None => return usage(), + }; + cmd_naive(port) + } + + "measure" => { + let mut target = String::new(); + let mut n: u64 = 0; + let mut warmup: u64 = 200; + let mut tag = String::new(); + let mut have_n = false; + let mut i = 1; + while i < args.len() { + match args[i].as_str() { + "--target" if i + 1 < args.len() => { + i += 1; + target = args[i].clone(); + } + "--n" if i + 1 < args.len() => { + i += 1; + match parse_u64(&args[i]) { + Some(v) => { + n = v; + have_n = true; + } + None => return usage(), + } + } + "--warmup" if i + 1 < args.len() => { + i += 1; + match parse_u64(&args[i]) { + Some(v) => warmup = v, + None => return usage(), + } + } + "--tag" if i + 1 < args.len() => { + i += 1; + tag = args[i].clone(); + } + _ => return usage(), + } + i += 1; + } + let colon = match target.rfind(':') { + Some(c) => c, + None => return usage(), + }; + if !have_n || n == 0 { + return usage(); + } + let port = match parse_port(&target[colon + 1..]) { + Some(p) => p, + None => return usage(), + }; + cmd_measure(&target[..colon], port, n, warmup, &tag) + } + + _ => usage(), + } +} + +fn main() { + let args: Vec = std::env::args().skip(1).collect(); + std::process::exit(run(&args)); +} diff --git a/examples/40-low-latency-fastpath/verify.lua b/examples/40-low-latency-fastpath/verify.lua new file mode 100644 index 0000000..4fabc1f --- /dev/null +++ b/examples/40-low-latency-fastpath/verify.lua @@ -0,0 +1,340 @@ +-- Verify chatterd-fastpath (40-low-latency-fastpath) for LSP_LANG. A vm-mode +-- example: driven directly against systems-target via plain scp/ssh to +-- /tmp/lsp40-/, never `demo.sh run` under TARGET -- +-- deploy-to-vm.sh's `ssh -t` injects CRLF that breaks Lua patterns, and its +-- fixed /home/fedora/app path can be poisoned by a stale binary left over +-- from an unrelated example (this happened once during golden capture). +-- Every remote invocation below uses the full absolute remote_bin / +-- remote_driver path -- never a leading `cd` relied on to apply across a +-- later `;`-separated command in the same string. Every multi-statement +-- remote script here is joined with "\n", never "; " -- a line ending in +-- `&` (backgrounding) followed by `; ` produces `cmd & ; next`, a bash +-- syntax error that silently makes the whole remote script a no-op parse +-- (a real bug hit while writing example 37's verify.lua). +-- +-- Asserts observable behavior, identical in shape across cpp/go/rust: +-- 1. build, then stage app + run-latency-bench.sh to /tmp/lsp40-/, +-- chmod 755 both -- both must exit 0. +-- 2. error shapes: no args / frobnicate / fastpath without --pin / naive +-- without --port / measure without --n all exit 2 with the 3-line +-- "usage: app fastpath ..." usage text. +-- 3. fastpath --pin (out of range): exit 1, +-- "app: error: cpu N out of range". +-- 4. measure connect-refused (127.0.0.1:1) and measure against a non-IPv4 +-- host: both exit 1, with "app: error: connect" and +-- "not an IPv4 address" respectively. +-- 5. the real run-latency-bench.sh (N=20000, WARMUP=500): exit 0, and its +-- output proves --pin actually restricts scheduling AT THE KERNEL +-- LEVEL (read from /proc//status by the driver script, not the +-- program's own claim) -- fastpath-cpus-allowed is exactly the single +-- pinned CPU while naive-cpus-allowed is the full online set. +-- 6. all six percentiles_ns lines (naive-1..3, fastpath-1..3) parse, each +-- with n=20000, a monotonic min<=p50<=p90<=p99<=p99.9<=max, a mean that +-- parses as a float, and every value > 0. +-- 7. median-of-three per variant: median(fastpath p50) < median(naive p50) +-- * 0.90 -- a conservative 10% requirement against an observed ~30-33% +-- margin across three independent golden-capture bench runs (1.425x - +-- 1.488x, never marginal or inverted) -- both medians are always +-- printed so a marginal run is diagnosable from the log even when this +-- assertion passes. +-- 8. SIGINT shutdown, tested separately per server with a bounded +-- (10s) wait via `timeout 10 tail --pid=$PID -f /dev/null` so a hung +-- server surfaces as a clear FAIL rather than a runner-level timeout -- +-- this is the assertion Go is most likely to fail, since Go's runtime +-- installs SA_RESTART handlers and a blocking read will not see EINTR +-- on its own. naive --port P: "app: naive shutting down", exit 0. +-- fastpath --port P --pin 0 --busy-poll: the exact listening line +-- "app: fastpath listening on 0.0.0.0:P pinned-cpu=0 busy-poll=on", +-- "app: fastpath shutting down", exit 0. +-- +-- Never compares formatted decimals across languages (C++, Go, and Rust +-- disagree on halfway rounding) -- only shapes and numeric ranges/ratios are +-- asserted; percentile values (p50 etc.) are plain integers so they ARE +-- compared numerically across the naive/fastpath medians, but never against +-- a fixed expected number. +-- +-- Invoked from the example directory with LSP_LANG, TARGET, and REPO_ROOT +-- set (EXAMPLE_DIR optional, defaults to "."). + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local target = os.getenv("TARGET") +if not target or target == "" then + checks.skip("no TARGET set -- this is a vm example (run with --mode vm)") +end + +local example_dir = os.getenv("EXAMPLE_DIR") or "." +local home = os.getenv("HOME") or "/root" + +-- Local path of the freshly built binary for this language. +local bin_rel = ({ + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +})[lang] +local local_bin = example_dir .. "/" .. bin_rel +local local_driver = example_dir .. "/run-latency-bench.sh" + +-- Resolve the guest IP (same helper deploy-to-vm.sh uses). +local ipf = io.popen(repo_root .. "/scripts/lab/vm-ip.sh '" .. target .. "' 2>/dev/null") +local ip = ipf and ipf:read("*l") or nil +if ipf then ipf:close() end +if not ip or ip == "" then + checks.skip("cannot resolve IP for VM '" .. target .. "' (lab down?)") +end + +local ssh_opts = + "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=" .. home .. "/.ssh/known_hosts" + +-- Run a (single-quote-free) remote script through ssh. +local function ssh_run(remote_script) + return checks.run(string.format("ssh %s 'fedora@%s' '%s'", ssh_opts, ip, remote_script)) +end + +-- Boolean assertion via the checks.lua pattern-match API: match "ok" against +-- "ok" when true, or force a mismatch (with the real detail in the label) +-- when false -- same helper as examples 37/39's verify.lua. +local function expect_true(cond, label, detail) + if cond then + checks.expect_match("ok", "ok", label) + else + local msg = label + if detail ~= nil then msg = msg .. " (" .. tostring(detail) .. ")" end + checks.expect_match("no", "yes", msg) + end +end + +local remote_dir = "/tmp/lsp40-" .. lang +local remote_bin = remote_dir .. "/app" +local remote_driver = remote_dir .. "/run-latency-bench.sh" + +-- --------------------------------------------------------------------------- +-- 1. build, then stage app + run-latency-bench.sh to /tmp/lsp40-/, +-- chmod 755 both. +-- --------------------------------------------------------------------------- + +local build = checks.run("./demo.sh " .. lang .. " build") +checks.expect_exit(build, 0, lang .. ": demo.sh build exits 0") + +local stage_cmd = string.format( + "ssh %s 'fedora@%s' 'mkdir -p %s' && " .. + "scp %s '%s' 'fedora@%s:%s' && " .. + "scp %s '%s' 'fedora@%s:%s' && " .. + "ssh %s 'fedora@%s' 'chmod 755 %s %s'", + ssh_opts, ip, remote_dir, + ssh_opts, local_bin, ip, remote_bin, + ssh_opts, local_driver, ip, remote_driver, + ssh_opts, ip, remote_bin, remote_driver) +local staged = checks.run(stage_cmd) +checks.expect_exit(staged, 0, lang .. ": stage app + run-latency-bench.sh to " .. remote_dir) + +-- The guest's real core count drives the out-of-range --pin value below. +local nproc_r = ssh_run("nproc") +local nproc = tonumber(nproc_r.out:match("%d+")) or 1 + +-- --------------------------------------------------------------------------- +-- 2. error shapes -- exit 2, "usage: app fastpath ..." on stderr. +-- --------------------------------------------------------------------------- + +local function expect_usage(args, label) + local r = ssh_run(remote_bin .. " " .. args) + checks.expect_exit(r, 2, lang .. ": " .. label .. " exits 2") + checks.expect_match(r.out, "usage: app fastpath", lang .. ": " .. label .. " prints usage") +end + +expect_usage("", "no args") +expect_usage("frobnicate", "frobnicate") +expect_usage("fastpath --port 9999", "fastpath --port 9999 (no --pin)") +expect_usage("naive", "naive (no port)") +expect_usage("measure --target 127.0.0.1:9999", "measure --target ... (no --n)") + +-- --------------------------------------------------------------------------- +-- 3. fastpath --pin out of range -- exit 1, cpu N out of range. +-- --------------------------------------------------------------------------- + +local bad_cpu = nproc + 50 +local pin_r = ssh_run(remote_bin .. " fastpath --port 19700 --pin " .. bad_cpu) +checks.expect_exit(pin_r, 1, lang .. ": fastpath --pin " .. bad_cpu .. " (out of range) exits 1") +checks.expect_match(pin_r.out, "app: error: cpu %d+ out of range", + lang .. ": out-of-range --pin prints the cpu-out-of-range error") + +-- --------------------------------------------------------------------------- +-- 4. measure error shapes -- connect refused, non-IPv4 host. +-- --------------------------------------------------------------------------- + +local connect_r = ssh_run(remote_bin .. " measure --target 127.0.0.1:1 --n 10") +checks.expect_exit(connect_r, 1, lang .. ": measure --target 127.0.0.1:1 (connect refused) exits 1") +checks.expect_match(connect_r.out, "app: error: connect", + lang .. ": measure connect-refused prints the connect error") + +local ipv4_r = ssh_run(remote_bin .. " measure --target nothost:9999 --n 5") +checks.expect_exit(ipv4_r, 1, lang .. ": measure --target nothost:9999 (not IPv4) exits 1") +checks.expect_match(ipv4_r.out, "not an IPv4 address", + lang .. ": measure against a non-IPv4 host reports not-an-IPv4-address") + +-- --------------------------------------------------------------------------- +-- 5, 6, 7. the real run-latency-bench.sh: kernel-level pinning proof, all +-- six percentiles_ns lines parse, and median(fastpath) beats median(naive). +-- --------------------------------------------------------------------------- + +local bench_script = table.concat({ + "OUT=/tmp/lsp40_bench_" .. lang .. ".$$", + "bash " .. remote_driver .. " " .. remote_bin .. " 20000 500 >$OUT 2>&1", + "RC=$?", + "cat $OUT", + "rm -f $OUT", + "echo BENCH_EXIT=$RC", +}, "\n") +local br = ssh_run(bench_script) +local bench_exit = tonumber(br.out:match("BENCH_EXIT=(%d+)")) +expect_true(bench_exit == 0, + lang .. ": run-latency-bench.sh /tmp/lsp40-" .. lang .. "/app 20000 500 exits 0", + tostring(bench_exit)) + +-- 5. Kernel-level pinning proof (phase A of the driver script), read from +-- /proc//status by the driver -- not the program's own printed claim. +local naive_allowed = br.out:match("app: naive%-cpus%-allowed (%S+)") +local fast_allowed = br.out:match("app: fastpath%-cpus%-allowed (%S+)") + +local want_naive_allowed = (nproc > 1) and ("0-" .. (nproc - 1)) or "0" +expect_true(naive_allowed == want_naive_allowed, + lang .. ": naive-cpus-allowed is the full online CPU set (untouched affinity mask)", + string.format("got=%s want=%s", tostring(naive_allowed), want_naive_allowed)) +expect_true(fast_allowed ~= nil and fast_allowed:match("^%d+$") ~= nil, + lang .. ": fastpath-cpus-allowed is a single CPU value (no range, no comma) -- " .. + "the Go port is most likely to fail this", + tostring(fast_allowed)) + +-- 6. All six percentiles_ns lines: n=20000, monotonic percentiles, mean +-- parses as a float, every value > 0. +local samples = { naive = {}, fastpath = {} } +local pct_pattern = + "app: percentiles_ns tag=(%a+)%-(%d+) p50=(%d+) p90=(%d+) p99=(%d+) p99%.9=(%d+) " .. + "min=(%d+) max=(%d+) mean=(%d+%.%d+) n=(%d+)" +local matched = 0 +for variant, trial, p50, p90, p99, p999, minv, maxv, mean, n in br.out:gmatch(pct_pattern) do + matched = matched + 1 + p50, p90, p99, p999 = tonumber(p50), tonumber(p90), tonumber(p99), tonumber(p999) + minv, maxv, n = tonumber(minv), tonumber(maxv), tonumber(n) + local mean_n = tonumber(mean) + local label = lang .. ": " .. variant .. "-" .. trial .. " percentiles_ns" + + expect_true(n == 20000, label .. " has n=20000", "n=" .. tostring(n)) + expect_true(minv ~= nil and minv > 0 and p50 >= minv and p90 >= p50 and p99 >= p90 and + p999 >= p99 and maxv >= p999, + label .. " is monotonic (min<=p50<=p90<=p99<=p99.9<=max)", + string.format("min=%s p50=%s p90=%s p99=%s p99.9=%s max=%s", + tostring(minv), tostring(p50), tostring(p90), tostring(p99), tostring(p999), tostring(maxv))) + expect_true(mean_n ~= nil and mean_n > 0, label .. " mean parses as a positive float", tostring(mean)) + + if variant == "naive" or variant == "fastpath" then + table.insert(samples[variant], p50) + end +end +expect_true(matched == 6, + lang .. ": exactly 6 percentiles_ns lines present (naive-1..3, fastpath-1..3)", + "matched=" .. tostring(matched)) + +-- 7. Median-of-three per variant, conservative threshold, both medians +-- always printed (per the golden-capture calibration: observed ratio +-- 1.425x-1.488x across 3 independent runs, so a 0.90 factor -- a 10% +-- requirement -- has comfortable headroom against a noisier CI run without +-- being a tautology). +local function median3(t) + if #t ~= 3 then return nil end + local s = { t[1], t[2], t[3] } + table.sort(s) + return s[2] +end +local naive_med = median3(samples.naive) +local fast_med = median3(samples.fastpath) +print(string.format("%s: median(naive p50)=%s ns median(fastpath p50)=%s ns", + lang, tostring(naive_med), tostring(fast_med))) +expect_true(naive_med ~= nil and fast_med ~= nil and fast_med < naive_med * 0.90, + lang .. ": median(fastpath p50) < median(naive p50) * 0.90", + string.format("naive_median=%s fastpath_median=%s", tostring(naive_med), tostring(fast_med))) + +-- --------------------------------------------------------------------------- +-- 8. SIGINT shutdown, tested separately per server, with a bounded (10s) +-- wait so a Go server that hangs on a blocking read (SA_RESTART, no EINTR) +-- surfaces as a clear FAIL rather than a runner-level timeout. +-- --------------------------------------------------------------------------- + +local function sigint_test(args, port) + local script = table.concat({ + "OUT=/tmp/lsp40_sigint_" .. lang .. "_" .. port .. ".$$", + remote_bin .. " " .. args .. " >$OUT 2>&1 &", + "PID=$!", + "FOUND=0", + "for i in $(seq 1 50); do", + " if (exec 3<>\"/dev/tcp/127.0.0.1/" .. port .. "\") 2>/dev/null; then", + " exec 3<&- 2>/dev/null", + " exec 3>&- 2>/dev/null", + " FOUND=1", + " break", + " fi", + " sleep 0.1", + "done", + "if [ \"$FOUND\" != \"1\" ]; then", + " kill -9 $PID 2>/dev/null", + " cat $OUT", + " rm -f $OUT", + " echo SIGINT_EXIT=port_never_opened", + " exit 0", + "fi", + "kill -INT $PID", + "if timeout 10 tail --pid=$PID -f /dev/null 2>/dev/null; then", + " wait $PID", + " RC=$?", + "else", + " RC=124", + " kill -9 $PID 2>/dev/null", + " wait $PID 2>/dev/null", + "fi", + "cat $OUT", + "rm -f $OUT", + "echo SIGINT_EXIT=$RC", + }, "\n") + return ssh_run(script) +end + +-- naive: "app: naive shutting down", exit 0. +do + local r = sigint_test("naive --port 19801", 19801) + local rc = r.out:match("SIGINT_EXIT=(%S+)") + expect_true(rc == "0", + lang .. ": naive exits 0 after SIGINT (bounded 10s wait, catches a Go SA_RESTART hang)", + "SIGINT_EXIT=" .. tostring(rc)) + checks.expect_match(r.out, "app: naive shutting down", + lang .. ": naive prints the shutdown line on SIGINT") +end + +-- fastpath --busy-poll: the exact listening line, then +-- "app: fastpath shutting down", exit 0. +do + local r = sigint_test("fastpath --port 19802 --pin 0 --busy-poll", 19802) + local rc = r.out:match("SIGINT_EXIT=(%S+)") + expect_true(rc == "0", + lang .. ": fastpath --busy-poll exits 0 after SIGINT (bounded 10s wait, catches a Go SA_RESTART hang)", + "SIGINT_EXIT=" .. tostring(rc)) + checks.expect_match(r.out, "app: fastpath listening on 0%.0%.0%.0:%d+ pinned%-cpu=0 busy%-poll=on", + lang .. ": fastpath prints the exact listening line (pinned-cpu=0 busy-poll=on)") + checks.expect_match(r.out, "app: fastpath shutting down", + lang .. ": fastpath prints the shutdown line on SIGINT") +end + +-- --------------------------------------------------------------------------- +-- 9. Cleanup. +-- --------------------------------------------------------------------------- + +checks.run(string.format("ssh %s 'fedora@%s' 'rm -rf %s'", ssh_opts, ip, remote_dir)) + +checks.finish() diff --git a/examples/41-capstone-fleet/README.md b/examples/41-capstone-fleet/README.md new file mode 100644 index 0000000..39b523c --- /dev/null +++ b/examples/41-capstone-fleet/README.md @@ -0,0 +1,51 @@ +# _template — hello-syscall + +The seed for every runnable example in the book. It implements the same tiny +program three times — C++23, Go, and Rust — each printing exactly one line: + +``` +pid on () +``` + +via `getpid(2)` + `uname(2)` equivalents, with each language's idiomatic error +handling (`std::expected` / wrapped errors via `%w` / `rustix`). + +## Layout + +``` +_template/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build, demo.sh +├── go/ # go build, demo.sh +└── rust/ # cargo build, demo.sh +``` + +## The demo contract + +Every language directory has a `demo.sh` with the same interface, identical +across the whole book: + +- `./demo.sh` — build then run +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh "$TARGET" -- [args]` instead of + running locally. + +The top-level `demo.sh` here is only a dispatcher: `./demo.sh cpp run` execs +`cpp/demo.sh run`; bare `./demo.sh` (or `all`) builds and runs all three. + +## Verification + +`verify.lua` is run from this directory with `LSP_LANG` set to one language at +a time. It uses `scripts/lib/checks.lua` to run `./demo.sh $LSP_LANG run` and +asserts the output matches `pid %d+ on Linux`. Exit 0 = pass, 1 = fail, +77 = skip. + +## Mapping to a chapter + +Each chapter's example is a copy of this template (`scripts/new-example.sh +NN-slug`) with the program body replaced by that chapter's code. Keep the +demo contract and `verify.lua` shape intact so CI and the manifest +(`examples/manifest.yaml`) keep working. diff --git a/examples/41-capstone-fleet/cpp/.clang-format b/examples/41-capstone-fleet/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/41-capstone-fleet/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/41-capstone-fleet/cpp/.clang-tidy b/examples/41-capstone-fleet/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/41-capstone-fleet/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/41-capstone-fleet/cpp/CMakeLists.txt b/examples/41-capstone-fleet/cpp/CMakeLists.txt new file mode 100644 index 0000000..ce99d1c --- /dev/null +++ b/examples/41-capstone-fleet/cpp/CMakeLists.txt @@ -0,0 +1,28 @@ +cmake_minimum_required(VERSION 3.25) +project(capstone_fleet CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(opentelemetry-cpp CONFIG REQUIRED) +find_package(Threads REQUIRED) + +add_executable(app + src/main.cpp + src/proto.cpp + src/telemetry.cpp + src/caps.cpp + src/chatterd.cpp + src/sysagent.cpp + src/fwatch.cpp + src/pmon.cpp +) + +target_link_libraries(app PRIVATE + opentelemetry-cpp::trace + opentelemetry-cpp::metrics + opentelemetry-cpp::otlp_http_exporter + opentelemetry-cpp::otlp_http_metric_exporter + Threads::Threads +) diff --git a/examples/41-capstone-fleet/cpp/CMakePresets.json b/examples/41-capstone-fleet/cpp/CMakePresets.json new file mode 100644 index 0000000..bb52022 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/CMakePresets.json @@ -0,0 +1,63 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "toolchainFile": "${sourceDir}/build/release/conan_toolchain.cmake", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Release" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/41-capstone-fleet/cpp/conanfile.txt b/examples/41-capstone-fleet/cpp/conanfile.txt new file mode 100644 index 0000000..391f042 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/conanfile.txt @@ -0,0 +1,15 @@ +[requires] +opentelemetry-cpp/1.24.0 + +[options] +opentelemetry-cpp/*:shared=False +opentelemetry-cpp/*:with_otlp_http=True +opentelemetry-cpp/*:with_otlp_grpc=False +opentelemetry-cpp/*:with_zipkin=False +opentelemetry-cpp/*:with_prometheus=False +opentelemetry-cpp/*:with_elasticsearch=False +libcurl/*:with_ssl=False + +[generators] +CMakeDeps +CMakeToolchain diff --git a/examples/41-capstone-fleet/cpp/demo.sh b/examples/41-capstone-fleet/cpp/demo.sh new file mode 100755 index 0000000..70b341d --- /dev/null +++ b/examples/41-capstone-fleet/cpp/demo.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + conan install . --output-folder=build/release -s build_type=Release --build=missing + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/41-capstone-fleet/cpp/src/caps.cpp b/examples/41-capstone-fleet/cpp/src/caps.cpp new file mode 100644 index 0000000..b2f11d9 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/caps.cpp @@ -0,0 +1,31 @@ +// caps.cpp — PR_CAPBSET_DROP for every known capability, then +// PR_SET_NO_NEW_PRIVS. See the Go caps.go for the full rationale: this +// shrinks the bounding set inherited by every fork/exec descendant, whether +// or not the calling process currently holds any capability at all. +#include "caps.hpp" + +#include + +#include + +namespace caps { + +namespace { +constexpr int kLastKnownCap = 40; // CAP_CHECKPOINT_RESTORE, current as of 6.x +} + +void drop_bounding_set() { + int dropped = 0; + for (int c = 0; c <= kLastKnownCap; ++c) { + if (::prctl(PR_CAPBSET_DROP, c, 0, 0, 0) == 0) { + ++dropped; + } + } + if (::prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) != 0) { + std::fprintf(stderr, "pmon: prctl(PR_SET_NO_NEW_PRIVS) failed\n"); + return; + } + std::fprintf(stderr, "pmon: capabilities bounding_set_dropped=%d no_new_privs=1\n", dropped); +} + +} // namespace caps diff --git a/examples/41-capstone-fleet/cpp/src/caps.hpp b/examples/41-capstone-fleet/cpp/src/caps.hpp new file mode 100644 index 0000000..d9a91a4 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/caps.hpp @@ -0,0 +1,6 @@ +// caps.hpp — capability-bounding-set drop (see caps.cpp for rationale). +#pragma once + +namespace caps { +void drop_bounding_set(); +} diff --git a/examples/41-capstone-fleet/cpp/src/fwatch.hpp b/examples/41-capstone-fleet/cpp/src/fwatch.hpp new file mode 100644 index 0000000..e34a051 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/fwatch.hpp @@ -0,0 +1,12 @@ +// fwatch.hpp — the book's recurring file watcher, reduced to what the fleet +// needs (see the .cpp for the Landlock rationale carried over from ch33). +#pragma once + +#include + +namespace fwatch { + +int snapshot(const std::string& dir); +int watch(const std::string& dir, bool sandbox, int timeout_ms); + +} // namespace fwatch diff --git a/examples/41-capstone-fleet/cpp/src/main.cpp b/examples/41-capstone-fleet/cpp/src/main.cpp new file mode 100644 index 0000000..74a956e --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/main.cpp @@ -0,0 +1,32 @@ +// hello-syscall: print "pid on ()" via getpid(2) + uname(2). + +#include +#include +#include +#include +#include + +#include +#include + +namespace { + +[[nodiscard]] std::expected system_info() { + utsname info{}; + if (::uname(&info) != 0) { + return std::unexpected(std::error_code{errno, std::system_category()}); + } + return info; +} + +} // namespace + +int main() { + const auto info = system_info(); + if (!info) { + std::println(stderr, "uname failed: {}", info.error().message()); + return 1; + } + std::println("pid {} on {} {} ({})", ::getpid(), info->sysname, info->release, info->machine); + return 0; +} diff --git a/examples/41-capstone-fleet/cpp/src/proto.cpp b/examples/41-capstone-fleet/cpp/src/proto.cpp new file mode 100644 index 0000000..52596f9 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/proto.cpp @@ -0,0 +1,94 @@ +#include "proto.hpp" + +#include +#include + +namespace proto { + +namespace { + +[[nodiscard]] std::expected write_all(int fd, const std::uint8_t* data, std::size_t len) { + std::size_t off = 0; + while (off < len) { + ssize_t n = ::write(fd, data + off, len - off); + if (n < 0) { + if (errno == EINTR) continue; + return std::unexpected(std::error_code{errno, std::system_category()}); + } + if (n == 0) return std::unexpected(std::make_error_code(std::errc::io_error)); + off += static_cast(n); + } + return {}; +} + +[[nodiscard]] std::expected read_all(int fd, std::uint8_t* data, std::size_t len) { + std::size_t off = 0; + while (off < len) { + ssize_t n = ::read(fd, data + off, len - off); + if (n < 0) { + if (errno == EINTR) continue; + return std::unexpected(std::error_code{errno, std::system_category()}); + } + if (n == 0) return std::unexpected(std::make_error_code(std::errc::connection_reset)); + off += static_cast(n); + } + return {}; +} + +} // namespace + +std::expected write_frame(int fd, const Frame& f) { + if (f.payload.size() > 0xFFFF) { + return std::unexpected(std::make_error_code(std::errc::message_size)); + } + std::uint8_t hdr[6]; + hdr[0] = kMagic0; + hdr[1] = kMagic1; + hdr[2] = kVersion; + hdr[3] = f.type; + hdr[4] = static_cast((f.payload.size() >> 8) & 0xFF); + hdr[5] = static_cast(f.payload.size() & 0xFF); + if (auto r = write_all(fd, hdr, sizeof(hdr)); !r) return r; + if (!f.payload.empty()) { + if (auto r = write_all(fd, f.payload.data(), f.payload.size()); !r) return r; + } + return {}; +} + +std::expected read_frame(int fd) { + std::uint8_t hdr[6]; + if (auto r = read_all(fd, hdr, sizeof(hdr)); !r) return std::unexpected(r.error()); + if (hdr[0] != kMagic0 || hdr[1] != kMagic1 || hdr[2] != kVersion) { + return std::unexpected(std::make_error_code(std::errc::protocol_error)); + } + Frame f; + f.type = hdr[3]; + std::size_t len = (static_cast(hdr[4]) << 8) | hdr[5]; + f.payload.resize(len); + if (len > 0) { + if (auto r = read_all(fd, f.payload.data(), len); !r) return std::unexpected(r.error()); + } + return f; +} + +std::vector deliver_payload(const std::string& nick, const std::string& text) { + std::vector out; + out.reserve(nick.size() + 1 + text.size()); + out.insert(out.end(), nick.begin(), nick.end()); + out.push_back(0); + out.insert(out.end(), text.begin(), text.end()); + return out; +} + +bool split_deliver(const std::vector& payload, std::string& nick, std::string& text) { + for (std::size_t i = 0; i < payload.size(); ++i) { + if (payload[i] == 0) { + nick.assign(payload.begin(), payload.begin() + static_cast(i)); + text.assign(payload.begin() + static_cast(i) + 1, payload.end()); + return true; + } + } + return false; +} + +} // namespace proto diff --git a/examples/41-capstone-fleet/cpp/src/proto.hpp b/examples/41-capstone-fleet/cpp/src/proto.hpp new file mode 100644 index 0000000..d3070ee --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/proto.hpp @@ -0,0 +1,40 @@ +// proto.hpp — the canonical chatterd chat frame (ch21, unchanged since): +// magic "CH", version 1, one byte type, big-endian u16 length, then payload. +#pragma once + +#include +#include +#include +#include +#include + +namespace proto { + +constexpr std::uint8_t kMagic0 = 'C'; +constexpr std::uint8_t kMagic1 = 'H'; +constexpr std::uint8_t kVersion = 1; + +constexpr std::uint8_t kJoin = 1; +constexpr std::uint8_t kMsg = 2; +constexpr std::uint8_t kDeliver = 3; + +struct Frame { + std::uint8_t type{}; + std::vector payload; +}; + +// Writes the 6-byte header + payload to fd. Returns an error_code on a +// short/failed write. +[[nodiscard]] std::expected write_frame(int fd, const Frame& f); + +// Reads one frame from fd (blocking on a normal, non-nonblocking socket). +// Returns an error_code on EOF/short-read/bad magic. +[[nodiscard]] std::expected read_frame(int fd); + +// "nick NUL text" DELIVER payload (ch21). +std::vector deliver_payload(const std::string& nick, const std::string& text); + +// Splits "nick NUL text"; returns false if there's no NUL. +bool split_deliver(const std::vector& payload, std::string& nick, std::string& text); + +} // namespace proto diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.cpp b/examples/41-capstone-fleet/cpp/src/telemetry.cpp new file mode 100644 index 0000000..f8bd439 --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/telemetry.cpp @@ -0,0 +1,113 @@ +#include "telemetry.hpp" + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace telemetry { + +std::atomic export_errors{0}; + +namespace { + +namespace trace_api = opentelemetry::trace; +namespace trace_sdk = opentelemetry::sdk::trace; +namespace metrics_api = opentelemetry::metrics; +namespace metrics_sdk = opentelemetry::sdk::metrics; +namespace resource_sdk = opentelemetry::sdk::resource; +namespace otlp = opentelemetry::exporter::otlp; + +// Counts every Error-level internal log line (the SDK's own signal that an +// export attempt did not get a 2xx / hit a transport failure) instead of +// crashing a supervised service over a telemetry hiccup. +class CountingLogHandler : public opentelemetry::sdk::common::internal_log::LogHandler { +public: + void Handle(opentelemetry::sdk::common::internal_log::LogLevel level, const char* file, int line, + const char* msg, const opentelemetry::sdk::common::AttributeMap& attrs) noexcept override { + using opentelemetry::sdk::common::internal_log::LogLevel; + if (level == LogLevel::Error) { + export_errors.fetch_add(1); + std::fprintf(stderr, "otel: export error: %s\n", msg); + } + (void)file; + (void)line; + (void)attrs; + } +}; + +std::shared_ptr g_tracer_provider; +std::shared_ptr g_meter_provider; + +} // namespace + +Handle::~Handle() = default; + +Handle init(const std::string& service, const std::string& node) { + const char* endpoint = std::getenv("OTEL_EXPORTER_OTLP_ENDPOINT"); + if (endpoint == nullptr || endpoint[0] == '\0') { + std::fprintf(stderr, "%s: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)\n", service.c_str()); + Handle h; + h.enabled = false; + h.tracer = trace_api::Provider::GetTracerProvider()->GetTracer(service); + h.meter = metrics_api::Provider::GetMeterProvider()->GetMeter(service); + return h; + } + + using namespace opentelemetry::sdk::common::internal_log; + GlobalLogHandler::SetLogHandler( + opentelemetry::nostd::shared_ptr(new CountingLogHandler())); + GlobalLogHandler::SetLogLevel(LogLevel::Error); + + auto resource = resource_sdk::Resource::Create({ + {"service.name", service}, + {"service.version", "41-capstone-fleet"}, + {"deployment.environment", "lab"}, + {"node.name", node}, + }); + + otlp::OtlpHttpExporterOptions trace_opts; + trace_opts.url = std::string(endpoint) + "/v1/traces"; + auto trace_exporter = otlp::OtlpHttpExporterFactory::Create(trace_opts); + trace_sdk::BatchSpanProcessorOptions batch_opts; + batch_opts.schedule_delay_millis = std::chrono::milliseconds(500); + auto processor = trace_sdk::BatchSpanProcessorFactory::Create(std::move(trace_exporter), batch_opts); + g_tracer_provider = trace_sdk::TracerProviderFactory::Create(std::move(processor), resource); + trace_api::Provider::SetTracerProvider(g_tracer_provider); + + otlp::OtlpHttpMetricExporterOptions metric_opts; + metric_opts.url = std::string(endpoint) + "/v1/metrics"; + auto metric_exporter = otlp::OtlpHttpMetricExporterFactory::Create(metric_opts); + metrics_sdk::PeriodicExportingMetricReaderOptions reader_opts; + reader_opts.export_interval_millis = std::chrono::milliseconds(2000); + reader_opts.export_timeout_millis = std::chrono::milliseconds(1000); + auto reader = std::make_shared(std::move(metric_exporter), reader_opts); + auto mp = metrics_sdk::MeterProviderFactory::Create( + std::make_unique(), resource); + static_cast(mp.get())->AddMetricReader(reader); + g_meter_provider = std::shared_ptr(std::move(mp)); + metrics_api::Provider::SetMeterProvider(g_meter_provider); + + std::fprintf(stderr, "%s: otel enabled endpoint=%s node=%s\n", service.c_str(), endpoint, node.c_str()); + + Handle h; + h.enabled = true; + h.tracer = trace_api::Provider::GetTracerProvider()->GetTracer(service); + h.meter = metrics_api::Provider::GetMeterProvider()->GetMeter(service); + return h; +} + +} // namespace telemetry diff --git a/examples/41-capstone-fleet/cpp/src/telemetry.hpp b/examples/41-capstone-fleet/cpp/src/telemetry.hpp new file mode 100644 index 0000000..fda667c --- /dev/null +++ b/examples/41-capstone-fleet/cpp/src/telemetry.hpp @@ -0,0 +1,40 @@ +// telemetry.hpp — OTel wiring shared by chatterd and sysagent, following the +// book's house C++ pattern (opentelemetry-cpp via Conan, OTLP/HTTP). Endpoint +// comes from the standard OTEL_EXPORTER_OTLP_ENDPOINT env var, which the +// exporters read themselves — the same variable +// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT. +#pragma once + +#include +#include +#include + +#include +#include + +namespace telemetry { + +// Count of OTel export failures observed via the global error/exception +// handler, the deterministic stand-in for "the exporter got a 200" (the +// batch/periodic exporters only invoke it on a non-2xx response or a +// transport failure). +extern std::atomic export_errors; + +struct Handle { + bool enabled = false; + std::shared_ptr tracer; + std::shared_ptr meter; + + ~Handle(); + Handle() = default; + Handle(const Handle&) = delete; + Handle& operator=(const Handle&) = delete; + Handle(Handle&&) = default; + Handle& operator=(Handle&&) = default; +}; + +// init sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT is set; +// otherwise returns a disabled, no-op handle (the service still runs). +Handle init(const std::string& service, const std::string& node); + +} // namespace telemetry diff --git a/examples/41-capstone-fleet/demo.sh b/examples/41-capstone-fleet/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/41-capstone-fleet/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/41-capstone-fleet/go/caps.go b/examples/41-capstone-fleet/go/caps.go new file mode 100644 index 0000000..1023936 --- /dev/null +++ b/examples/41-capstone-fleet/go/caps.go @@ -0,0 +1,38 @@ +// caps.go — capability-bounding-set drop, the cheap, always-available half of +// this chapter's sandboxing story (the other half, Landlock, lives in +// fwatch.go and was fully proven with seccomp alongside it in ch33; this +// capstone does not re-derive the seccomp BPF program, only reuses the +// mechanism it already established). +// +// PR_CAPBSET_DROP removes a capability from the *bounding set*: even a future +// execve of a setuid-root binary can never regain it, for this process and +// every descendant (the bounding set is inherited across fork/exec and can +// only shrink). It is meaningful regardless of whether the calling process +// currently holds any capabilities at all — that is exactly pmon's case when +// deployed unprivileged on a lab guest. +package main + +import ( + "fmt" + "os" + + "golang.org/x/sys/unix" +) + +// lastKnownCap is CAP_CHECKPOINT_RESTORE (40) on a 6.x+ kernel; probing past +// the kernel's actual last cap just returns EINVAL, which we ignore. +const lastKnownCap = 40 + +func dropCapabilityBoundingSet() { + dropped := 0 + for c := 0; c <= lastKnownCap; c++ { + if err := unix.Prctl(unix.PR_CAPBSET_DROP, uintptr(c), 0, 0, 0); err == nil { + dropped++ + } + } + if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil { + fmt.Fprintf(os.Stderr, "pmon: prctl(PR_SET_NO_NEW_PRIVS): %v\n", err) + return + } + fmt.Fprintf(os.Stderr, "pmon: capabilities bounding_set_dropped=%d no_new_privs=1\n", dropped) +} diff --git a/examples/41-capstone-fleet/go/chatterd.go b/examples/41-capstone-fleet/go/chatterd.go new file mode 100644 index 0000000..311b4ca --- /dev/null +++ b/examples/41-capstone-fleet/go/chatterd.go @@ -0,0 +1,264 @@ +// chatterd.go — the book's recurring peer-to-peer chat daemon (ch21-27), +// reduced to what the capstone needs: serve local clients, and (new in this +// chapter) bridge two chatterd instances across hosts so a message sent on +// one node is delivered to clients on the other. The wire frame (proto.go) +// is unchanged from every prior chatterd chapter. +// +// Bridging design: a bridge worker is just an ordinary client of the *other* +// node's server (nick "bridge@"). Concretely, with --peer set on +// both sides: +// +// target dials peer, joins as "bridge@target" (this connection lives in +// peer's client registry) +// peer dials target, joins as "bridge@peer" (lives in target's registry) +// +// A local MSG is broadcast as DELIVER to every registered connection +// (including a bridge connection dialed in from the other side) exactly like +// any other client — no protocol change. When a bridge *worker* itself +// receives a DELIVER over the connection it dialed, it re-broadcasts locally +// with includeBridges=false, so the message never goes back out over any +// bridge: no ping-pong between two nodes. +package main + +import ( + "context" + "fmt" + "net" + "os" + "strconv" + "strings" + "sync" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" +) + +type peerConn struct { + conn net.Conn + nick string + isBridge bool + wmu sync.Mutex +} + +func (c *peerConn) send(f frame) error { + c.wmu.Lock() + defer c.wmu.Unlock() + return writeFrame(c.conn, f) +} + +type hub struct { + mu sync.Mutex + clients map[*peerConn]struct{} +} + +func newHub() *hub { return &hub{clients: make(map[*peerConn]struct{})} } + +func (h *hub) add(c *peerConn) { + h.mu.Lock() + h.clients[c] = struct{}{} + h.mu.Unlock() +} + +func (h *hub) remove(c *peerConn) { + h.mu.Lock() + delete(h.clients, c) + h.mu.Unlock() +} + +func (h *hub) broadcastDeliver(exclude *peerConn, includeBridges bool, nick, text string) { + f := frame{typ: typeDeliver, payload: deliverPayload(nick, text)} + h.mu.Lock() + targets := make([]*peerConn, 0, len(h.clients)) + for c := range h.clients { + if c == exclude { + continue + } + if c.isBridge && !includeBridges { + continue + } + targets = append(targets, c) + } + h.mu.Unlock() + for _, c := range targets { + _ = c.send(f) + } +} + +func chatterdServe(host string, port int, node, peerAddr, peerNode string, tel *telemetry) int { + addr := net.JoinHostPort(host, strconv.Itoa(port)) + ln, err := net.Listen("tcp", addr) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: listen %s: %v\n", addr, err) + return 1 + } + fmt.Fprintf(os.Stderr, "chatterd: listening on %s node=%s\n", addr, node) + + h := newHub() + sig := installSignalFlag() + + go func() { + for !sig.Load() { + time.Sleep(50 * time.Millisecond) + } + fmt.Fprintln(os.Stderr, "chatterd: shutdown") + ln.Close() + }() + + if peerAddr != "" { + if peerNode == "" { + peerNode = "remote" + } + go bridgeWorker(peerAddr, node, peerNode, h, sig) + } + + for { + conn, err := ln.Accept() + if err != nil { + if sig.Load() { + return 0 + } + continue + } + go handleClient(conn, h, node, tel) + } +} + +func handleClient(conn net.Conn, h *hub, node string, tel *telemetry) { + pc := &peerConn{conn: conn} + defer func() { + h.remove(pc) + conn.Close() + }() + + first, err := readFrame(conn) + if err != nil || first.typ != typeJoin { + return + } + pc.nick = string(first.payload) + pc.isBridge = strings.HasPrefix(pc.nick, "bridge@") + h.add(pc) + + for { + f, err := readFrame(conn) + if err != nil { + return + } + if f.typ != typeMsg { + continue + } + text := string(f.payload) + + ctx, span := tel.tracer.Start(context.Background(), "chatterd.deliver") + span.SetAttributes( + attribute.String("chat.from", pc.nick), + attribute.Int("chat.text_len", len(text)), + attribute.String("chat.node", node), + attribute.Bool("chat.from_bridge", pc.isBridge), + ) + h.broadcastDeliver(pc, true, pc.nick, text) + span.SetStatus(codes.Ok, "") + if tel.enabled { + fmt.Printf("chatterd: trace_id=%s node=%s from=%s\n", + span.SpanContext().TraceID().String(), node, pc.nick) + } + span.End() + _ = ctx + } +} + +func bridgeWorker(peerAddr, localNode, peerNode string, h *hub, sig interface{ Load() bool }) { + backoff := 300 * time.Millisecond + for !sig.Load() { + conn, err := net.DialTimeout("tcp", peerAddr, 3*time.Second) + if err != nil { + time.Sleep(backoff) + continue + } + bridgeNick := "bridge@" + localNode + if err := writeFrame(conn, frame{typ: typeJoin, payload: []byte(bridgeNick)}); err != nil { + conn.Close() + time.Sleep(backoff) + continue + } + fmt.Fprintf(os.Stderr, "chatterd: bridge connected peer=%s as=%s\n", peerAddr, bridgeNick) + + for { + f, err := readFrame(conn) + if err != nil { + break + } + if f.typ != typeDeliver { + continue + } + nick, text, ok := splitDeliver(f.payload) + if !ok { + continue + } + if !strings.Contains(nick, "@") { + nick = nick + "@" + peerNode + } + h.broadcastDeliver(nil, false, nick, text) + } + conn.Close() + fmt.Fprintf(os.Stderr, "chatterd: bridge disconnected peer=%s (retrying)\n", peerAddr) + time.Sleep(backoff) + } +} + +// --------------------------------------------------------------------------- +// send / listen — minimal test clients used by verify to prove cross-host +// delivery deterministically (this capstone trims ch27's fuller "loadtest"). +// --------------------------------------------------------------------------- + +func chatterdSend(host string, port int, nick, text string, timeoutMs int) int { + addr := net.JoinHostPort(host, strconv.Itoa(port)) + conn, err := net.DialTimeout("tcp", addr, time.Duration(timeoutMs)*time.Millisecond) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: send: dial %s: %v\n", addr, err) + return 1 + } + defer conn.Close() + if err := writeFrame(conn, frame{typ: typeJoin, payload: []byte(nick)}); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: send: join: %v\n", err) + return 1 + } + if err := writeFrame(conn, frame{typ: typeMsg, payload: []byte(text)}); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: send: msg: %v\n", err) + return 1 + } + time.Sleep(150 * time.Millisecond) // let the write actually flush before closing + fmt.Printf("chatterd: sent nick=%s text=%s\n", nick, text) + return 0 +} + +func chatterdListen(host string, port int, nick string, timeoutMs int) int { + addr := net.JoinHostPort(host, strconv.Itoa(port)) + conn, err := net.DialTimeout("tcp", addr, 3*time.Second) + if err != nil { + fmt.Fprintf(os.Stderr, "chatterd: listen: dial %s: %v\n", addr, err) + return 1 + } + defer conn.Close() + if err := writeFrame(conn, frame{typ: typeJoin, payload: []byte(nick)}); err != nil { + fmt.Fprintf(os.Stderr, "chatterd: listen: join: %v\n", err) + return 1 + } + conn.SetReadDeadline(time.Now().Add(time.Duration(timeoutMs) * time.Millisecond)) + for { + f, err := readFrame(conn) + if err != nil { + fmt.Fprintln(os.Stderr, "chatterd: listen timeout") + return 1 + } + if f.typ != typeDeliver { + continue + } + fromNick, text, ok := splitDeliver(f.payload) + if !ok { + continue + } + fmt.Printf("chatterd: received from=%s text=%s\n", fromNick, text) + return 0 + } +} diff --git a/examples/41-capstone-fleet/go/demo.sh b/examples/41-capstone-fleet/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/41-capstone-fleet/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/41-capstone-fleet/go/fwatch.go b/examples/41-capstone-fleet/go/fwatch.go new file mode 100644 index 0000000..b6cea42 --- /dev/null +++ b/examples/41-capstone-fleet/go/fwatch.go @@ -0,0 +1,212 @@ +// fwatch.go — the book's recurring file watcher (ch07 polling -> ch09 +// inotify/epoll -> ch33 Landlock+seccomp), reduced here to what the fleet +// needs: watch a directory, print one line per create/write/delete event, +// and optionally run under a Landlock ruleset that restricts reads to that +// directory. Full seccomp allow-listing was proven once in ch33 and is not +// re-derived here; capability-bounding-set drop (caps.go) plus Landlock are +// this capstone's sandboxing layers, applied by pmon before it execs fwatch. +package main + +import ( + "fmt" + "os" + "path/filepath" + "time" + "unsafe" + + "golang.org/x/sys/unix" +) + +// --------------------------------------------------------------------------- +// Landlock — raw syscalls; x/sys/unix has the SYS_LANDLOCK_* numbers but no +// wrapper. Read-only ruleset scoped to one directory (ch33's technique). +// --------------------------------------------------------------------------- + +const ( + landlockAccessFSReadFile = 1 << 2 + landlockAccessFSReadDir = 1 << 3 + + landlockCreateRulesetVersion = 1 << 0 + landlockRulePathBeneath = 1 + landlockRestrictSelfTSYNC = 1 << 3 +) + +type landlockRulesetAttr struct { + HandledAccessFS uint64 + HandledAccessNet uint64 + Scoped uint64 +} + +type landlockPathBeneathAttr struct { + AllowedAccess uint64 + ParentFd int32 + _ [4]byte +} + +func landlockABI() int { + v, _, errno := unix.Syscall(unix.SYS_LANDLOCK_CREATE_RULESET, 0, 0, + uintptr(landlockCreateRulesetVersion)) + if errno != 0 { + return 0 + } + return int(v) +} + +func applyLandlock(dir string) error { + attr := landlockRulesetAttr{ + HandledAccessFS: landlockAccessFSReadFile | landlockAccessFSReadDir, + } + rs, _, errno := unix.Syscall(unix.SYS_LANDLOCK_CREATE_RULESET, + uintptr(unsafe.Pointer(&attr)), unsafe.Sizeof(attr), 0) + if errno != 0 { + return fmt.Errorf("landlock_create_ruleset: %w", errno) + } + rulesetFd := int(rs) + defer unix.Close(rulesetFd) + + dirFd, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0) + if err != nil { + return fmt.Errorf("open %s: %w", dir, err) + } + pb := landlockPathBeneathAttr{ + AllowedAccess: landlockAccessFSReadFile | landlockAccessFSReadDir, + ParentFd: int32(dirFd), + } + _, _, errno = unix.Syscall6(unix.SYS_LANDLOCK_ADD_RULE, uintptr(rulesetFd), + uintptr(landlockRulePathBeneath), uintptr(unsafe.Pointer(&pb)), 0, 0, 0) + unix.Close(dirFd) + if errno != 0 { + return fmt.Errorf("landlock_add_rule: %w", errno) + } + + if err := unix.Prctl(unix.PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0); err != nil { + return fmt.Errorf("prctl(PR_SET_NO_NEW_PRIVS): %w", err) + } + _, _, errno = unix.Syscall(unix.SYS_LANDLOCK_RESTRICT_SELF, uintptr(rulesetFd), + uintptr(landlockRestrictSelfTSYNC), 0) + if errno != 0 { + _, _, errno = unix.Syscall(unix.SYS_LANDLOCK_RESTRICT_SELF, uintptr(rulesetFd), 0, 0) + if errno != 0 { + return fmt.Errorf("landlock_restrict_self: %w", errno) + } + } + return nil +} + +// --------------------------------------------------------------------------- +// watch loop — inotify on the directory (non-recursive, matches ch09). +// --------------------------------------------------------------------------- + +func fwatchWatch(dir string, sandbox bool, timeoutMs int) int { + if sandbox { + abi := landlockABI() + if abi == 0 { + fmt.Fprintln(os.Stderr, "fwatch: Landlock not supported by this kernel") + return 1 + } + if err := applyLandlock(dir); err != nil { + fmt.Fprintf(os.Stderr, "fwatch: landlock: %v\n", err) + return 1 + } + fmt.Fprintf(os.Stderr, "fwatch: landlock ABI=%d enforced dir=%s\n", abi, dir) + } + + fd, err := unix.InotifyInit1(unix.IN_CLOEXEC | unix.IN_NONBLOCK) + if err != nil { + fmt.Fprintf(os.Stderr, "fwatch: inotify_init1: %v\n", err) + return 1 + } + defer unix.Close(fd) + + mask := uint32(unix.IN_CREATE | unix.IN_MODIFY | unix.IN_DELETE | unix.IN_CLOSE_WRITE) + if _, err := unix.InotifyAddWatch(fd, dir, mask); err != nil { + fmt.Fprintf(os.Stderr, "fwatch: inotify_add_watch %s: %v\n", dir, err) + return 1 + } + + fmt.Fprintf(os.Stderr, "fwatch: watching %s (sandbox=%v)\n", dir, sandbox) + + deadline := time.Time{} + if timeoutMs > 0 { + deadline = time.Now().Add(time.Duration(timeoutMs) * time.Millisecond) + } + sig := installSignalFlag() + + buf := make([]byte, 4096) + for { + if sig.Load() { + fmt.Println("(signal)") + return 0 + } + if !deadline.IsZero() && time.Now().After(deadline) { + fmt.Println("(timeout)") + return 0 + } + pfd := []unix.PollFd{{Fd: int32(fd), Events: unix.POLLIN}} + n, err := unix.Poll(pfd, 200) + if err != nil { + if err == unix.EINTR { + continue + } + fmt.Fprintf(os.Stderr, "fwatch: poll: %v\n", err) + return 1 + } + if n == 0 { + continue + } + nr, err := unix.Read(fd, buf) + if err != nil || nr <= 0 { + continue + } + off := 0 + for off+16 <= nr { + nameLen := int(le32(buf[off+12 : off+16])) + maskv := le32(buf[off+4 : off+8]) + name := "" + if nameLen > 0 { + end := off + 16 + nameLen + for i := off + 16; i < end; i++ { + if buf[i] == 0 { + name = string(buf[off+16 : i]) + break + } + } + } + fmt.Printf("event: %s %s\n", eventName(maskv), filepath.Join(dir, name)) + off += 16 + nameLen + } + } +} + +func le32(b []byte) uint32 { + return uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24 +} + +func eventName(mask uint32) string { + switch { + case mask&unix.IN_CREATE != 0: + return "create" + case mask&unix.IN_DELETE != 0: + return "delete" + case mask&unix.IN_CLOSE_WRITE != 0: + return "modify" + case mask&unix.IN_MODIFY != 0: + return "modify" + default: + return "other" + } +} + +func fwatchSnapshot(dir string) int { + entries, err := os.ReadDir(dir) + if err != nil { + fmt.Fprintf(os.Stderr, "fwatch: readdir %s: %v\n", dir, err) + return 1 + } + for _, e := range entries { + if e.Type().IsRegular() { + fmt.Println(e.Name()) + } + } + return 0 +} diff --git a/examples/41-capstone-fleet/go/go.mod b/examples/41-capstone-fleet/go/go.mod new file mode 100644 index 0000000..8dcf908 --- /dev/null +++ b/examples/41-capstone-fleet/go/go.mod @@ -0,0 +1,32 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/41-capstone-fleet/go + +go 1.26 + +toolchain go1.26.5 + +require golang.org/x/sys v0.47.0 + +require ( + github.com/cenkalti/backoff/v5 v5.0.3 // indirect + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/otel v1.44.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0 // indirect + go.opentelemetry.io/otel/metric v1.44.0 // indirect + go.opentelemetry.io/otel/sdk v1.44.0 // indirect + go.opentelemetry.io/otel/sdk/metric v1.44.0 // indirect + go.opentelemetry.io/otel/trace v1.44.0 // indirect + go.opentelemetry.io/proto/otlp v1.10.0 // indirect + golang.org/x/net v0.55.0 // indirect + golang.org/x/text v0.37.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/grpc v1.81.1 // indirect + google.golang.org/protobuf v1.36.11 // indirect +) diff --git a/examples/41-capstone-fleet/go/go.sum b/examples/41-capstone-fleet/go/go.sum new file mode 100644 index 0000000..682e2fb --- /dev/null +++ b/examples/41-capstone-fleet/go/go.sum @@ -0,0 +1,47 @@ +github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM= +github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 h1:5VipnvEpbqr2gA2VbM+nYVbkIF28c5ZQfqCBQ5g2xfk= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0/go.mod h1:Hyl3n6Twe1hvtd9XUXDec4pTvgMSEixRuQKPTMH2bNs= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU= +go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0 h1:RuynHbfU8JUEw7DyONgkVYg2SVtsoF28y0LGIr69jgA= +go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.44.0/go.mod h1:qZF+/lBs71APw8mlnEZcqZHMzqrYrsFiJOv83lX1OGo= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 h1:4YsVu3B8+3qtWYYrsUYgn0OG78pN0rnNPRGX4SbokQI= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0/go.mod h1:+wnlSn0mD1ADVMe3v9Z/WIaiz6q6gL2J/ejaAmdmv80= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0 h1:lgh3PiVrRUWMLOVSkQicxzZll5NjF1r+AtsX1XRIHw0= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.44.0/go.mod h1:5Cnhth3m/AgOeTgE3ex12pPmiu/gGtZit03kSzx9X7s= +go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc= +go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo= +go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58= +go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0= +go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI= +go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= +go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk= +go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE= +go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g= +go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk= +golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= +golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= +golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1:Kjn0N0tCrDgiAFW+lGO4JZ3ck44CehvJQMAwj9QF0G8= +google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ= +google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= diff --git a/examples/41-capstone-fleet/go/main.go b/examples/41-capstone-fleet/go/main.go new file mode 100644 index 0000000..d79f55a --- /dev/null +++ b/examples/41-capstone-fleet/go/main.go @@ -0,0 +1,192 @@ +// 41-capstone-fleet — pmon supervises chatterd + sysagent + fwatch across +// two lab hosts, capability-dropped and Landlock-sandboxed, all telemetry +// exported to the host LGTM stack. One binary, four subcommands (the same +// self-re-exec shape ch34's container entrypoint uses): pmon is the fleet's +// init; chatterd/sysagent/fwatch are the services it supervises. +package main + +import ( + "context" + "fmt" + "os" + "strconv" +) + +func usage() { + fmt.Fprintln(os.Stderr, `usage: app + pmon [--node NAME] [--sandbox-dir DIR] [--peer HOST:PORT] [--peer-node NAME] + [--chatterd-port P] [--health-interval-ms N] + chatterd serve [--host H] [--port P] [--node NAME] [--peer HOST:PORT] [--peer-node NAME] + chatterd send --host H --port P --nick NICK --text TEXT [--timeout-ms T] + chatterd listen --host H --port P --nick NICK [--timeout-ms T] + sysagent [--node NAME] [--interval-ms N] [--once] + sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M] + fwatch snapshot DIR + fwatch watch DIR [--sandbox] [--timeout-ms T]`) +} + +// argFlag scans args for "--name value" and returns value (and true), or def. +func argFlag(args []string, name, def string) (string, bool) { + for i, a := range args { + if a == name { + if i+1 < len(args) { + return args[i+1], true + } + return "", false + } + } + return def, true +} + +func hasFlag(args []string, name string) bool { + for _, a := range args { + if a == name { + return true + } + } + return false +} + +func firstPositional(args []string) string { + skip := false + for _, a := range args { + if skip { + skip = false + continue + } + if len(a) >= 2 && a[:2] == "--" { + skip = true + continue + } + return a + } + return "" +} + +func atoiOr(s string, def int) int { + if s == "" { + return def + } + v, err := strconv.Atoi(s) + if err != nil { + return def + } + return v +} + +func main() { + if len(os.Args) < 2 { + usage() + os.Exit(2) + } + cmd := os.Args[1] + rest := os.Args[2:] + + switch cmd { + case "pmon": + node, ok1 := argFlag(rest, "--node", "node1") + sandboxDir, ok2 := argFlag(rest, "--sandbox-dir", "/tmp/fwatch-sandbox") + peer, ok3 := argFlag(rest, "--peer", "") + peerNode, ok4 := argFlag(rest, "--peer-node", "") + portS, ok5 := argFlag(rest, "--chatterd-port", "47100") + healthS, ok6 := argFlag(rest, "--health-interval-ms", "2000") + if !ok1 || !ok2 || !ok3 || !ok4 || !ok5 || !ok6 { + usage() + os.Exit(2) + } + os.Exit(pmonRun(node, sandboxDir, peer, peerNode, atoiOr(portS, 47100), atoiOr(healthS, 2000))) + + case "chatterd": + if len(rest) == 0 { + usage() + os.Exit(2) + } + switch rest[0] { + case "serve": + args := rest[1:] + host, _ := argFlag(args, "--host", "0.0.0.0") + portS, _ := argFlag(args, "--port", "47100") + node, _ := argFlag(args, "--node", "node1") + peer, _ := argFlag(args, "--peer", "") + peerNode, _ := argFlag(args, "--peer-node", "") + tel := initTelemetry(context.Background(), "chatterd", node) + defer tel.shutdown(context.Background()) + os.Exit(chatterdServe(host, atoiOr(portS, 47100), node, peer, peerNode, tel)) + case "send": + args := rest[1:] + host, _ := argFlag(args, "--host", "127.0.0.1") + portS, _ := argFlag(args, "--port", "47100") + nick, ok1 := argFlag(args, "--nick", "") + text, ok2 := argFlag(args, "--text", "") + timeoutS, _ := argFlag(args, "--timeout-ms", "3000") + if !ok1 || !ok2 || nick == "" || text == "" { + usage() + os.Exit(2) + } + os.Exit(chatterdSend(host, atoiOr(portS, 47100), nick, text, atoiOr(timeoutS, 3000))) + case "listen": + args := rest[1:] + host, _ := argFlag(args, "--host", "127.0.0.1") + portS, _ := argFlag(args, "--port", "47100") + nick, ok1 := argFlag(args, "--nick", "") + timeoutS, _ := argFlag(args, "--timeout-ms", "5000") + if !ok1 || nick == "" { + usage() + os.Exit(2) + } + os.Exit(chatterdListen(host, atoiOr(portS, 47100), nick, atoiOr(timeoutS, 5000))) + default: + usage() + os.Exit(2) + } + + case "sysagent": + if len(rest) > 0 && rest[0] == "saturate" { + args := rest[1:] + resource, _ := argFlag(args, "--resource", "") + secondsS, _ := argFlag(args, "--seconds", "10") + workersS, _ := argFlag(args, "--workers", "0") + mbS, _ := argFlag(args, "--mb", "0") + os.Exit(saturate(resource, atoiOr(secondsS, 10), atoiOr(workersS, 0), atoiOr(mbS, 0))) + } + node, _ := argFlag(rest, "--node", "node1") + intervalS, _ := argFlag(rest, "--interval-ms", "2000") + once := hasFlag(rest, "--once") + tel := initTelemetry(context.Background(), "sysagent", node) + defer tel.shutdown(context.Background()) + os.Exit(sysagentRun(node, atoiOr(intervalS, 2000), once, tel)) + + case "fwatch": + if len(rest) == 0 { + usage() + os.Exit(2) + } + switch rest[0] { + case "snapshot": + args := rest[1:] + dir := firstPositional(args) + if dir == "" { + usage() + os.Exit(2) + } + os.Exit(fwatchSnapshot(dir)) + case "watch": + args := rest[1:] + dir := firstPositional(args) + if dir == "" { + usage() + os.Exit(2) + } + sandbox := hasFlag(args, "--sandbox") + timeoutS, _ := argFlag(args, "--timeout-ms", "0") + os.Exit(fwatchWatch(dir, sandbox, atoiOr(timeoutS, 0))) + default: + usage() + os.Exit(2) + } + + default: + usage() + os.Exit(2) + } +} diff --git a/examples/41-capstone-fleet/go/pmon.go b/examples/41-capstone-fleet/go/pmon.go new file mode 100644 index 0000000..9f4e320 --- /dev/null +++ b/examples/41-capstone-fleet/go/pmon.go @@ -0,0 +1,158 @@ +// pmon.go — the book's recurring process supervisor (ch11-14, 18-19, 32, +// 34), grown here into the capstone's fleet init: it drops the capability +// bounding set once, then fork/execs (self-re-exec, same technique ch34's +// container entrypoint uses) chatterd, sysagent, and fwatch as children, +// restarting any of them that exits unexpectedly, forwarding SIGTERM/SIGINT +// to all three on its own shutdown, and printing a health line every tick so +// an operator (or verify.lua, over ssh) can poll fleet state without +// touching /proc. +package main + +import ( + "fmt" + "os" + "os/exec" + "os/signal" + "path/filepath" + "strconv" + "sync" + "sync/atomic" + "syscall" + "time" +) + +type childSpec struct { + name string + args []string +} + +type childState struct { + state atomic.Value // string: "starting"|"up"|"restarting"|"down" + restarts atomic.Int64 + pid atomic.Int64 +} + +func pmonRun(node, sandboxDir, peer, peerNode string, chatterdPort int, healthIntervalMs int) int { + if err := os.MkdirAll(sandboxDir, 0o755); err != nil { + fmt.Fprintf(os.Stderr, "pmon: mkdir %s: %v\n", sandboxDir, err) + return 1 + } + + dropCapabilityBoundingSet() + + selfExe, err := os.Executable() + if err != nil { + fmt.Fprintf(os.Stderr, "pmon: os.Executable: %v\n", err) + return 1 + } + selfExe, _ = filepath.Abs(selfExe) + + chatterdArgs := []string{"chatterd", "serve", "--host", "0.0.0.0", + "--port", strconv.Itoa(chatterdPort), "--node", node} + if peer != "" { + chatterdArgs = append(chatterdArgs, "--peer", peer, "--peer-node", peerNode) + } + + specs := []childSpec{ + {"chatterd", chatterdArgs}, + {"sysagent", []string{"sysagent", "--node", node, "--interval-ms", "2000"}}, + {"fwatch", []string{"fwatch", "watch", sandboxDir, "--sandbox"}}, + } + + states := make(map[string]*childState, len(specs)) + for _, s := range specs { + cs := &childState{} + cs.state.Store("starting") + states[s.name] = cs + } + + var shuttingDown atomic.Bool + var wg sync.WaitGroup + + for _, spec := range specs { + spec := spec + cs := states[spec.name] + wg.Add(1) + go func() { + defer wg.Done() + for { + cmd := exec.Command(selfExe, spec.args...) + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + cmd.Env = os.Environ() + if err := cmd.Start(); err != nil { + fmt.Fprintf(os.Stderr, "pmon: start service=%s: %v\n", spec.name, err) + time.Sleep(500 * time.Millisecond) + continue + } + cs.pid.Store(int64(cmd.Process.Pid)) + cs.state.Store("up") + fmt.Printf("pmon: started service=%s pid=%d\n", spec.name, cmd.Process.Pid) + + waitErr := cmd.Wait() + if shuttingDown.Load() { + cs.state.Store("down") + return + } + n := cs.restarts.Add(1) + cs.state.Store("restarting") + fmt.Printf("pmon: restart service=%s attempt=%d reason=%v\n", spec.name, n, waitErr) + time.Sleep(300 * time.Millisecond) + } + }() + } + + sigCh := make(chan os.Signal, 2) + signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT) + go func() { + <-sigCh + shuttingDown.Store(true) + for name, cs := range states { + if pid := cs.pid.Load(); pid > 0 { + _ = syscall.Kill(int(pid), syscall.SIGTERM) + } + _ = name + } + }() + + done := make(chan struct{}) + go func() { wg.Wait(); close(done) }() + + order := []string{"chatterd", "sysagent", "fwatch"} + ticker := time.NewTicker(time.Duration(healthIntervalMs) * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-done: + fmt.Println("pmon: shutdown") + return 0 + case <-ticker.C: + printHealth(order, states) + } + } +} + +func printHealth(order []string, states map[string]*childState) { + parts := make([]string, 0, len(order)) + restartParts := make([]string, 0, len(order)) + for _, name := range order { + cs := states[name] + parts = append(parts, name+"="+cs.state.Load().(string)) + restartParts = append(restartParts, fmt.Sprintf("%s:%d", name, cs.restarts.Load())) + } + fmt.Print("pmon: health ") + for i, p := range parts { + if i > 0 { + fmt.Print(" ") + } + fmt.Print(p) + } + fmt.Print(" restarts=") + for i, p := range restartParts { + if i > 0 { + fmt.Print(",") + } + fmt.Print(p) + } + fmt.Println() +} diff --git a/examples/41-capstone-fleet/go/proto.go b/examples/41-capstone-fleet/go/proto.go new file mode 100644 index 0000000..074b5d7 --- /dev/null +++ b/examples/41-capstone-fleet/go/proto.go @@ -0,0 +1,86 @@ +// proto.go — the canonical chatterd chat frame (introduced ch21, unchanged +// since): magic "CH", version 1, one byte type, big-endian u16 length, then +// payload. This capstone speaks only JOIN/MSG/DELIVER, the same three types +// every prior chatterd chapter uses. +package main + +import ( + "encoding/binary" + "errors" + "fmt" + "io" +) + +const ( + frameMagic0 = 'C' + frameMagic1 = 'H' + frameVer = 1 + + typeJoin = 1 + typeMsg = 2 + typeDeliver = 3 +) + +type frame struct { + typ byte + payload []byte +} + +func writeFrame(w io.Writer, f frame) error { + if len(f.payload) > 0xFFFF { + return fmt.Errorf("payload too large: %d bytes", len(f.payload)) + } + hdr := make([]byte, 6) + hdr[0], hdr[1] = frameMagic0, frameMagic1 + hdr[2] = frameVer + hdr[3] = f.typ + binary.BigEndian.PutUint16(hdr[4:], uint16(len(f.payload))) + if _, err := w.Write(hdr); err != nil { + return fmt.Errorf("write header: %w", err) + } + if len(f.payload) > 0 { + if _, err := w.Write(f.payload); err != nil { + return fmt.Errorf("write payload: %w", err) + } + } + return nil +} + +var errBadMagic = errors.New("bad frame magic/version") + +func readFrame(r io.Reader) (frame, error) { + hdr := make([]byte, 6) + if _, err := io.ReadFull(r, hdr); err != nil { + return frame{}, err + } + if hdr[0] != frameMagic0 || hdr[1] != frameMagic1 || hdr[2] != frameVer { + return frame{}, errBadMagic + } + n := binary.BigEndian.Uint16(hdr[4:]) + payload := make([]byte, n) + if n > 0 { + if _, err := io.ReadFull(r, payload); err != nil { + return frame{}, fmt.Errorf("read payload: %w", err) + } + } + return frame{typ: hdr[3], payload: payload}, nil +} + +// deliverPayload builds the ch21 "nick NUL text" DELIVER payload. +func deliverPayload(nick, text string) []byte { + b := make([]byte, 0, len(nick)+1+len(text)) + b = append(b, nick...) + b = append(b, 0) + b = append(b, text...) + return b +} + +// splitDeliver reverses deliverPayload; ok=false if there is no NUL. +func splitDeliver(payload []byte) (nick, text string, ok bool) { + for i, c := range payload { + if c == 0 { + return string(payload[:i]), string(payload[i+1:]), true + } + } + return "", "", false +} diff --git a/examples/41-capstone-fleet/go/sysagent.go b/examples/41-capstone-fleet/go/sysagent.go new file mode 100644 index 0000000..37ab72d --- /dev/null +++ b/examples/41-capstone-fleet/go/sysagent.go @@ -0,0 +1,235 @@ +// sysagent.go — the book's /proc-reading metrics agent (ch36), reduced to +// three signals (cpu%, mem%, load1) exported both as a stdout line (cheap, +// always-on local observability) and as OTel gauges over OTLP when +// OTEL_EXPORTER_OTLP_ENDPOINT is set. Also carries the chaos driver's +// "saturate" helper, which the fleet's chaos drill uses to induce load that +// the emitted cpu_pct/mem_pct should visibly reflect. +package main + +import ( + "bufio" + "context" + "fmt" + "os" + "runtime" + "strconv" + "strings" + "sync" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +type cpuTimes struct{ idle, total uint64 } + +func readCPUTimes() (cpuTimes, error) { + f, err := os.Open("/proc/stat") + if err != nil { + return cpuTimes{}, err + } + defer f.Close() + sc := bufio.NewScanner(f) + if !sc.Scan() { + return cpuTimes{}, fmt.Errorf("empty /proc/stat") + } + fields := strings.Fields(sc.Text()) + if len(fields) < 5 || fields[0] != "cpu" { + return cpuTimes{}, fmt.Errorf("unexpected /proc/stat format: %q", sc.Text()) + } + var total uint64 + var idle uint64 + for i, s := range fields[1:] { + v, err := strconv.ParseUint(s, 10, 64) + if err != nil { + continue + } + total += v + if i == 3 { // idle is the 4th value + idle = v + } + } + return cpuTimes{idle: idle, total: total}, nil +} + +func cpuPct(prev, cur cpuTimes) float64 { + dTotal := float64(cur.total - prev.total) + dIdle := float64(cur.idle - prev.idle) + if dTotal <= 0 { + return 0 + } + return (1 - dIdle/dTotal) * 100 +} + +func memPct() (float64, error) { + f, err := os.Open("/proc/meminfo") + if err != nil { + return 0, err + } + defer f.Close() + var total, avail float64 + sc := bufio.NewScanner(f) + for sc.Scan() { + fields := strings.Fields(sc.Text()) + if len(fields) < 2 { + continue + } + v, err := strconv.ParseFloat(fields[1], 64) + if err != nil { + continue + } + switch strings.TrimSuffix(fields[0], ":") { + case "MemTotal": + total = v + case "MemAvailable": + avail = v + } + } + if total <= 0 { + return 0, fmt.Errorf("MemTotal not found") + } + return (1 - avail/total) * 100, nil +} + +func load1() (float64, error) { + b, err := os.ReadFile("/proc/loadavg") + if err != nil { + return 0, err + } + fields := strings.Fields(string(b)) + if len(fields) < 1 { + return 0, fmt.Errorf("empty /proc/loadavg") + } + return strconv.ParseFloat(fields[0], 64) +} + +type sample struct { + cpuPct, memPct, load1 float64 +} + +func takeSample(prev *cpuTimes) (sample, error) { + cur, err := readCPUTimes() + if err != nil { + return sample{}, err + } + var s sample + if prev.total > 0 { + s.cpuPct = cpuPct(*prev, cur) + } + *prev = cur + if s.memPct, err = memPct(); err != nil { + return sample{}, err + } + if s.load1, err = load1(); err != nil { + return sample{}, err + } + return s, nil +} + +func sysagentRun(node string, intervalMs int, once bool, tel *telemetry) int { + var mu sync.Mutex + var latest sample + + if tel.enabled { + cpuGauge, _ := tel.meter.Float64ObservableGauge("sysagent.cpu.pct", + metric.WithDescription("CPU utilization percent, from /proc/stat")) + memGauge, _ := tel.meter.Float64ObservableGauge("sysagent.mem.pct", + metric.WithDescription("Memory-used percent, from /proc/meminfo")) + loadGauge, _ := tel.meter.Float64ObservableGauge("sysagent.load1", + metric.WithDescription("1-minute load average, from /proc/loadavg")) + _, err := tel.meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error { + mu.Lock() + s := latest + mu.Unlock() + attrs := metric.WithAttributes(attribute.String("node", node)) + o.ObserveFloat64(cpuGauge, s.cpuPct, attrs) + o.ObserveFloat64(memGauge, s.memPct, attrs) + o.ObserveFloat64(loadGauge, s.load1, attrs) + return nil + }, cpuGauge, memGauge, loadGauge) + if err != nil { + fmt.Fprintf(os.Stderr, "sysagent: otel callback registration: %v\n", err) + } + } + + sig := installSignalFlag() + var prev cpuTimes + // Prime the CPU-time baseline so the first printed sample isn't a bogus 0. + prev, _ = readCPUTimes() + time.Sleep(200 * time.Millisecond) + + for { + s, err := takeSample(&prev) + if err != nil { + fmt.Fprintf(os.Stderr, "sysagent: sample: %v\n", err) + return 1 + } + mu.Lock() + latest = s + mu.Unlock() + fmt.Printf("sysagent: node=%s cpu_pct=%.2f mem_pct=%.2f load1=%.2f ts=%d\n", + node, s.cpuPct, s.memPct, s.load1, time.Now().Unix()) + + if once { + break + } + if sig.Load() { + break + } + time.Sleep(time.Duration(intervalMs) * time.Millisecond) + if sig.Load() { + break + } + } + if tel.enabled { + fmt.Fprintf(os.Stderr, "sysagent: otel export_errors=%d\n", exportErrors.Load()) + } + return 0 +} + +// saturate — chaos helper: busy-spin goroutines (cpu) or hold allocated, +// touched memory (mem) for --seconds, so a concurrently-running sysagent's +// cpu_pct/mem_pct visibly rises (this is the fleet's USE-method callback). +func saturate(resource string, seconds, workers, mb int) int { + switch resource { + case "cpu": + if workers <= 0 { + workers = runtime.NumCPU() + } + fmt.Printf("sysagent: saturate resource=cpu seconds=%d workers=%d started\n", seconds, workers) + stop := time.Now().Add(time.Duration(seconds) * time.Second) + var wg sync.WaitGroup + for i := 0; i < workers; i++ { + wg.Add(1) + go func() { + defer wg.Done() + x := 0.0001 + for time.Now().Before(stop) { + x = x*1.0000001 + 1.0 + } + if x == 0 { // never true; keeps the compiler from eliding the loop + fmt.Println(x) + } + }() + } + wg.Wait() + fmt.Println("sysagent: saturate done") + return 0 + case "mem": + if mb <= 0 { + mb = 256 + } + fmt.Printf("sysagent: saturate resource=mem seconds=%d mb=%d started\n", seconds, mb) + buf := make([]byte, mb*1024*1024) + for i := range buf { + buf[i] = byte(i) // touch every page so it's resident, not just reserved + } + time.Sleep(time.Duration(seconds) * time.Second) + _ = buf[0] + fmt.Println("sysagent: saturate done") + return 0 + default: + fmt.Fprintf(os.Stderr, "usage: sysagent saturate --resource cpu|mem --seconds N [--workers K|--mb M]\n") + return 2 + } +} diff --git a/examples/41-capstone-fleet/go/telemetry.go b/examples/41-capstone-fleet/go/telemetry.go new file mode 100644 index 0000000..66b74c0 --- /dev/null +++ b/examples/41-capstone-fleet/go/telemetry.go @@ -0,0 +1,120 @@ +// telemetry.go — OTel wiring shared by chatterd and sysagent. Both signals +// (traces for chatterd deliveries, metrics for sysagent's /proc samples) are +// exported over OTLP/HTTP. Endpoint comes from the standard +// OTEL_EXPORTER_OTLP_ENDPOINT env var (the exporters read it themselves, per +// the OTel env-var spec) — this is exactly the variable +// scripts/lab/deploy-to-vm.sh forwards from a caller's OTEL_ENDPOINT, so a +// service started under pmon on a lab guest picks it up with no extra flag. +package main + +import ( + "context" + "fmt" + "os" + "sync/atomic" + "time" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + semconv "go.opentelemetry.io/otel/semconv/v1.26.0" + "go.opentelemetry.io/otel/trace" +) + +// exportErrors counts OTel export failures observed via the global error +// handler. verify polls this (via the "otel: export_errors=N" line each +// service prints at exit / on request) as the deterministic stand-in for +// "the exporter got a 200" — the batch/periodic exporters only ever call the +// error handler on a non-2xx response or a transport failure. +var exportErrors atomic.Int64 + +type telemetry struct { + enabled bool + tracer trace.Tracer + meter metric.Meter + shutdown func(context.Context) error +} + +func noopTelemetry(service string) *telemetry { + return &telemetry{ + enabled: false, + tracer: otel.Tracer(service), + meter: otel.Meter(service), + shutdown: func(context.Context) error { return nil }, + } +} + +// initTelemetry sets up trace+metric providers if OTEL_EXPORTER_OTLP_ENDPOINT +// is set; otherwise it returns a disabled, no-op telemetry (the service still +// runs — OTel is additive, never a hard dependency of the fleet coming up). +func initTelemetry(ctx context.Context, service, node string) *telemetry { + endpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT") + if endpoint == "" { + fmt.Fprintf(os.Stderr, "%s: otel disabled (OTEL_EXPORTER_OTLP_ENDPOINT not set)\n", service) + return noopTelemetry(service) + } + + otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) { + exportErrors.Add(1) + fmt.Fprintf(os.Stderr, "%s: otel export error: %v\n", service, err) + })) + + res, err := resource.New(ctx, + resource.WithAttributes( + semconv.ServiceName(service), + semconv.ServiceVersion("41-capstone-fleet"), + semconv.DeploymentEnvironment("lab"), + attribute.String("node.name", node), + ), + ) + if err != nil { + // resource.New only fails on attribute-schema conflicts, which cannot + // happen with the static set above; treat as unreachable but degrade + // to no-op rather than crash a supervised service over telemetry. + fmt.Fprintf(os.Stderr, "%s: otel resource: %v (telemetry disabled)\n", service, err) + return noopTelemetry(service) + } + + traceExp, err := otlptracehttp.New(ctx) + if err != nil { + fmt.Fprintf(os.Stderr, "%s: otel trace exporter: %v (telemetry disabled)\n", service, err) + return noopTelemetry(service) + } + tp := sdktrace.NewTracerProvider( + sdktrace.WithBatcher(traceExp, sdktrace.WithBatchTimeout(500*time.Millisecond)), + sdktrace.WithResource(res), + ) + otel.SetTracerProvider(tp) + + metricExp, err := otlpmetrichttp.New(ctx) + if err != nil { + fmt.Fprintf(os.Stderr, "%s: otel metric exporter: %v (telemetry disabled)\n", service, err) + return noopTelemetry(service) + } + mp := sdkmetric.NewMeterProvider( + sdkmetric.WithResource(res), + sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp, sdkmetric.WithInterval(2*time.Second))), + ) + otel.SetMeterProvider(mp) + + fmt.Fprintf(os.Stderr, "%s: otel enabled endpoint=%s node=%s\n", service, endpoint, node) + + return &telemetry{ + enabled: true, + tracer: otel.Tracer(service), + meter: otel.Meter(service), + shutdown: func(ctx context.Context) error { + err1 := tp.Shutdown(ctx) + err2 := mp.Shutdown(ctx) + if err1 != nil { + return err1 + } + return err2 + }, + } +} diff --git a/examples/41-capstone-fleet/go/util.go b/examples/41-capstone-fleet/go/util.go new file mode 100644 index 0000000..39240cc --- /dev/null +++ b/examples/41-capstone-fleet/go/util.go @@ -0,0 +1,23 @@ +// util.go — small helpers shared across the fleet's subcommands. +package main + +import ( + "os" + "os/signal" + "sync/atomic" + "syscall" +) + +// installSignalFlag returns an atomic bool that flips to true the first time +// SIGTERM or SIGINT arrives, so a poll loop can unwind cleanly instead of +// dying mid-write. +func installSignalFlag() *atomic.Bool { + flag := &atomic.Bool{} + ch := make(chan os.Signal, 2) + signal.Notify(ch, syscall.SIGTERM, syscall.SIGINT) + go func() { + <-ch + flag.Store(true) + }() + return flag +} diff --git a/examples/41-capstone-fleet/rust/Cargo.lock b/examples/41-capstone-fleet/rust/Cargo.lock new file mode 100644 index 0000000..9fc0988 --- /dev/null +++ b/examples/41-capstone-fleet/rust/Cargo.lock @@ -0,0 +1,66 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "template-hello" +version = "0.1.0" +dependencies = [ + "rustix", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] diff --git a/examples/41-capstone-fleet/rust/Cargo.toml b/examples/41-capstone-fleet/rust/Cargo.toml new file mode 100644 index 0000000..62c962f --- /dev/null +++ b/examples/41-capstone-fleet/rust/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "template-hello" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +rustix = { version = "1", features = ["system"] } diff --git a/examples/41-capstone-fleet/rust/demo.sh b/examples/41-capstone-fleet/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/41-capstone-fleet/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/41-capstone-fleet/rust/rust-toolchain.toml b/examples/41-capstone-fleet/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/41-capstone-fleet/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/41-capstone-fleet/rust/src/main.rs b/examples/41-capstone-fleet/rust/src/main.rs new file mode 100644 index 0000000..9536d12 --- /dev/null +++ b/examples/41-capstone-fleet/rust/src/main.rs @@ -0,0 +1,12 @@ +use rustix::system::uname; + +fn main() { + let info = uname(); + println!( + "pid {} on {} {} ({})", + std::process::id(), + info.sysname().to_string_lossy(), + info.release().to_string_lossy(), + info.machine().to_string_lossy(), + ); +} diff --git a/examples/41-capstone-fleet/verify.lua b/examples/41-capstone-fleet/verify.lua new file mode 100644 index 0000000..230569e --- /dev/null +++ b/examples/41-capstone-fleet/verify.lua @@ -0,0 +1,17 @@ +-- Verify the hello-syscall template: run the demo for LSP_LANG and check its output. +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and EXAMPLE_DIR set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local r = checks.run("./demo.sh " .. lang .. " run") +checks.expect_exit(r, 0, lang .. ": demo exits 0") +checks.expect_match(r.out, "pid %d+ on Linux", lang .. ": prints pid/uname line") +checks.finish() diff --git a/examples/manifest.yaml b/examples/manifest.yaml index e816357..0661b3d 100644 --- a/examples/manifest.yaml +++ b/examples/manifest.yaml @@ -146,3 +146,29 @@ examples: langs: [cpp, go, rust] mode: local timeout: 360 + - id: 36-proc-sys-and-the-agent + langs: [cpp, go, rust] + mode: local + timeout: 300 + - id: 37-use-method-60s-analysis + langs: [cpp, go, rust] + mode: vm + timeout: 360 + - id: 38-opentelemetry-to-lgtm + langs: [cpp, go, rust] + mode: local + requires: [lgtm] + timeout: 420 + - id: 39-benchmarking-without-lies + langs: [cpp, go, rust] + mode: local + timeout: 360 + - id: 40-low-latency-fastpath + langs: [cpp, go, rust] + mode: vm + timeout: 420 + - id: 41-capstone-fleet + langs: [cpp, go, rust] + mode: vm-peer + requires: [lgtm] + timeout: 600