diff --git a/_docs/28-gdb-and-remote-debugging.md b/_docs/28-gdb-and-remote-debugging.md new file mode 100644 index 0000000..28cc1b9 --- /dev/null +++ b/_docs/28-gdb-and-remote-debugging.md @@ -0,0 +1,560 @@ +--- +title: "gdb and Remote Debugging" +order: 28 +part: "Debugging" +description: "app crashes on purpose, three ways: a real SIGSEGV and a systemd-coredump core in C++, a runtime-caught panic in Go, an Option::unwrap() panic in Rust — and gdb, rust-gdb, coredumpctl, readelf, and objdump read the same crash from four directions, then gdbserver and target remote carry the session from the host to a VM." +duration: "55 minutes" +--- + +Every debugger so far in this book has been a `println` you had to recompile +to move. This chapter replaces that habit with the real tool: a crash that +already happened, frozen in a core file or a live process, that you can +walk backwards through without changing a line of code. `app` is this +chapter's target — a small pmon-like supervisor that behaves identically in +`run` mode across C++, Go, and Rust, then diverges completely in `crash` +mode, because the *entire point* of this chapter is that "the same bug" does +not mean "the same crash": C++ gets a hardware `SIGSEGV` and a core file, +Go's runtime intercepts the fault and turns it into a panic, and Rust's +`Option::unwrap()` never touches invalid memory at all. One call chain, three +debugging stories. + +The code is in `examples/28-gdb-and-remote-debugging/`. `./demo.sh build` +builds all three; each language directory's `demo.sh run [crash]` runs it, +and the top-level `README.md` documents the full reproduction and +verification contract. + +{% include excalidraw.html + file="28-stack-frame-anatomy" + alt="A column of five stacked frames from a real gdb bt full on app's core file, growing toward lower addresses as they go down: frame 0 std::println<int,int> at print colon 164, the actual crash PC; frame 1 deref at main.cpp:52 showing child equals pid 4242 exit_slot 0x0; frame 2 handle_child at main.cpp:56 with return address 0x4016b2; frame 3 run_supervisor at main.cpp:65 with return address 0x40177c and locals child, guard equals optimized out; frame 4 main at main.cpp:104 with return address 0x400964. Beside it, a DWARF cross-check column shows readelf/objdump decodedline results: main.cpp:52 maps to 0x401680, an exact match to gdb's own breakpoint address; main.cpp:56 maps to 0x4016b2, an exact match to frame 2's return address; main.cpp:65 maps to 0x401772, close to but not exactly frame 3's return address of 0x40177c since that is a post-call return address, not a statement start. A footer note says every frame function is noinline so -O2 cannot fold the call chain away, and ShutdownGuard's destructor never runs because SIGSEGV does not unwind the stack." + caption="Figure 28.1 — the five real frames gdb names on app's core, cross-checked address for address against the DWARF line table readelf and objdump report independently" %} + +> **Tools used** — `gdb` 17.2, `rust-gdb`, `coredumpctl`, `readelf`, `objdump` +> (host); `gdb-gdbserver` (preinstalled on the `systems-target` VM image via +> cloud-init, documented but not exercised live in this local lane — no +> `systems-target` VM was up this session); `dlv` (delve) is referenced for +> contrast but is a soft/optional host tool per `scripts/check-host.sh` and +> was not installed or run here. Everything else appears in +> `scripts/check-host.sh`. + +## Reading a crash: core, gdb, and the languages that don't need one + +A **core dump** is the kernel's own crash-scene photograph: on an unhandled +fatal signal (`SIGSEGV`, `SIGABRT`, `SIGILL`, ...), the default disposition +is to write the process's entire address space, register state, and open-fd +table to disk before terminating it. On a plain workstation that used to +mean a `core` file dropped in the crashing process's working directory; on +this host, and on any modern Fedora/systemd machine, +`/proc/sys/kernel/core_pattern` instead reads +`|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %d %F` — the +leading `|` means the kernel *pipes* the core to that program rather than +writing a file itself, so "the crash dropped a core" means +`coredumpctl` has a compressed copy indexed by pid, not that a bare `core` +file appeared anywhere. `coredumpctl info ` confirms one exists and +prints its own compact backtrace; `coredumpctl dump -o core.app` +extracts it as an ordinary file gdb can open directly. + +gdb's job starts where that photograph does. Opened against a binary that +still carries debug info — this chapter's C++ build uses the `release` +CMake preset, which is `RelWithDebInfo` (`-O2 -g`, optimized *and* +debuggable) — gdb reconstructs source-level frames, local variables, and +even C++ types from the DWARF the compiler embedded, not from guesswork. +That last part is what "pretty-printers" buy you: without them, a +`std::string_view` is a two-word struct (`{ _M_str, _M_len }`) and you'd read +its text by hand out of memory; with libstdc++'s printers loaded (they ship +with the distro's gdb and register themselves automatically), the same +value prints as the string itself. Breaking at `parse_mode` and feeding it a +bad argument shows exactly this: + +```console +[host]$ gdb --batch -ex 'set pagination off' -ex 'set debuginfod enabled off' \ + -ex 'break parse_mode' -ex 'run bogus-thing' -ex 'print sub' -ex 'whatis sub' \ + ./app +Breakpoint 1, (anonymous namespace)::parse_mode (sub="bogus-thing") at src/main.cpp:77 +$1 = "bogus-thing" +type = std::string_view +``` + +`sub="bogus-thing"` in the frame header and `$1 = "bogus-thing"` are both the +pretty-printer at work — `info pretty-printer` confirms the object file +registering it: `objfile /lib64/libstdc++.so.6 pretty-printers: +libstdc++-v6`. The same mechanism turns a struct reference into a readable +value: breaking at `deref` and printing `child` (a `const ChildRecord&`) +gives `$1 = (const (anonymous namespace)::ChildRecord &) @0x7fff...: {pid = +4242, exit_slot = 0x0}` — field names, not offsets. And because gdb reads +target memory rather than executing your code's checks, you can trigger the +exact fault *before* the program does: `print *child.exit_slot` at that +breakpoint returns `Cannot access memory at address 0x0` — gdb hit the same +null pointer the CPU is about to, just without a signal. + +`rust-gdb` is the same gdb binary wrapped with a small Python autoload script +that ships with `rustc` and teaches it Rust's type layout — enum +discriminants, `Option`, slices. Breaking at `app::deref` on the Rust build +and printing the child record shows the payoff directly: + +```console +[host]$ rust-gdb --batch -ex 'break app::deref' -ex 'run crash' -ex 'print *child' ./app +Breakpoint 1, app::deref (child=0x7fff...) at src/main.rs:38 +$1 = app::ChildRecord {pid: 4242, exit_slot: core::option::Option::None} +``` + +`exit_slot: core::option::Option::None` is the variant *name*, not the +tag byte a plain gdb without the Rust autoload would show. Plain `gdb` (no +`rust-gdb` wrapper) can still open the same binary — Rust's DWARF is real +DWARF — but you lose this layout-aware rendering and read raw enum +representations by hand instead. + +Go breaks this whole story on purpose. There is no core file to open in the +first place: the Go runtime installs its own handler for the fault-causing +signals and, on recognizing that the fault came from ordinary user code (not +a bug inside the runtime itself), converts it into a normal — if +unrecovered — `panic`, never re-raising `SIGSEGV` with the default +disposition that would dump core. `gdb` *can* still attach to a running Go +binary (Fedora's gdb ships "Loading Go Runtime support" goroutine helpers), +and on this exact crash it stops cleanly at the fault and even walks a +correct four-frame backtrace, because the whole program is one goroutine +blocking on one OS thread when it dies: + +```console +[host]$ gdb --batch -ex 'run crash' -ex bt --args ./app crash +Thread 1 "app" received signal SIGSEGV, Segmentation fault. +0x000000000049fd45 in main.deref (child=...) at main.go:42 +#0 main.deref (child=...) at main.go:42 +#1 main.handleChild (child=...) at main.go:47 +#2 main.runSupervisor (injectBug=true, ~r0=) at main.go:56 +#3 main.main () at main.go:79 +``` + +That result is flattering and it hides the real problem. gdb's thread list +maps to **OS threads** (the `M`s in Go's `G`/`M`/`P` scheduler), not to +**goroutines** (`G`s) — a goroutine that is parked on a channel, blocked in a +`select`, or simply not currently scheduled onto any `M` has no OS thread for +gdb to show you, and gdb has no notion of "goroutine 41" as a selectable +frame context the way it has "thread 3". The moment your program has more +than a handful of concurrent goroutines, `info threads` and `bt` stop mapping +onto your program's logical concurrency at all. Go's own tooling doesn't have +this blind spot because it was built with the scheduler in mind: `dlv exec +./app`, then `goroutines` to list every logical goroutine (parked or +running) and `goroutine bt` to backtrace one specifically — `delve` +walks the runtime's own goroutine table, not the OS thread list. This +host does not have `dlv` installed (it is a soft, optional entry in +`scripts/check-host.sh`, not a hard requirement), so that specific +`goroutines`/`goroutine bt` session is unverified here — the point stands on the gdb +transcript above being accidentally complete only because this crash never +gets past goroutine 1. + +Rust needs neither gdb nor a core for this particular bug class either. +`.unwrap()` on a `None` calls `panic!` directly — no signal is raised at +all, so there is nothing for gdb to catch even if you attached. The +idiomatic tool is `RUST_BACKTRACE=1`, which walks the unwinder's own frame +records and prints every one by name, entirely in userspace: + +```console +[host]$ RUST_BACKTRACE=1 ./app crash +thread 'main' panicked at src/main.rs:38:25: +called `Option::unwrap()` on a `None` value + 5: app::deref + at ./src/main.rs:38:25 + 6: app::handle_child + at ./src/main.rs:44:5 + 7: app::run_supervisor + at ./src/main.rs:59:9 + 8: app::main + at ./src/main.rs:102:16 +exit=101 +``` + +Without `RUST_BACKTRACE=1` the same panic and the same `exit=101` still +happen, but the frame names are gone — the backtrace is opt-in, the crash +is not. + +## Remote debugging: gdbserver, target remote, and CLion + +Everything above assumes gdb and the crashing binary share a filesystem. +That stops being true the moment the thing you need to debug lives on a +different machine — this book's `systems-target` lab VM, a container, a +device you cannot install a full toolchain on. `gdbserver` solves this by +splitting gdb in two: a thin stub (`gdbserver`) runs *next to* the target +process and does nothing but single-step it, read its memory, and set +breakpoints on request; your full gdb runs on the host, keeps all its +symbol tables and pretty-printers locally, and drives the stub over gdb's +**remote serial protocol**, typically carried over TCP. Only the protocol's +small binary command set crosses the wire — source, DWARF, and the +pretty-printer scripts never leave the host. + +{% include excalidraw.html + file="28-remote-debug-topology" + alt="Two stacked bands. The upper host band, marked verified this session, shows the local core-dump path: app crash producing a SIGSEGV, an arrow into systemd-coredump, coredumpctl dump extracting core.app, and gdb app core.app doing a full bt. Beside it, a second host-side node reads gdb (client), target remote systems-target:2345, with an amber bidirectional arrow labeled gdb remote serial protocol, TCP:2345 descending into the lower systems-target VM band, marked documented, not run this session. That band holds a gdbserver :2345 --once app crash node feeding an app (crash target) child process node below it, plus a dashed ghost box off to the side reading CLion, Remote GDB Server run config, unverified. A footer note says gdbserver is preinstalled on the lab VM image via cloud-init but not on this host, and no systems-target VM was up this session." + caption="Figure 28.2 — the local core-dump path this chapter's evidence is built on, beside the gdbserver/target-remote/CLion path documented for the systems-target VM" %} + +The session looks like this in practice: + +```console +[vm]$ gdbserver --once :2345 ./app crash +Process ./app created; pid = 4183 +Listening on port 2345 +``` + +```console +[host]$ gdb ./cpp/build/release/app +(gdb) target remote systems-target:2345 +Remote debugging using systems-target:2345 +(gdb) continue +``` + +`--once` tells `gdbserver` to serve exactly one debug session and exit +afterward — the right default for a scripted reproduction, since a stray +`gdbserver` left listening on a lab VM is a debug backdoor with no +authentication. `target remote` is the one gdb command that changes +everything downstream of it: every `break`, `run` (really `continue`, since +the process is already started), `print`, and `bt` you already know behaves +identically, dispatched over the wire instead of a local `ptrace(2)`. The +binary path you pass to gdb on the host must be the *unstripped*, debug-info +copy — gdbserver on the VM only needs the raw executable, gdb on the host +supplies all the symbols. + +This exact transcript is unverified in this chapter: no `systems-target` +VM was up during this session (`virsh list --all` shows no defined guests +in this local lane), so the local core-dump path above carries this +chapter's verified evidence, and the remote path is documented from the lab +image's cloud-init (`gdb-gdbserver` is provisioned there) and the command +shapes gdb itself specifies, not from a live run. Confirm it for real by +bringing the lab up (`scripts/lab/lab-up.sh`) and repeating the two +transcripts above against the deployed binary. + +CLion's remote debug story is the same protocol wearing an IDE: a **"Remote +GDB Server" run configuration** points at a `target-name` (an SSH-reachable +host, here `systems-target`), a path to the *local* copy of the binary with +debug info, and a `gdbserver` command line to run on that target — CLion +then opens the SSH session, launches `gdbserver` remotely, and drives it +with its own bundled gdb, giving you the IDE's breakpoint gutter and +variable inspector over the identical remote-serial link. This +configuration is unverified +in this chapter — it is real, documented CLion functionality (Settings → +Build, Execution, Deployment → Toolchains → a remote toolchain over SSH, +then a Remote GDB Server run configuration referencing it), not something +run here. + +## How the code works + +`app` walks the identical four-function call chain in every language — +`main -> run_supervisor -> handle_child -> deref` — and it is written to +resist the optimizer erasing that chain: every frame function is marked +noinline (`[[gnu::noinline]]` / `//go:noinline` / `#[inline(never)]`), and +all three builds keep debug info in an otherwise optimized binary (C++'s +`release` preset is `RelWithDebInfo`; Go's toolchain always emits DWARF; +Rust's `Cargo.toml` sets `[profile.release] debug = true`). Rust needed one +more trick: `handle_child`'s call to `deref` is its last statement, so +without a following `std::hint::black_box(())`, LLVM turned it into a tail +call and reused `handle_child`'s own frame for `deref` — erasing +`handle_child` from every backtrace. Verified directly: removing the +`black_box` line made `RUST_BACKTRACE=1`'s trace jump straight from +`app::deref` to `app::run_supervisor`. + +The data shape is one `ChildRecord` — a pid plus a slot for the child's exit +status that a supervisor's reaper is supposed to fill in before anything +reads it. In `run` mode that slot is populated first; in `crash` mode +`handle_child` runs *as if it fired before the reaper got there* — the +exact race a real signal-driven supervisor (this book's `pmon`, from Part 3 +onward) can hit if a `SIGCHLD` notification and the code consuming it get +out of step. Each language expresses "not populated yet" the way its type +system prefers, and that choice is the whole reason the crash differs: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +struct ChildRecord { + int pid = 0; + const int* exit_slot = nullptr; +}; + +[[gnu::noinline]] void deref(const ChildRecord& child) { + // BUG: no null check before the read. In "crash" mode exit_slot is + // nullptr because handle_child ran before the reaper recorded the exit + // status -- exactly the kind of race a real supervisor can hit. + std::println("pmon: child {} exited status={}", child.pid, + *child.exit_slot); +} + +[[gnu::noinline]] void handle_child(const ChildRecord& child) { deref(child); } +``` + +```go +type childRecord struct { + pid int + exitSlot *int +} + +//go:noinline +func deref(child childRecord) { + // BUG: no nil check before the read. In "crash" mode exitSlot is nil + // because handleChild ran before the reaper recorded the exit status -- + // exactly the kind of race a real supervisor can hit. + fmt.Printf("pmon: child %d exited status=%d\n", child.pid, *child.exitSlot) +} + +//go:noinline +func handleChild(child childRecord) { + deref(child) +} +``` + +```rust +#[derive(Clone, Copy)] +struct ChildRecord { + pid: u32, + exit_slot: Option, +} + +#[inline(never)] +fn deref(child: &ChildRecord) { + // BUG: unwrap() with no check. In "crash" mode exit_slot is None because + // handle_child ran before the reaper recorded the exit status -- exactly + // the kind of race a real supervisor can hit. + println!( + "pmon: child {} exited status={}", + child.pid, + child.exit_slot.unwrap() + ); +} + +#[inline(never)] +fn handle_child(child: &ChildRecord) { + deref(child); + // A no-op after the call: without it, `deref` is the last thing this + // function does and the optimizer turns the call into a tail jump, + // reusing this frame and erasing `handle_child` from the backtrace. + std::hint::black_box(()); +} +``` + +C++'s `const int* exit_slot = nullptr` is the raw pointer the whole bug class +is named after — `*child.exit_slot` reads through address `0x0`, the MMU +raises a page fault on an unmapped page, and the kernel delivers `SIGSEGV` +with no C++-level participation at all. Go's `*int` is the same shape at the +machine level, but the runtime stands between the fault and the process: +its signal handler recognizes a `nil`-pointer-shaped fault from user code +and converts it into `runtime.Error`, which surfaces as an ordinary panic. +Rust rejected the raw-pointer option outright — safe Rust has no way to +leave a pointer dangling like this — so the equivalent "not filled in yet" +state is `Option::None`, and `.unwrap()` is an explicit, checked +operation that panics on `None` rather than reading through anything. +`run_supervisor` is the wiring in all three: it builds one `ChildRecord`, +chooses `nullptr`/`nil`/`None` versus a real value based on `inject_bug`, and +calls `handle_child` either way — the crash path and the clean path are +*the same code*, differing only in the data, which is the point: this bug +is not a different code path, it's a value that should have arrived and +didn't. + +## Errors, three ways + +The subcommand parsing is this book's usual three-way split and is +identical output across languages: no argument or an unrecognized one +prints the two-line `usage: app run` / `app crash` block to stderr and +exits 2. `app run` prints three fixed lines and exits 0 in every language — +byte-identical, dual-use, unremarkable. `app crash` is where the three +languages part ways, and the table is the chapter in miniature: + +| | mechanism | exit | core? | names the frame | +|---|---|---|---|---| +| C++ | null pointer deref → `SIGSEGV` | 139 | yes, via `systemd-coredump` | `gdb --batch -ex run -ex bt` | +| Go | nil deref → runtime catches `SIGSEGV`, converts to panic | 2 | no | the goroutine trace in the panic output | +| Rust | `Option::unwrap()` on `None` → panic | 101 | no | `RUST_BACKTRACE=1` | + +One more thing worth reading off the C++ trace directly: the `ShutdownGuard` +RAII type in `run_supervisor`'s clean path prints "supervisor exiting +cleanly" from its destructor, and that line never appears on the crash path +— confirmed in every transcript above. RAII buys you cleanup on every +*C++-level* exit (`return`, a thrown exception, `std::exit` via a registered +handler); it buys you nothing against a signal, because a signal does not +unwind the stack — the kernel simply stops the process where it stands. + +## What each language's tooling gives you + +This chapter is tooling-heavy rather than concurrency-heavy, so the lens +here is what each debugger sees, not a data race: gdb sees raw memory, +registers, and DWARF — it is precise but blind to any concept a language +runtime invented for itself. That is exactly why it is the *only* tool for +the C++ crash (there is no higher-level runtime to ask) and only an +*accidentally* adequate one for Go (it happened to catch this crash while +the program was still single-goroutine) — `delve` exists specifically to +close that gap by understanding the `G`/`M`/`P` scheduler gdb has no model +of. Rust splits the difference: `rust-gdb` gives gdb's raw power *plus* +enum/`Option`-aware rendering, but for an ordinary panic-and-unwind exit +(no signal, no fault), the unwinder's own `RUST_BACKTRACE=1` output is +faster to reach for and needs no debugger session at all. + +## Build, run, observe + +```console +[host]$ cd examples/28-gdb-and-remote-debugging && ./demo.sh build +``` + +```console +[host]$ ./cpp/build/release/app run +pmon: supervisor starting +pmon: child 4242 exited status=0 +pmon: supervisor exiting cleanly +``` + +Now reproduce the crash and pull the core by hand, exactly as `verify.lua` +does it: + +```console +[host]$ ws=$(mktemp -d); cd "$ws" +[host]$ ( ulimit -c unlimited; /path/to/cpp/build/release/app crash & echo $! >pid; wait $! ) +pmon: supervisor starting +[host]$ coredumpctl info "$(cat pid)" +Signal: 11 (SEGV) +Storage: /var/lib/systemd/coredump/core.app.1000...zst (present) +Stack trace of thread 2906298: + #0 ... _ZSt7printlnIJRKiS1_EEvSt19basic_format_stringIcJDpNSt13type_identityIT_E4typeEEEDpOS4_ (app + 0x169a) + #1 ... run_supervisor (app + 0x177c) + #2 ... main (app + 0x964) +``` + +Notice `coredumpctl`'s own quick trace, which is itself worth reading +carefully: it names only three frames — `println`'s formatting internals, +`run_supervisor`, `main` — silently skipping `deref` and `handle_child`. +That trace is a lightweight summary, not a full DWARF-driven unwind; it is +*not* the tool to trust for a real backtrace, and gdb's full walk against +the extracted core proves it: + +```console +[host]$ coredumpctl dump "$(cat pid)" -o core.app +[host]$ gdb --batch -ex 'set debuginfod enabled off' -ex 'bt full' \ + /path/to/cpp/build/release/app core.app +#0 0x000000000040169a in std::println (__fmt=...) at /usr/include/c++/16/print:164 +#1 (anonymous namespace)::deref (child=...) at src/main.cpp:52 +#2 0x00000000004016b2 in (anonymous namespace)::handle_child (child=...) at src/main.cpp:56 +#3 0x000000000040177c in (anonymous namespace)::run_supervisor (inject_bug=true) at src/main.cpp:65 + child = {pid = 4242, exit_slot = 0x0} +#4 0x0000000000400964 in main (argc=, argv=) at src/main.cpp:104 +``` + +All five frames, `deref` and `handle_child` included, with the exact +`exit_slot = 0x0` that caused the fault sitting right there in frame #3's +locals. The runner agrees end to end: `python3 scripts/test-all-examples.py +--only 28-gdb-and-remote-debugging` reports `PASS PASS PASS`, and each +language's `verify.lua` run separately confirms its own crash contract — +`cpp`'s scripted `gdb --batch -ex run -ex bt` names `deref`, +`handle_child`, and `run_supervisor`; `go`'s exit code is 2 with the +standard nil-deref panic text; `rust`'s exit code is 101 with +`RUST_BACKTRACE=1` naming all four Rust frames and no frame names at all +without it. + +## Cross-check: gdb's addresses against the DWARF line table directly + +Every address gdb reported above is derived from the same `.debug_line` +section `readelf` and `objdump` can decode independently, with no gdb +involved at all — if they agree, gdb isn't inventing anything. `readelf -S` +first confirms the binary actually carries full DWARF (it is optimized, not +stripped): + +```console +[host]$ readelf -S ./app | grep -i debug + [31] .debug_aranges ... + [32] .debug_info ... + [34] .debug_line ... + [35] .debug_str ... +``` + +`objdump --dwarf=decodedline` decodes that section into source-line -> +address rows directly: + +```console +[host]$ objdump --dwarf=decodedline ./app | grep -E "main.cpp\s+(52|56|65)\b" +main.cpp 52 0x401680 +main.cpp 56 0x4016b0 +main.cpp 56 0x4016b2 +main.cpp 65 0x401772 +``` + +Two of these match gdb's own numbers exactly, independently derived: gdb's +breakpoint on line 52 (`Breakpoint 1 at 0x401680: file .../main.cpp, line +52`) is the identical address `objdump` reports for that line, and the core +backtrace's frame #2 return address (`0x00000000004016b2`) is the identical +address `objdump` reports for line 56. Line 65's row (`0x401772`) is close +to but not identical to frame #3's return address (`0x40177c`) — expected, +since a frame's *return* address sits a few bytes past a `call` +instruction's own encoding, not at the statement's first byte the way a +breakpoint address does. `readelf -n` supplies one more independent +identity check, the build ID, which ties this exact binary to this exact +debug-info walk: + +```console +[host]$ readelf -n ./app | grep "Build ID" + Build ID: 6c7d2d0445c2e75e68424a4c7c2564d2902c6f12 +``` + +`readelf -S` on the Go and Rust binaries confirms the same discipline holds +across all three languages: Go's toolchain always emits a `.debug_info` / +`.debug_line` pair (Go tooling doesn't offer a way to omit DWARF from a +default build), and Rust's release binary carries the same sections because +of this crate's `[profile.release] debug = true` — three different +compilers, one DWARF-based line-number contract that `readelf`, `objdump`, +and gdb all read the same way. + +## What you learned + +- A core dump on a modern systemd host is not a bare `core` file — it's a + `systemd-coredump` entry `coredumpctl` finds by pid and extracts on + request; opening that extracted core in gdb (`gdb app core.app`) gives a + full DWARF-driven backtrace that a lightweight summary trace (like + `coredumpctl info`'s own) can silently omit frames from. +- gdb's pretty-printers turn `std::string_view`/`ChildRecord`/etc. into + readable values instead of raw offsets, and `rust-gdb` extends the same + idea to Rust's `Option`/enum layout (`exit_slot: + core::option::Option::None`, not a bare discriminant byte). +- gdb's thread model is OS threads, not goroutines — it can look complete on + a single-goroutine crash and still be blind to the concurrency that + matters; `delve`'s `goroutines`/`goroutine bt` exists specifically to + close that gap. +- `gdbserver` + `target remote` split gdb across a network boundary without + losing any capability — only the remote serial protocol crosses the wire, + every symbol and pretty-printer stays on the host — and `readelf`/`objdump` + on `.debug_line` are the independent cross-check that confirms gdb's own + addresses rather than trusting them blind. + +Next, **valgrind, sanitizers, and Miri**: three very different ways of +catching the memory bugs a debugger only shows you *after* they've already +happened. + +--- + +

verified — every transcript +above except the two spans marked unverified (the `systems-target` gdbserver/target +remote session, the CLion remote-debug configuration, and delve's +`goroutines`/`goroutine bt`) was produced on the Fedora 44 reference host +(kernel 7.1.3-200.fc44, gdb 17.2, gcc 16.1.1, go1.26.5, rustc 1.97.1, readelf/ +objdump 2.46.1) this session: app crash under +ulimit -c unlimited exited 139 and coredumpctl +recorded and extracted a real core; a scripted +gdb --batch -ex run -ex bt and a separate gdb ... bt +full against that extracted core both named +deref/handle_child/run_supervisor/main, +with frame addresses 0x4016b2 (handle_child) and +0x400964 (main); rust-gdb broke at +app::deref and printed +ChildRecord {pid: 4242, exit_slot: core::option::Option<i32>::None}; +Go's ./app crash exited 2 with the standard nil-pointer-deref +panic text and a goroutine trace naming main.deref, +main.handleChild, main.runSupervisor; a direct +gdb ... bt on the running Go binary independently named the +same four frames; Rust's RUST_BACKTRACE=1 ./app crash exited +101 and named app::deref through app::main, while +a bare ./app crash exited 101 with no frame names; +readelf -S confirmed .debug_info/.debug_line +present and unstripped in all three binaries; objdump +--dwarf=decodedline reported main.cpp:52 -> 0x401680 +(matching gdb's own breakpoint address) and main.cpp:56 -> +0x4016b2 (matching the core backtrace's frame #2 return address) +exactly; and the runner printed +28-gdb-and-remote-debugging PASS PASS PASS. The gdbserver/ +target-remote path is documented from the lab VM's cloud-init +(gdb-gdbserver is provisioned there) and gdb's own command +semantics, not from a live session — confirm it by bringing +systems-target up and repeating the two-line transcript in +"Remote debugging" against the deployed binary.

diff --git a/_docs/29-valgrind-sanitizers-miri.md b/_docs/29-valgrind-sanitizers-miri.md new file mode 100644 index 0000000..b1d70f2 --- /dev/null +++ b/_docs/29-valgrind-sanitizers-miri.md @@ -0,0 +1,444 @@ +--- +title: "valgrind, Sanitizers, and Miri" +order: 29 +part: "Debugging" +description: "bugfarm seeds five defects — leak, use-after-free, an uninitialised read, a heap overflow, and a data race — behind one `app ` CLI in C++, Go, and Rust, and this chapter builds a real catch/miss matrix from actual runs: valgrind's memcheck/helgrind/massif against a plain binary (10-50x slower, and outright confused by Go's own runtime), the compile-time-instrumented ASan/UBSan/TSan/LSan against a rebuilt one, and Rust's miri catching the identical use-after-free class by symbolic interpretation of a small `unsafe` snippet instead of either." +duration: "60 minutes" +--- + +Chapter 28 read a crash after it happened. This chapter catches five bugs +*before* they get the chance to crash anything — or, in one case, watches a +tool fail to catch one at all. `bugfarm` is a single CLI, +`app `, seeding one specific memory or +concurrency defect per subcommand, built three times over. C++ has no +compile-time protection against any of the five, so it is the only language +that can express all of them; Go's garbage collector and runtime checks rule +three out by construction; Rust's borrow checker rules out four. What is +left, in every language, is watched by a different kind of tool — a runtime +binary translator, a compile-time instrumentation pass, or a symbolic +interpreter — and this chapter runs all of them, for real, against the same +five bugs, so the catch/miss matrix below is built entirely from commands +executed this session, not from what each tool's documentation claims it does. + +The code is in `examples/29-valgrind-sanitizers-miri/`. `./demo.sh build` +builds the plain binaries; each language's `README.md`-documented CMake +presets (`asan`, `tsan`) and `go build -race` build the instrumented ones +directly, since instrumented builds are outside every chapter's normal +`demo.sh` contract. + +{% include excalidraw.html + file="29-tool-coverage-matrix" + alt="A grid with defect down the rows -- leak, uninit, uaf, overflow, race -- and C++, Go, Rust across the columns. C++/leak: valgrind memcheck caught it, definitely lost. C++/uninit: valgrind memcheck caught it, uninitialised value(s); ASan alone missed it, needs MemorySanitizer. C++/uaf: valgrind caught it (invalid write, freed) and ASan caught it (heap-use-after-free). C++/overflow: valgrind caught it (0 bytes after a block) and ASan caught it (heap-buffer-overflow). C++/race: memcheck is not built for this and was not the tool tried; helgrind caught it (possible data race) and ThreadSanitizer caught it. Go/leak: self-detected via goroutines before and after; valgrind was confused, 630 errors and 0 real signal. Go/uninit, uaf, overflow: prevented by the runtime, exit 64, no tool needed. Go/race: go -race caught it, WARNING DATA RACE, exit 66. Rust/leak: valgrind caught it, 65536 bytes lost, and miri on nightly additionally reported memory leaked. Rust/uninit, uaf, overflow, race: prevented at compile time by the borrow checker, exit 64; a note says unsafe Rust plus miri catches the use-after-free class separately, shown elsewhere in the chapter. A footer legend says amber cells mean a tool run this session caught the defect, dark cells mean the language prevents it outright, and plain cells mean a real blind spot this session measured; every cell traces to a command in Build run observe or Cross-check." + caption="Figure 29.1 — the five bugfarm defects against three languages, every cell backed by a command this chapter actually ran" %} + +> **Tools used** — `valgrind` 3.27.1 (`--tool=memcheck`, `--tool=helgrind`, +> `--tool=massif`), `cmake`/`g++` 16.1.1 with the `asan`/`tsan` CMake presets +> (`-fsanitize=address,undefined` / `-fsanitize=thread`), `go build -race`, +> `cargo +nightly miri` (all host). `valgrind` and the sanitizer runtimes +> (`libasan`, `libtsan`) are hard requirements in `scripts/check-host.sh`; +> `miri` is not — it tracks Rust nightly, and this chapter documents exactly +> what happens whether or not a nightly-plus-miri toolchain is installed. + +## Two ways to watch a program: translate the binary, or instrument the source + +**valgrind** needs no rebuild and no source cooperation at all: point it at +any binary, plain `RelWithDebInfo` included, and its core (a virtual CPU +built on an IR called VEX) re-translates every basic block before it runs, +inserting extra instructions around every memory access. `memcheck`, the +default tool, gives every byte of memory a shadow bit tracking +*addressability* (has this byte been allocated?) and a shadow byte tracking +*definedness* (has this byte been written to?), checked in software on every +load and store — no hardware assist, because ordinary CPUs have no such +instruction. That software check, repeated for every single memory access of +the whole program, is the entire reason for the well-known 10-50x slowdown: +this chapter's own `uaf` subcommand took **0.002s** plain, **0.040s** under +ASan, and **0.97-0.99s** under valgrind memcheck — valgrind ran roughly +**25x slower than the sanitizer build** for the identical bug, on a binary so +small that most of that second is JIT-translation and process startup, not +steady-state emulation. `helgrind` swaps memcheck's shadow bytes for a +lockset-plus-happens-before model over the same binary-translation core (the +`race` subcommand measured 1.22s under it, slightly *heavier* than +memcheck's 1.02s on the same input); `massif` swaps in a heap-usage +sampler, taking periodic snapshots of live allocation size instead of +checking every access — running it against `leak` produced a real, +plottable curve peaking at **80.02 KB** before the process exits with the +leaked block still counted. + +Because valgrind operates on the compiled binary with no notion of what +produced it, it also has no notion of runtimes that manage their own memory +outside libc — and Go is exactly that. Pointing valgrind at the Go `leak` +binary produces **630 errors from 411 contexts**, every single one a false +"Conditional jump or move depends on uninitialised value(s)" or "Use of +uninitialised value" deep inside `runtime.lock2`, `runtime.gopark`, and +`runtime.mcall` — the goroutine scheduler's own stack-switching and lock-free +bookkeeping look like impossible, undefined states to an instruction-level +model built around one C-style stack per OS thread. Worse, the `HEAP +SUMMARY` reports **0 allocs, 0 frees, 0 bytes allocated** for the entire run, +even though the program printed a real `leaked=1` on stdout — Go's allocator +talks to the kernel directly via `mmap`, bypassing the `malloc`/`free` calls +valgrind intercepts, so its leak-tracking machinery has nothing to attach to +at all. Neither the 630 false positives nor the invisible real leak is a +configuration mistake; valgrind and the Go runtime are built on +incompatible assumptions about what a stack and a heap look like, and this +chapter's own transcript is the proof. + +**Sanitizers** take the opposite trade: instrument the *source*, at compile +time, and ship a differently-linked binary (`libasan.so`, `libtsan.so`) +whose every load and store already carries its own inline check — no +separate CPU, no re-translation, just a few extra machine instructions the +compiler emitted. ASan's mechanism is **shadow memory**: one shadow byte +covers eight bytes of application memory (`shadow_addr = (app_addr >> 3) + +offset`), and specific byte values mean specific things — `00` fully +addressable, `fa` a heap redzone the allocator placed around every block to +catch overflow, `fd` a freed block's bytes repainted so any further access +trips instantly. + +{% include excalidraw.html + file="29-shadow-memory" + alt="Two stacked bands. The upper application-memory band shows three boxes: a left redzone at addresses ending ff80 through ffff, a 4-byte int* p at address ending 0010 marked as new'd then delete'd with the note that writing through it is undefined behavior, and a right redzone at addresses ending 0018 through 007f. The lower shadow-memory band shows the corresponding three shadow bytes: fa labeled heap left redzone, fd labeled freed heap region and marked as the exact byte the write trips, and fa labeled heap left redzone again for the next block's guard. An arrow labeled shift right by 3 then add offset connects the freed block down to its fd shadow byte. A footer formula reads shadow address equals app address shifted right 3 plus offset, one comparison before every load or store, not a full CPU emulation, and a note gives the real legend from this session's ASan run -- 00 addressable, 01 through 07 partially addressable, fa heap redzone, fd freed heap region, f1 through f8 stack states, f9 global redzone -- plus the measured gap: 0.002s native, 0.046s under ThreadSanitizer, 1.0s under valgrind memcheck, for the same tiny program." + caption="Figure 29.2 — ASan's shadow-memory model, with the real freed-block shadow bytes this chapter's `uaf` run produced" %} + +Running the `uaf` binary under ASan pulls exactly the `fd` byte at +`0x7b29be3e0010` and reports `heap-use-after-free` with the freed-at and +allocated-at stacks attached; running `overflow` reports +`heap-buffer-overflow` the instant the write lands one byte into the `fa` +redzone past a 40-byte block. **LSan** is bundled into the ASan runtime by +default on Linux and runs its own reachability walk at process exit — the +same `leak` binary rebuilt under the `asan` preset reports `detected memory +leaks` with no separate leak-checking flag needed. **UBSan** rides along in +the same build (`-fsanitize=address,undefined`) but none of this chapter's +five bugs happen to trip one of its specific checks (signed overflow, +invalid enum values, and the like are its domain, not heap safety), so it +stays silent here — worth naming so its absence in these transcripts doesn't +read as a miss. **TSan** needs its own separate build (ASan and TSan cannot +be linked into the same binary) and uses yet another shadow model — a vector +clock per thread recording happens-before relationships — to catch a race +*while it actually executes*, meaning it can only report interleavings the +run actually hits. The overhead lands close to native either way: `race` +measured **0.03-0.046s** under TSan versus **0.002s** plain, roughly the +constant-factor cost of a few inlined instructions per access, not +valgrind's order-of-magnitude software CPU. + +**Go's `-race`** is the same idea under a different name: `go build -race` +links the ThreadSanitizer runtime directly into the Go binary, which is why +its `WARNING: DATA RACE` output — two conflicting accesses, each with a full +goroutine stack, plus the creation site of both goroutines — looks so much +like TSan's C++ report. It is TSan, wearing Go's tooling. + +**miri** is neither of the above: it never runs your compiled machine code +at all. It interprets the program's MIR (Rust's mid-level IR) inside an +abstract machine that tracks every allocation's provenance and validity, so +it can flag undefined behavior the borrow checker never got a chance to rule +out — specifically, UB reachable only through `unsafe`. This host's pinned +toolchain (`rust-toolchain.toml`, `1.97.1`) has no miri component (miri +tracks nightly, not stable), but a `nightly-x86_64-unknown-linux-gnu` +toolchain with miri **is** installed here, so this chapter ran it for real +rather than only documenting it. Against the actual `bugfarm` `leak` +binary, `cargo +nightly miri run -- leak` reports `error: memory leaked: +alloc1026 (Rust heap, size: 24, align: 8)` naming the exact `Box::leak` call +site — the same finding valgrind reached by a completely different route. +To see miri catch a defect *only* `unsafe` can express, a five-line +standalone snippet frees a `Box` through a raw pointer and then writes +through it anyway: + +```rust +let raw: *mut i32 = unsafe { Box::into_raw(Box::new(42i32)) }; +unsafe { + drop(Box::from_raw(raw)); // frees the allocation + *raw = 7; // UB: write through a dangling pointer +} +``` + +`cargo +nightly run` on this prints `read back: 7` and exits 0 — the +write "worked," which is exactly the danger UB represents. `cargo +nightly +miri run` on the identical source refuses: + +```console +error: Undefined Behavior: memory access failed: alloc236 has been freed, so this pointer is dangling + --> src/main.rs:12:9 +help: alloc236 was allocated here: src/main.rs:7:21 +help: alloc236 was deallocated here: src/main.rs:11:9 +``` + +That is the same use-after-free class ASan catches in C++ (right down to +naming the allocation and deallocation sites), reached through Rust's +`unsafe` escape hatch instead of a language with no such checks at all — +which is exactly the point: a type system cannot forbid what it never sees, +and `unsafe` is where Rust admits that plainly. + +## How the code works + +Each language's `main.cpp` / `main.go` / `main.rs` shares one shape: a +`Bug` enum with exactly five variants (`enum class Bug` in C++, a plain +string switch in Go, `enum Bug` in Rust), a `parse_bug` function returning +that enum or an error through each language's Chapter 5 error-handling +idiom (`std::expected`, `(string, error)`, `Result`), and one +`do_*` function per variant holding the seeded defect. The enum, not a raw +string comparison scattered through `main`, is what makes "five bugs, three +languages, one dispatch shape" a closed, exhaustive match instead of an +open-ended set of string checks — C++'s `switch` over `Bug` even gets +`std::unreachable()` after it, telling the compiler (and a reader) that +every variant is handled. + +The C++ `do_leak`, `do_uaf`, `do_uninit`, and `do_overflow` functions each +route their pointer through a `touch()` helper — +`asm volatile("" : : "r,m"(value) : "memory")`, an empty inline-asm block +that tells the compiler "something outside this translation unit might read +this value," the same trick benchmark libraries call `DoNotOptimize`. It +matters more than it looks: rebuilding `do_leak` with `touch(buf)` removed +but its per-element `buf[i] = i` loop left in place *still* reports the +leak, because GCC's escape analysis treats a write into heap memory as an +observable side effect it won't remove without proof nothing depends on it. +Strip the loop too — an allocation nothing ever writes to or reads — and +`HEAP SUMMARY` drops straight to **0 allocs, 0 frees, 0 bytes allocated**: +GCC proved the entire `new[]` unobservable and deleted it, malloc and all, +confirmed directly on this host. `touch()` is what closes that gap; without +it (or without a loop that keeps writing), there is no bug left to catch, +which is itself the lesson — an unobserved bug is dead code, and the +compiler is entitled to remove dead code. Rust's `leak()` uses +`std::hint::black_box(leaked.as_ptr())` for the identical reason: LLVM can +see nothing observable depends on the buffer's *contents* (only its +statically-known `.len()` gets printed), so without the barrier the whole +64 KiB `vec![0xAAu8; 65536]` — and its `Box::leak` — can vanish too. + +The bugs themselves are small and literal. `do_uaf` calls `delete p` +then writes `*p = 7` through the now-dangling pointer; `do_uninit` +allocates `new int` with no initializer and reads it; `do_overflow` +writes `arr[10]` into a 10-element array; `do_race` spawns two +`std::jthread`s that each increment a shared, unguarded `int counter` in a +tight loop, then joins both (via `jthread`'s own RAII) before printing the +result. Go's `leak()` spawns a goroutine blocked forever on an unbuffered +channel receive nobody will ever send to, then compares +`runtime.NumGoroutine()` before and after — there is no external tool +watching the Go heap for this the way valgrind watches `malloc`, so the +program has to notice its own leak. Rust's `leak()` is the one bug safe +Rust can still express: `Box::leak` is a real, safe standard-library API +whose entire purpose is opting an allocation out of `Drop`-based +reclamation, which is why it is the only row in the matrix where Rust has +anything for valgrind or miri to find at all. + +## Errors, three ways + +Bad usage is identical across all three: no subcommand, or an unrecognized +one, prints `usage: app ` alone (for no +argument) or `app: unknown bug: ` above that same usage line (for an +unrecognized one), and exits **2** — the same shape as every other +chapter's `app`. The interesting three-way split is what each language does +with a bug it *cannot* express: Go prints `go: prevented by the runtime — +see chapter` for `uaf`/`uninit`/`overflow`, Rust prints +`rust: prevented at compile time — see chapter` for +`uaf`/`uninit`/`overflow`/`race`, and both exit **64** (`EX_USAGE`) rather +than attempting anything. That fixed message and exit code *is* the +finding for those cells — not a placeholder, but the same "no reproducible +bug here" result a real investigation would reach chasing the equivalent +C++ defect in that language, arrived at by a runtime check or a rejected +compilation instead of a crash. + +## Concurrency lens: TSan vs. helgrind vs. `-race` + +The race is one unsynchronized `int counter` incremented from two threads +with no lock — the canonical lost-update bug — and this chapter ran all +three of that bug's detectors against it directly. **ThreadSanitizer** +(the C++ `tsan` preset) named the exact colliding line, +`main.cpp:67` (`++counter;`), printing both threads' stacks and a summary +that exits **66**; a plain, uninstrumented three-run sample of the same +binary showed the race is not guaranteed to *manifest* every time — +`counter=400000` (the mathematically correct total for the workload) came +back in one plain sample, while the TSan-instrumented run's altered timing +produced `counter=200000`, half the expected total, alongside its report. +**helgrind**, valgrind's other thread tool, found the identical line +through a completely different mechanism — a lockset-and-happens-before +model instead of TSan's vector clocks — reporting "Possible data race... +at (anonymous namespace)::bump() (main.cpp:67)" between the same two +threads, both created with "Locks held: none." **`go -race`** found Go's +version of the same shape at `main.go:70`, printing `WARNING: DATA RACE` +for the conflicting pair and `Found 2 data race(s)` before exiting **66**; +a plain Go run in this same session came back `counter=100302` instead of +the expected `200000` — a visibly corrupted count, unlike the C++ plain +run's lucky correct total, which is itself a reminder that "it printed the +right number" is not evidence a race is absent. Rust needs none of this: +the race subcommand exits 64 with `prevented at compile time`, because +sharing `&mut i32` across `std::thread::spawn` boundaries without +synchronization is rejected by the `Send`/`Sync` rules before there is a +binary to run a detector against at all. + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +int counter = 0; // race: shared, unsynchronized on purpose + +void bump() { + for (int i = 0; i < 200'000; ++i) { + ++counter; + } +} +``` + +```go +var counter int // race: shared, unsynchronized on purpose + +func bump(wg *sync.WaitGroup) { + defer wg.Done() + for i := 0; i < 100_000; i++ { + counter++ + } +} +``` + +```rust + match bug { + Bug::Leak => { + leak(); + Ok(()) + } + Bug::Uaf | Bug::Uninit | Bug::Overflow | Bug::Race => { + eprintln!("{PREVENTED}"); + Err(64) + } + } +``` + +The C++ and Go excerpts are the bug itself, written by hand in both +languages; the Rust excerpt is the same `Bug::Race` variant reaching the +`match` arm that refuses to try — there is no racy Rust function to show, +because the compiler never let one exist. + +## Build, run, observe + +```console +[host]$ cd examples/29-valgrind-sanitizers-miri && ./demo.sh build +``` + +```console +[host]$ valgrind --leak-check=full --error-exitcode=1 ./cpp/build/release/app leak +bugfarm: leak: allocated 4096 bytes on the heap, never freed (intentional) +==...== 4,096 bytes in 1 blocks are definitely lost in loss record 1 of 1 +``` + +Build the instrumented variants directly — they sit outside `demo.sh`'s +build/run contract on purpose, since this chapter (unlike most) is about +the tool, not a happy-path run: + +```console +[host]$ cd cpp && cmake --preset asan && cmake --build --preset asan +[host]$ cmake --preset tsan && cmake --build --preset tsan +[host]$ ./build/asan/app uaf +==...==ERROR: AddressSanitizer: heap-use-after-free on address ... +[host]$ ./build/tsan/app race +WARNING: ThreadSanitizer: data race (pid=...) +[host]$ go build -race -o bin/app-race . && ./bin/app-race race +WARNING: DATA RACE +``` + +`python3 scripts/test-all-examples.py --only 29-valgrind-sanitizers-miri` +reports `PASS PASS PASS`: the plain build and bad-usage/prevented-message +contract per language, then the C++ suite (memcheck on `leak`/`uninit`, +ASan on `uaf`/`overflow`, TSan on `race`), Go's self-detected `leak` plus +`-race`'s `race`, and Rust's valgrind-on-`leak` plus its informational +miri probe. + +## Cross-check: the same use-after-free, two tools, two speeds + +The clearest single proof that these are two different techniques, not two +brands of the same one, is running the identical bug under both: the C++ +`uaf` binary, unmodified, first under valgrind, then rebuilt under ASan. + +```console +[host]$ valgrind --error-exitcode=1 ./cpp/build/release/app uaf +==...== Invalid write of size 4 +==...== at 0x400A3B: do_uaf (main.cpp:87) +==...== Address 0x4e7f080 is 0 bytes inside a block of size 4 free'd +==...== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) +``` + +```console +[host]$ ./cpp/build/asan/app uaf +==...==ERROR: AddressSanitizer: heap-use-after-free on address 0x7b29be3e0010 +WRITE of size 4 at 0x7b29be3e0010 thread T0 + #0 do_uaf .../main.cpp:87 +freed by thread T0 here: ... #1 do_uaf .../main.cpp:85 +previously allocated by thread T0 here: ... #1 do_uaf .../main.cpp:84 +``` + +Both name line 87 as the bad write and lines 84/85 as the allocation and +free — the same defect, independently confirmed. What differs is *how much +work* got them there: valgrind's memcheck took **0.97-0.99s** (repeated +twice for stability) against the plain binary with no rebuild required; +the ASan build, needing `cmake --preset asan` first, then ran in +**0.040s** — about **25x faster** — and its report additionally carries a +full shadow-byte dump (`fa fa fa fa[fd]fa fa ...`) that pinpoints exactly +which byte tripped the check, something memcheck's report describes in +words but ASan shows you directly. Neither number should be read as "the" +overhead for real programs — this binary does almost nothing, so both +times are dominated by fixed startup cost — but the *ratio*, valgrind an +order of magnitude slower than a rebuild-and-relink sanitizer for +identical output, is exactly the trade-off the tool-selection question in +this chapter's opening comes down to: rebuild if you can, translate the +binary if you cannot. + +## What you learned + +- valgrind instruments the *binary* at runtime (no rebuild, works on + anything) at the cost of a software-emulated CPU — measured at roughly + **25x** the ASan build's time for an identical `uaf` report — and that + same binary-translation model has no concept of a runtime that manages + its own stacks and heap outside libc, which is why pointing it at a Go + binary produced **630 false-positive errors** and a heap summary blind + to a real, on-stdout goroutine leak. +- Sanitizers instrument the *source* at compile time via shadow memory (one + byte per eight application bytes, `>>3 + offset`), trading a required + rebuild for overhead close to native (TSan's race run at **0.03-0.046s** + versus plain's **0.002s**) — and LSan/UBSan ride along in the same ASan + build rather than needing separate invocations. +- miri interprets MIR symbolically instead of running compiled machine + code at all, which lets it catch undefined behavior reachable only + through `unsafe` — demonstrated live on this host's nightly toolchain + against both the real `Box::leak` bugfarm binary and a standalone + use-after-free snippet, naming the exact allocation and deallocation + sites the same way ASan does for C++. +- The same race bug, checked three ways (TSan, helgrind, `go -race`), gets + three different detection mechanisms (vector clocks, lockset/happens-before, + and TSan borrowed wholesale) converging on the identical source line — + while Rust's `Send`/`Sync` rules mean there is no racy binary for any of + them to point at in the first place. + +Next, **the eBPF observation toolkit**: bcc-tools and bpftrace watch these +same kinds of programs from outside entirely, with no instrumentation and +no rebuild, on the lab VM this chapter's tools never needed. + +--- + +

verified — every transcript, +number, and diagnostic above was produced on the Fedora 44 reference host +(kernel 7.1.3-200.fc44, valgrind-3.27.1, gcc 16.1.1, go1.26.5, rustc/cargo +1.97.1, miri 0.1.0 nightly-2026-07-18) this session. Confirmed live: +valgrind memcheck reported definitely lost for leak +and uninitialised value(s) for uninit, and also +caught uaf (Invalid write ... freed) and +overflow (Invalid write ... 0 bytes after a block) +on the plain binary; the ASan build reported heap-use-after-free +and heap-buffer-overflow with real shadow-byte dumps, and its +bundled LSan reported detected memory leaks for leak; +the TSan build reported data race at main.cpp:67 +(exit 66) and helgrind independently reported Possible data race +at the same line; go build -race reported +WARNING: DATA RACE at main.go:70, +Found 2 data race(s) (exit 66), and a plain Go run showed a +corrupted counter=100302 against the expected 200000; valgrind +against the Go leak binary produced 630 errors from 411 +contexts (all inside the Go runtime) and a heap summary of +0 allocs, 0 frees, 0 bytes allocated; valgrind against the Rust +leak binary reported 65,536 bytes ... definitely lost, +and cargo +nightly miri run -- leak independently reported +memory leaked: alloc1026; a standalone unsafe snippet ran clean +under cargo +nightly run (read back: 7) and was +rejected by cargo +nightly miri run with +Undefined Behavior: memory access failed: alloc236 has been freed; +massif against leak produced a real peak of +80.02 KB; timing the same uaf binary gave +0.002s plain, 0.040s under ASan, and +0.97-0.99s under valgrind memcheck (repeated twice); and +removing every observable use of do_leak's buffer (not just its +touch() barrier, but its per-element writes too) reproducibly +collapsed valgrind's report to 0 allocs, 0 frees, 0 bytes allocated +on this host's GCC 16.1.1 at -O2.

diff --git a/_docs/30-ebpf-observation-toolkit.md b/_docs/30-ebpf-observation-toolkit.md new file mode 100644 index 0000000..c0e28f2 --- /dev/null +++ b/_docs/30-ebpf-observation-toolkit.md @@ -0,0 +1,742 @@ +--- +title: "The eBPF Observation Toolkit" +order: 30 +part: "Debugging" +description: "app work --seconds N runs an ordinary, unmodified workload on systems-target, and this chapter points the standard eBPF observation stack at it live: bcc-tools' execsnoop/opensnoop/funccount/offcputime/biolatency (and a real, plainly-reported profile failure), a bpftrace crash course from one-liners to a uprobe+uretprobe latency histogram, bpftool naming the loaded programs, c++filt demangling the exact mangled symbol a uprobe attaches to in C++ and Rust, and a live, measured RDI-vs-RAX register mismatch that shows exactly why uprobes on Go need more care than the other two — with SystemTap's kernel-module and dyninst backends compared alongside, one working and one blocked by a real kernel-devel mismatch." +duration: "55 minutes" +--- + +Chapter 28 read a crash gdb already knew about; Chapter 29 caught bugs a +compile-time instrumentation pass or a binary-translating emulator watched +for. Both chapters needed something extra glued onto the target: debug +info gdb could walk, or a sanitizer/valgrind rebuild. This chapter needs +neither. `app work --seconds N` is an ordinary, unmodified binary — no +special build, no `-fsanitize`, not even a debugger attached — and the +standard eBPF observation stack (bcc-tools, bpftrace, bpftool) watches it +entirely from outside, live, on the `systems-target` lab VM this book has +used since Part 3. The workload is deliberately small and legible: every +~230ms it opens/writes/closes one fixed file, forks and execs a short-lived +`true` every fourth iteration, calls one named hot function `busy_hash()`, +and sleeps through most of the interval — four separate baits for five +different tools, wired identically into C++23, Go, and Rust so the same +observation session works against all three languages' binaries. This +chapter also attaches uprobes to *our own* symbols directly, which is where +C++ name mangling, Rust's own mangling scheme, and Go's register-based +calling convention each become a real, measured concern rather than a +footnote. + +The code is in `examples/30-ebpf-observation-toolkit/`. `./demo.sh build` +builds all three; `observe.sh [seconds]`, run as root on +the lab guest, is this chapter's actual demo — it starts the workload and +points five tools at it in turn, and the top-level `README.md` documents +the full contract and a real run's tail. + +{% include excalidraw.html + file="30-ebpf-tooling-stack" + alt="Three stacked bands. The top kernel band holds five small boxes -- kprobe/kretprobe on kernel functions, uprobe/uretprobe on userspace functions, tracepoint as the stable kernel ABI, perf_event as timer/PMU sampling, and BTF/CO-RE for struct layout without headers. Amber arrows descend from three of these into the middle eBPF-tooling band, which holds bcc-tools (execsnoop/opensnoop/biolatency/offcputime/funccount/profile), bpftrace (one-liners and scripts, probe:target /filter/ action), bpftool (prog show/map show), and c++filt (demangle the exact symbol you attach to), with a note that libbpf loads a small verified BPF program per probe that the kernel runs in-kernel on each hit. Below that, a third band holds SystemTap as the sibling technology: its classic kernel-module backend needing matching kernel-devel, beside stap --runtime=dyninst, which patches the target binary directly in userspace with no kernel module. A fourth band at the bottom shows the three target binaries -- C++'s busy_hash under the SysV ABI with a correct arg0, Rust's cmd_work also correct under the SysV ABI once c++filt demangles its v0-mangled name, and Go's cmdWork in a dark box reading arg0/RDI wrong at 1, RAX right at 9, with a dashed amber arrow from it up to the eBPF-tooling band labeled arg0 mismatch. A footer note says every amber box ran live against systems-target this session; SystemTap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, and the dyninst backend attached without erroring." + caption="Figure 30.1 — the eBPF observation stack this chapter exercises live: kernel probe primitives at the bottom, bcc-tools/bpftrace/bpftool/c++filt above them, SystemTap as the sibling technology beside them, and the one real register-ABI mismatch that only shows up on the Go binary" %} + +> **Tools used** — bcc-tools `execsnoop`, `opensnoop`, `funccount`, +> `offcputime`, `biolatency`, `profile` (`systems-target` VM, root); +> `bpftrace` v0.24.2 (`systems-target` VM, root); `bpftool` (`systems-target` +> VM, root); `c++filt` (host, to demangle before attaching); `nm` (host, to +> find the real symbol names); `stap` 5.5 (`systems-target` VM, root, for +> the SystemTap comparison callout). Everything under `systems-target` is +> provisioned by `cloud-init` there per `references/vm-lab.md`; `c++filt` +> and `nm` are part of `binutils` on `scripts/check-host.sh`. + +## Watching from outside: the third way to observe a program + +Chapter 29 drew a line between two ways of catching a bug: **translate the +binary** (valgrind, no rebuild, a software-emulated CPU) or **instrument the +source** (sanitizers, a required rebuild, near-native overhead). eBPF-based +observation is a third way, and it needs neither: point a probe at a running +kernel function, a running userspace function, or a stable tracepoint, and +the **kernel itself** runs a small, verified BPF program on every hit, +copying whatever you asked for back to userspace through a map or a ring +buffer. No rebuild, because the probe attaches to the compiled code exactly +as it stands; no restart even, because a `uprobe` can attach to a process +that is already running. The trade is `CAP_BPF`/`CAP_PERFMON` — on this lab +image that means root, which is why every tool in this chapter runs under +`sudo` on `systems-target` and never on the host, matching this book's rule +that eBPF is tooling we point at our own userspace programs, never kernel +code we write ourselves. + +Four probe types cover everything this chapter touches. A **kprobe** (and +its return-side twin, **kretprobe**) attaches to almost any non-inlined +kernel function by name — `vfs_write`, say — and fires on entry or return. +A **uprobe**/**uretprobe** is the identical idea one level up: instead of a +kernel symbol, it takes a userspace **binary path plus a symbol name** (or a +raw offset), and it needs nothing from the target process except that the +symbol exist in its ELF symbol table — no debug info, no cooperation, no +special build. A **tracepoint** is a stable, kernel-maintained hook baked +into specific spots (`syscalls:sys_enter_openat` fires on every `openat(2)` +entry, arguments included) that will not move or disappear the way a raw +kprobe's target function occasionally does across kernel versions. A +**perf_event** probe fires on a timer or a PMU counter rather than a +specific code location — that is what an on-CPU sampling profiler rides on. +BTF (BPF Type Format) is the piece that lets modern tools read kernel +`struct` layouts without a matching `kernel-devel` package at all, which +matters directly to this chapter's SystemTap comparison below. + +## bcc-tools: five purpose-built tools, one workload + +**bcc** ships as a directory of small, ready-made tools +(`/usr/share/bcc/tools/` on this image), each one a purpose-built BPF +program with a Python front end — you do not write BPF C for any of these, +you invoke the tool and read its output. `observe.sh` runs five of them +against `app work --seconds N` in sequence, each one confirming a specific +bait fired. A real run against the C++ binary, staged at +`/tmp/lsp30-cpp/app` on `systems-target`: + +```console +[vm]$ sudo bash /tmp/lsp30-observe.sh /tmp/lsp30-cpp/app 40 +=== opensnoop (open() bait) === +15320 app 3 0 /tmp/lsp-ebpf-work-bait.txt +15320 app 3 0 /tmp/lsp-ebpf-work-bait.txt +... +=== execsnoop (fork/exec bait) === +true 15339 15320 0 /usr/sbin/true +true 15340 15320 0 /usr/sbin/true +... +=== funccount busy_hash (uprobe/funccount bait) === +FUNC COUNT +busy_hash 21 +=== offcputime (sleep off-CPU bait) === + - app (15322) + __clock_nanosleep_2 + do_nanosleep + hrtimer_nanosleep +=== bpftrace uprobe:busy_hash === +Attached 2 probes +@calls: 22 +=== summary === +opensnoop_opens=20 +execsnoop_execs=6 +funccount_calls=21 +offcputime_hits=2 +bpftrace_calls=22 +app_summary: work: done seconds=40 iters=173 opens=173 execs=44 busy_calls=173 busy_hash=1836395105209072940 +``` + +`opensnoop` traces every `open`/`openat` and names the exact path our +workload writes, by comm `app`, not an inference from timing. `execsnoop` +shows `true` with **PPID 15320**, our workload's own pid — proof this is +*our* fork/exec, not unrelated system noise. `funccount` attaches a uprobe +at the pattern `:*busy_hash*` and just counts hits, reporting +**21** for a ~40s window at ~230ms/iteration (173 total iterations, sampled +mid-run for a shorter tool budget). `offcputime` samples stacks of threads +that are *not* running, bucketed by how long they were off-CPU, and the +sleep inside every iteration reliably parks our thread in +`__clock_nanosleep_2` — exactly the bait. `biolatency` is the fifth tool in +the family tour, run separately against real block I/O (the lab guest's +`/tmp` is tmpfs, so this needs the root filesystem): + +```console +[vm]$ sudo timeout 6 /usr/share/bcc/tools/biolatency 1 3 +Tracing block device I/O... Hit Ctrl-C to end. + usecs : count distribution + 128 -> 255 : 3 |*************** | + 256 -> 511 : 4 |****************************************| + 512 -> 1023 : 2 |********************* | +``` + +That histogram came from five `dd ... conv=fsync` writes to `/var/tmp` +completing in the low hundreds of microseconds — a real, measured +distribution, not a synthetic example. `profile`, the sixth named tool, is +where this tour hit a real wall worth reporting rather than hiding: + +```console +[vm]$ sudo /usr/share/bcc/tools/profile -F 49 -p 15912 3 +include/linux/ns_common.h:25:23: error: no member named 'ns_id' in 'struct ns_common' +... +1 warning and 3 errors generated. +Traceback (most recent call last): + File "/usr/share/bcc/tools/profile", line 314, in + b = BPF(text=bpf_text) +``` + +`profile` compiles a small C snippet against the running kernel's own +headers at load time — even bcc tools with BTF support are not entirely +immune to kernel drift — and on this lab guest's kernel (6.19.10) that +snippet references a `struct ns_common` field the installed headers no +longer have under that name. bpftrace's own `profile:hz:N` probe type +sidesteps the header dependency and attached cleanly, but on this exact +workload it reported **zero samples** at both 49Hz and 99Hz over several +seconds — not a bug, but the real result of an on-CPU sampler pointed at +a process that spends over 99% of its time asleep in `nanosleep`. +`offcputime` and `profile` are answering two different questions (where did +the *waiting* time go, versus where did the *running* time go), and a +workload this sleep-dominated has an answer for exactly one of them. + +## bpftrace: a crash course from one-liners to a script + +**bpftrace** is a small, high-level language over the same probe types: +`probe:target /filter/ { action }`, with built-ins (`pid`, `comm`, `arg0` +through `argN`, `retval`, `str()`, `nsecs`) and aggregations (`count()`, +`hist()`, `@name` maps) that erase most of the boilerplate a hand-written +BPF program would need. A tracepoint one-liner, filtered to our workload's +real pid: + +```console +[vm]$ sudo timeout 4 bpftrace -e 'tracepoint:syscalls:sys_enter_openat + /pid == 17092/ { printf("openat comm=%s file=%s\n", comm, str(args.filename)); }' +Attached 1 probe +openat comm=app file=/tmp/lsp-ebpf-work-bait.txt +openat comm=app file=/tmp/lsp-ebpf-work-bait.txt +... +``` + +A kprobe one-liner aggregating into a histogram — every `vfs_write` call +from that same pid, bucketed by byte count (`arg2` is `vfs_write`'s third +kernel argument, the length): + +```console +[vm]$ sudo timeout 3 bpftrace -e 'kprobe:vfs_write /pid == 17092/ + { @bytes = hist(arg2); } interval:s:3 { print(@bytes); exit(); }' +@bytes: +[8, 16) 13 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| +[16, 32) 3 |@@@@@@@@@@@@ | +``` + +Every bait line the workload writes (`"iter N\n"`) is 8-15 bytes most of +the time, 16-31 once digits roll over — the histogram is reading the exact +shape of our own `write(2)` calls. The short script below is the crash +course's third piece: a **uprobe/uretprobe pair**, timing one function +entry-to-return and bucketing the result, saved as +`busy_hash_latency.bt`: + +``` +uprobe:$1:busy_hash { + @start[tid] = nsecs; +} + +uretprobe:$1:busy_hash /@start[tid]/ { + @latency_ns = hist(nsecs - @start[tid]); + delete(@start[tid]); +} + +interval:s:6 { exit(); } +``` + +`@start[tid]` is a per-thread-id associative map — the entry probe stashes +a timestamp keyed by the calling thread, and the matching return probe +looks it up, computes the delta, and folds it into a histogram before +deleting the entry so a later call cannot read a stale timestamp. Run +against the live C++ binary: + +```console +[vm]$ sudo timeout 8 bpftrace /tmp/busy_hash_latency.bt /tmp/lsp30-cpp/app +Attached 5 probes +@latency_ns: +[256K, 512K) 4 |@@@@@@@@@@ | +[512K, 1M) 3 |@@@@@@@@ | +[1M, 2M) 19 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| +``` + +`busy_hash`'s 300,000-round xorshift loop takes roughly 1-2ms most calls on +this host — a real, measured latency distribution for a function this +book's example code deliberately keeps simple, produced with five lines of +bpftrace and no rebuild of `app` at all. + +## bpftool: naming what is actually loaded + +**bpftool** is the inspection half of the story — it reads back what the +kernel currently has loaded, independent of whichever tool put it there. +With the `busy_hash` uprobe from the previous section still attached in a +background shell, a second root shell on the same guest: + +```console +[vm]$ sudo bpftool prog show +935: kprobe name uprobe__tmp_lsp30_cpp_app_busy_hash_1 tag 1c2c09b63327bc49 gpl + loaded_at 2026-07-19T17:16:51+0000 uid 0 + xlated 848B jited 467B memlock 4096B map_ids 24 +``` + +Two things are worth reading closely here. First, `bpftrace` compiled the +attach point's name straight into the BPF program's own name +(`uprobe__tmp_lsp30_cpp_app_busy_hash_1`) — `bpftool` is reading that name +back from the kernel, not from bpftrace's own bookkeeping, so this is an +independent confirmation the probe really loaded. Second, the kernel +classifies this program's `type` as plain **`kprobe`** even though we asked +bpftrace for a `uprobe` — uprobes and kprobes share the same underlying BPF +program type at the kernel ABI level; the distinction between "kernel +function" and "userspace function" lives in how the probe is *attached* +(a kernel symbol versus a binary path + symbol), not in a separate program +type. `bpftool map show` in the same window lists the maps bpftrace +allocated for its own script state, each one tagged with the owning +process: + +```console +[vm]$ sudo bpftool map show +555: array name 56464433.rodata flags 0x480 + key 4B value 16B max_entries 1 memlock 8192B + btf_id 605 frozen + pids bpftrace(16154) +``` + +`pids bpftrace(16154)` is the cross-check: this map is held open by the +exact bpftrace process running our script, not a leftover from some +earlier session — `bpftool` and `bpftrace` are two independent programs +agreeing on what is actually resident in the kernel right now. + +## SystemTap: the sibling technology + +**SystemTap** predates eBPF-based tracing by close to a decade and solves +the same problem — attach a probe, run a small script, read the answer — +through a different mechanism: its classic backend compiles your script +into a loadable **kernel module**, which means it needs a `kernel-devel` +package that matches the *exact* running kernel, not just a recent one. +This lab guest's mismatch is real and was captured, not invented: + +```console +[vm]$ sudo stap -e 'probe begin { println("stap alive"); exit() }' +Checking "/lib/modules/6.19.10-300.fc44.x86_64/build/.config" failed with error: No such file or directory +Incorrect version or missing kernel-devel package, use: dnf install kernel-devel-6.19.10-300.fc44.x86_64 +``` + +The installed `kernel-devel` on this image is built for `7.1.3-201.fc44`, +one version off the actually-running `6.19.10-300.fc44` — exactly the +class of drift bpftrace and bcc's BTF-backed tools were built to avoid. +SystemTap also ships a userspace-only **dyninst** backend +(`--runtime=dyninst`), which patches the target process's own code directly +and needs no kernel module at all: + +```console +[vm]$ sudo stap --runtime=dyninst -e + 'probe process("/tmp/lsp30-cpp/app").function("busy_hash") { println("stap saw busy_hash"); exit() }' \ + -x 17092 +Pass 5: run completed in 9810usr/680sys/13771real ms. +``` + +That attached and ran to completion with no compile or attach error, but it +printed nothing in this session's test window — a real, if inconclusive, +result reported as such rather than claimed as a hit. The real comparison +this callout can make from live evidence: SystemTap's classic path is more +sensitive to exact kernel-version matching than the BTF-based bpftrace/bcc +path used everywhere else in this chapter (which needed no matching +`kernel-devel` at all), while its dyninst backend trades that dependency +for a userspace-patching approach this session did not get a positive +result from in the time available. SystemTap has real strengths bpftrace +does not — a proper scripting language and years of production hardening +among them — and Chapter 45 covers it in depth; this callout is only the +comparison this chapter's own live evidence supports. + +## How the code works + +Every observation above needs a target worth watching, and `app work +--seconds N` is written specifically to be one: four deliberate baits, one +per tool family, wired identically into C++23, Go, and Rust so `observe.sh` +never needs a per-language branch. Each language keeps one counters +structure the loop mutates and the final summary line reads back: + +```cpp +struct Counters { + std::atomic iters{0}; + std::atomic opens{0}; + std::atomic execs{0}; + std::atomic busy_calls{0}; + // The accumulated busy_hash() result. Printed in the summary line below -- + // that's what keeps the optimizer from proving the whole busy_hash call + // chain's result is unused and deleting it outright (noinline only stops + // inlining, not interprocedural dead-code elimination of a provably-pure, + // provably-unused call chain). + std::atomic busy_hash_val{0}; +}; +``` + +C++'s fields are `std::atomic` because the workload loop runs on a +`std::jthread` while `main` could in principle read them concurrently; +Go's equivalent `result` struct has no such requirement (a single goroutine +owns it end to end, handed to `main` once, complete, over a channel) and +Rust's `Counters` is a plain `Default`-derived struct for the same reason — +the concurrency shape, not the language, decides whether the counters need +to be atomic. The comment on `busy_hash_val` documents this chapter's own +`README.md`-recorded gotcha: the first version of this workload printed +`busy_calls` but never the accumulated hash, and GCC's interprocedural +analysis proved the entire `busy_hash()` call chain had no observable +effect and deleted it — `funccount` and the bpftrace uprobe both reported +**zero** calls even though `nm`/`objdump` still showed the symbol sitting +right there in the binary. `noinline` only forbids folding the function into +its caller; it says nothing about deleting a call whose result nothing +reads. Printing the final hash in the `done` line, in all three languages, +is what makes the call chain provably necessary again — a fragile-but-named +requirement worth restating here: if you trim this program down, keep the +final `busy_hash` print or the whole bait disappears. + +The bait-producing helpers are small and literal, and each closes its own +resource idiomatically rather than by hand. C++'s `Fd` is a one-field RAII +wrapper whose destructor calls `::close` no matter how the scope is left; +Rust's `open_bait` returns a `rustix::fs::OwnedFd` and never calls +`close(2)` at all, because the `OwnedFd`'s `Drop` impl does it the moment +the value goes out of scope at the end of the function; Go relies on +`defer f.Close()` right after the successful open, the same "close no +matter what" guarantee spelled a third way. All three open the bait file +`O_CREAT|O_WRONLY|O_TRUNC` at mode `0666` — world-writable on purpose, +since the file may be created once by a root-run `observe.sh` and again by +an unprivileged `demo.sh run`, and a stricter mode would make the second +owner's open fail. `spawn_true`/`spawnTrue` forks (or lets the language +runtime fork on its behalf, for Go and Rust's `std::process::Command`) and +execs `true`, then waits for it — the exact shape `execsnoop` watches, with +the child's real parent pid always the workload's own. + +The loop itself, `run_workload`/`runWorkload`, is the wiring: on every pass +it calls `open_bait` (opensnoop bait), execs `true` every fourth iteration +(execsnoop bait), calls `busy_hash()` once (funccount/uprobe bait), then +sleeps ~230ms (offcputime bait), incrementing counters as it goes until a +deadline computed from `--seconds` at start. C++ runs this loop on a +`std::jthread` and `main` calls `worker.join()` explicitly, so the summary +line always sees the final counters rather than racing the still-running +thread; Go starts it as a goroutine and blocks `main` on a buffered result +channel — the channel receive *is* the join; Rust runs it straight-line on +the main thread, no concurrency at all, since nothing else in the Rust +build needs to run alongside it. `cmd_work`/`cmdWork` is the one-line +dispatcher tying `main`'s parsed `--seconds` value to this loop and printing +the `start`/`done` lines around it — the exact function this chapter's +uprobe/mangling section below attaches a probe to directly. The named hot +function itself, verbatim in all three languages: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +// The named hot function every uprobe/funccount/bpftrace command in this +// chapter targets by name. extern "C" so the symbol isn't mangled, matching +// the Rust build; noinline so -O2 can't fold it into the caller and make the +// uprobe's attach point disappear. +extern "C" [[gnu::noinline]] std::uint64_t busy_hash(std::uint64_t x) noexcept { + for (std::uint64_t i = 0; i < kBusyRounds; ++i) { + x ^= x << 7; + x ^= x >> 9; + } + return x; +} +``` + +```go +// busy_hash is the named hot function every uprobe/funccount/bpftrace command +// in this chapter targets by name. Go 1.26 emits it as "main.busy_hash" in +// the symbol table; //go:noinline keeps the compiler from folding it into its +// caller and making the uprobe's attach point disappear. +// +//go:noinline +func busy_hash(x uint64) uint64 { + for i := 0; i < busyRounds; i++ { + x ^= x << 7 + x ^= x >> 9 + } + return x +} +``` + +```rust +/// The named hot function every uprobe/funccount/bpftrace command in this +/// chapter targets by name. `#[unsafe(no_mangle)]` keeps the symbol as the +/// plain string "busy_hash" (matching the C++ build); `#[inline(never)]` +/// keeps the optimizer from folding it into its caller and making the +/// uprobe's attach point disappear. +#[unsafe(no_mangle)] +#[inline(never)] +pub extern "C" fn busy_hash(mut x: u64) -> u64 { + for _ in 0..BUSY_ROUNDS { + x ^= x << 7; + x ^= x >> 9; + } + x +} +``` + +## Errors, three ways + +The CLI contract is this book's usual shape and is byte-identical across +languages — `app work --seconds N` on success, and a bad or missing +`--seconds` prints a two-line usage/diagnostic message and exits **2** in +every language, confirmed live: + +```console +[vm]$ /tmp/lsp30-cpp/app work; echo exit=$? +usage: app work --seconds N +exit=2 +[vm]$ /tmp/lsp30-go/app work --seconds nope; echo exit=$? +work: bad --seconds value: nope +exit=2 +[vm]$ /tmp/lsp30-rust/app; echo exit=$? +usage: app work --seconds N +exit=2 +``` + +The more interesting three-way split is what the *observation tools* +report when a target symbol does not exist, and it differs by tool as much +as by language. Asking `bpftrace` to attach to a function that is not in +the binary at all fails loudly and refuses to attach: + +```console +[vm]$ sudo bpftrace -e 'uprobe:/tmp/lsp30-cpp/app:no_such_function { printf("never\n"); }' +ERROR: Could not resolve symbol: /tmp/lsp30-cpp/app:no_such_function +ERROR: Unable to attach probe: uprobe:.../app:no_such_function. If this is expected, set the 'missing_probes' config variable to 'warn'. +``` + +`funccount`, by contrast, treats an unmatched pattern as an **empty result +set**, not an error — asking it for the exact unqualified name `busy_hash` +against the *Go* binary, which actually exports `main.busy_hash`: + +```console +[vm]$ sudo /usr/share/bcc/tools/funccount -p 1 -d 1 '/tmp/lsp30-go/app:busy_hash' +No functions matched by pattern b'^busy_hash$' +``` + +That single line is the entire reason `observe.sh` and `verify.lua` use the +wildcard `:*busy_hash*` instead of the exact name everywhere in this +chapter: it is the one pattern that matches C++'s and Rust's plain +`busy_hash` *and* Go's package-qualified `main.busy_hash`, without a +per-language branch anywhere in the driver script. + +## What each binary gives the uprobe tooling + +`busy_hash` is deliberately kept unmangled in every language +(`extern "C"` in C++, `#[unsafe(no_mangle)]` in Rust, and Go never mangles +at all) precisely so one wildcard pattern (`*busy_hash*`) works across all +three without a per-language branch anywhere in `observe.sh`. Attaching to +any *other* function in the same binaries is where each language's real +symbol story — and one real ABI trap — shows up. `cmd_work`, the function +that dispatches `--seconds` into the workload loop, carries no such +protection in any language, so `nm` shows what each compiler really does +with a name by default: + +```console +[host]$ nm cpp/build/release/app | grep cmd_work +00000000004025b0 t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 +[host]$ nm rust/target/release/app | grep cmd_work +000000000001b1d0 t _RNvCs6FzKXswKZCc_3app8cmd_work +[host]$ nm go/bin/app | grep -i cmdWork +00000000004c6760 T main.cmdWork +``` + +**C++** mangles by the Itanium ABI: `cmd_work` sits in an anonymous +namespace (`(anonymous namespace)::`), which the mangler folds into +`_ZN12_GLOBAL__N_1L8cmd_workEi`, and GCC's interprocedural specialization +pass appended `.isra.0` (a clone with some arguments replaced by scalars, +"interprocedural scalar replacement of aggregates" — here a no-op clone, +but the suffix survives regardless). `c++filt` demangles it without any +flag, auto-detecting the Itanium scheme: + +```console +[host]$ c++filt _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 +(anonymous namespace)::cmd_work(int) [clone .isra.0] +``` + +**Rust** mangles under its own **v0** scheme (the `_R` prefix), and modern +`c++filt` auto-detects that too — no `--format=rust` flag needed on this +host's binutils: + +```console +[host]$ c++filt _RNvCs6FzKXswKZCc_3app8cmd_work +app[4db24ee1e6ed4506]::cmd_work +``` + +**Go never mangles anything.** `main.cmdWork` *is* the linker symbol, +package-qualified and immediately readable — there is no demangling step +for Go at all, which sounds like the simplest story of the three until the +next section shows what it costs. + +{% include excalidraw.html + file="30-uprobe-attachment" + alt="Three parallel columns, one per language, each four boxes tall. C++ column: nm reports the mangled symbol t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0; c++filt (auto Itanium ABI) demangles it to (anonymous namespace)::cmd_work(int); a uprobe attaches using the MANGLED string as the attach target; the probe fires with seconds_arg=5 for --seconds 5, and arg0==RDI is correct because C++ follows the plain SysV x86-64 ABI. Rust column: nm reports t _RNvCs6FzKXswKZCc_3app8cmd_work; c++filt auto-detects Rust v0 mangling and demangles it to app[hash]::cmd_work; a uprobe attaches using that same mangled string; the probe fires with seconds_arg=7 for --seconds 7, and arg0==RDI is again correct under the SysV ABI. Go column: nm reports T main.cmdWork, never mangled; no demangling step is needed since the package-qualified name is already the symbol; a uprobe attaches to the plain unmangled name main.cmdWork; the probe fires for --seconds 9 but RDI(arg0) reads 1, which is wrong, while RAX correctly reads 9 -- a dashed amber arrow connects this box back up to the tooling band above, labeled arg0 mismatch. A footer band spans all three columns explaining that Go's ABIInternal (1.17+) passes integer arguments in the order RAX, RBX, RCX, RDI, RSI, R8, R9, ... rather than the SysV order RDI, RSI, RDX, ... that bpftrace's arg0/arg1 assume, and that goroutine stacks also grow by copying, so a captured stack address can move out from under a probe that read it from the stack instead of a register." + caption="Figure 30.2 — the same uprobe-attach recipe (find the symbol, demangle it to read it, attach to the mangled string) across three binaries, ending in the one real, measured argument-register mismatch that shows up only on Go" %} + +Attaching directly to the mangled string on the C++ and Rust binaries, with +the workload started immediately after so the one-shot call to `cmd_work` +is still ahead of it: + +```console +[vm]$ sudo timeout 4 bpftrace -e 'uprobe:/tmp/lsp30-cpp/app:_ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 + { printf("hit cmd_work seconds_arg=%d\n", arg0); }' & +[vm]$ sudo /tmp/lsp30-cpp/app work --seconds 5 >/tmp/cpp2.log 2>&1 +hit cmd_work seconds_arg=5 +``` + +```console +[vm]$ sudo timeout 4 bpftrace -e 'uprobe:/tmp/lsp30-rust/app:_RNvCs6FzKXswKZCc_3app8cmd_work + { printf("rust hit cmd_work seconds_arg=%d\n", arg0); }' & +[vm]$ sudo /tmp/lsp30-rust/app work --seconds 7 >/tmp/rust2.log 2>&1 +rust hit cmd_work seconds_arg=7 +``` + +Both print exactly the `--seconds` value passed on the command line — +`arg0` (bpftrace's shorthand for the first System V x86-64 integer +argument register, `RDI`) is correct in both, because both C++ (this +function is not `extern "C"`, but the Itanium C++ ABI still lays out +integer arguments the plain SysV way) and Rust place their first integer +argument in `RDI` in practice on this target. **Go does not.** Since Go +1.17, the Go compiler's own **ABIInternal** passes integer and pointer +arguments in registers in the order `RAX, RBX, RCX, RDI, RSI, R8, R9, R10, +R11` — a different order than the SysV convention every uprobe tool's +`arg0`/`arg1` shorthand assumes. The same attach recipe against Go's +`main.cmdWork(seconds int) int`, reading both the naive `arg0` and the true +first-argument register side by side: + +```console +[vm]$ sudo timeout 4 bpftrace -e 'uprobe:/tmp/lsp30-go/app:main.cmdWork + { printf("go cmdWork rdi(arg0)=%d rax=%d\n", arg0, reg("ax")); }' & +[vm]$ sudo /tmp/lsp30-go/app work --seconds 9 >/tmp/go2.log 2>&1 +go cmdWork rdi(arg0)=1 rax=9 +``` + +`rax=9` is exactly `--seconds 9`; `rdi(arg0)=1` is not — it is whatever +happened to be sitting in `RDI` from a previous call, because Go's compiler +never put the argument there in the first place. This is the concrete +version of "uprobes on Go are trickier": a tool that reads arguments by the +SysV convention's positional shorthand silently returns a *plausible-looking +wrong number* rather than an error, and the only way to get the right +answer is to know Go's own calling convention for that specific function +(which can itself vary by whether the function is inlined, its argument +count, and the Go toolchain version) rather than trust a generic `argN`. +The second half of "trickier" is one this session did not need to +reproduce to state accurately: Go's goroutine stacks are **movable** — the +runtime grows a goroutine's stack by copying it to a larger allocation and +rewriting every pointer *it* tracks into the new location, but a raw stack +address a uprobe captured externally is not one of the pointers the runtime +knows to fix up, so an argument read off the stack (rather than a register) +can point at stale memory after a stack-growth event the tracer had no way +to see coming. + +## Build, run, observe + +```console +[vm]$ export LIBVIRT_DEFAULT_URI=qemu:///system TARGET=systems-target +[host]$ cd examples/30-ebpf-observation-toolkit && ./cpp/demo.sh build +[host]$ scp cpp/build/release/app "fedora@$IP:/tmp/lsp30-cpp/app" +[host]$ scp observe.sh "fedora@$IP:/tmp/lsp30-observe.sh" +[vm]$ sudo bash /tmp/lsp30-observe.sh /tmp/lsp30-cpp/app 60 +``` + +`observe.sh` prints the five-tool tour and a `=== summary ===` block +deriving one hit count per tool, all nonzero, all tied to the workload's own +pid or bait path — never merely "the tool ran and exited 0." The runner +drives the same contract for all three languages against the live VM: + +```console +[host]$ python3 scripts/test-all-examples.py --only 30-ebpf-observation-toolkit --mode vm +30-ebpf-observation-toolkit PASS PASS PASS +3 passed, 0 failed, 0 skipped +``` + +## Cross-check: three independent views of one workload agreeing + +No single tool in this chapter is trusted alone — every tool's count is +checked against either another tool watching the same process, or against +`bpftool` reading back what the kernel actually loaded. From the same +`observe.sh` run quoted above, `execsnoop` (fork/exec) and `bpftrace`'s +uprobe (function calls) are two completely different probe types — a +tracepoint-adjacent syscall tracer versus a userspace-symbol uprobe — +watching the same 40-second window of the same pid, and they agree with +each other and with the program's own printed counters: + +``` +execsnoop_execs=6 <- 6 "true" children (44 execs at 4s granularity, sampled) +bpftrace_calls=22 <- 22 busy_hash() uprobe hits +app_summary: work: done seconds=40 iters=173 opens=173 execs=44 busy_calls=173 busy_hash=1836395105209072940 +``` + +`execsnoop`'s 6 and the program's own `execs=44` differ only because the +tool's own capture window (a `timeout`-bounded slice near the start of the +run) is shorter than the workload's full 40 seconds — exactly what +`observe.sh`'s design note says to expect, and the ratio (6 execs in roughly +6 seconds of `execsnoop` runtime, at one exec every ~1 second given the +4-iteration cadence) lines up. The second, independent cross-check is +`bpftool prog show` naming the exact uprobe program bpftrace loaded while +it was still attached (`uprobe__tmp_lsp30_cpp_app_busy_hash_1`, `type +kprobe`), read from the kernel's own bookkeeping rather than from anything +bpftrace itself reported — two separate programs reading the same kernel +state and agreeing. The third is the mangled-symbol pair: `nm` and +`c++filt` independently identify and translate `cmd_work`'s real linker +name off the binary, and a live uprobe attached to that exact string is +what actually fires and reports the right `--seconds` value back — proof +the demangled name and the attach target are the same symbol, not a +coincidence of two unrelated strings. + +## What you learned + +- eBPF-based observation is a third way to watch a program, beside binary + translation (valgrind) and compile-time instrumentation (sanitizers): the + kernel runs a small verified program on a real kprobe/uprobe/tracepoint/ + perf_event hit, live, with no rebuild and no restart — bcc-tools' + `execsnoop`/`opensnoop`/`funccount`/`offcputime`/`biolatency` and + bpftrace's one-liners and scripts all ran this way against an unmodified + binary on `systems-target`. +- `bpftool prog show`/`map show` is the independent check on what the + kernel actually has loaded — it named bpftrace's own uprobe program by + the exact attach-point string bpftrace compiled into it, confirmed from + the kernel's bookkeeping, not bpftrace's. +- A uprobe attaches to the ELF symbol table's literal string, mangled or + not: `c++filt` auto-detects and demangles both C++'s Itanium scheme and + Rust's v0 scheme for reading, but the attach target itself stays the + mangled string in both languages — and both fire correctly (`arg0` + matching `--seconds`) because both follow the plain SysV x86-64 ABI. +- Go never mangles, but its own register-based ABIInternal (`RAX` first, + not `RDI`) means the same `arg0` shorthand silently reads the *wrong* + register — measured live as `rdi(arg0)=1` (wrong) beside `rax=9` (right, + matching `--seconds 9`) — and goroutine stacks moving under a copying + stack-grow makes any stack-based argument read riskier still. + +Next, **per-language toolbelts**: `perf`, `pprof`, `delve`, and +`cargo-flamegraph` each answer "where does the time go" their own native +way, on the same kind of unmodified binary this chapter just watched from +outside. + +--- + +

verified — every transcript +above (except the SystemTap dyninst run's silence, reported as +inconclusive rather than a positive result) was produced against the +running `systems-target` lab VM (kernel 6.19.10-300.fc44.x86_64, +bpftrace v0.24.2, bcc-tools bundled with that image, bpftool from the same +kernel package, stap 5.5) and the Fedora 44 host (binutils +c++filt/nm) this session. Confirmed live: +observe.sh against the staged C++ binary printed +opensnoop_opens=20, execsnoop_execs=6, +funccount_calls=21, offcputime_hits=2, +bpftrace_calls=22, and +app_summary: work: done seconds=40 iters=173 opens=173 execs=44 +busy_calls=173 busy_hash=1836395105209072940; +biolatency 1 3 produced a real block-I/O latency histogram +against five dd conv=fsync writes; bcc's profile +failed to load with a genuine +no member named 'ns_id' in 'struct ns_common' compile error on +this kernel, while bpftrace's own profile:hz probe attached +cleanly but reported zero on-CPU samples against a workload that sleeps +over 99% of the time; the busy_hash_latency.bt +uprobe/uretprobe script reported a real latency histogram peaking at +[1M, 2M) 19 (nanoseconds); bpftool prog show +named a live kprobe-type program +uprobe__tmp_lsp30_cpp_app_busy_hash_1 while bpftrace held it +attached, and bpftool map show attributed its maps to +pids bpftrace(16154); c++filt demangled +_ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 to +(anonymous namespace)::cmd_work(int) [clone .isra.0] and +auto-detected Rust v0 mangling on +_RNvCs6FzKXswKZCc_3app8cmd_work as +app[4db24ee1e6ed4506]::cmd_work, and live uprobes on both exact +mangled strings fired with seconds_arg=5/7 +matching their real --seconds flags; the Go ABI experiment +reported rdi(arg0)=1 (wrong) beside rax=9 (right) +for --seconds 9 on main.cmdWork; the bad-usage +contract exited 2 identically in all three languages; +bpftrace refused to attach to a nonexistent symbol with +ERROR: Could not resolve symbol, and bcc's +funccount reported No functions matched by pattern +b'^busy_hash$' against the Go binary's real +main.busy_hash; and stap's kernel-module backend +failed with a real kernel-devel version mismatch +(6.19.10-300.fc44 running versus +kernel-devel-7.1.3-201.fc44 installed) while +--runtime=dyninst attached and ran to completion with no +error. The formal three-language runner +(scripts/test-all-examples.py --only +30-ebpf-observation-toolkit --mode vm) confirmed the same +verify.lua contract end to end against this VM this session, +printing 30-ebpf-observation-toolkit PASS PASS PASS (3 +passed, 0 failed, 0 skipped) for cpp/go/rust in turn — each language's own +run independently reproducing opensnoop's bait-path match, execsnoop's +PPID-matched true child, funccount's nonzero +busy_hash count, offcputime's comm-app stack, and +bpftrace's nonzero @calls.

diff --git a/_docs/31-per-language-toolbelts.md b/_docs/31-per-language-toolbelts.md new file mode 100644 index 0000000..4ac6b16 --- /dev/null +++ b/_docs/31-per-language-toolbelts.md @@ -0,0 +1,549 @@ +--- +title: "Per-Language Toolbelts" +order: 31 +part: "Debugging" +description: "app hot and app alloc are the same trial-division prime count and the same string-churn index build in C++, Go, and Rust — and each language's own profiler, run for real against the built binary, points at the same widest frame: perf record/report and clang-tidy/-fanalyzer for C++, the pprof ecosystem and delve for Go, cargo-flamegraph and clippy for Rust." +duration: "55 minutes" +--- + +Chapter 28 opened this part by reading a crash gdb already knew about; +Chapter 29 broke code on purpose and watched a sanitizer catch it. Both +chapters answer "what is wrong". This one answers a different question that +comes up on every program that runs correctly and still feels slow: "where +does the time go, and which line is spending it?" That question has no +generic answer — each language ships its own native way of pointing a +profiler at a running binary — but the *skill* of reading what comes back is +the same skill everywhere: find the widest frame in a call-stack sample and +that is your hot path. `app` is this chapter's target, one program built +three times with one contract: `app [--n N]` runs either an +unoptimized trial-division prime count (`hot`) or a 1,000-key string-churn +index build (`alloc`), and both modes route every bit of real work through +exactly one `noinline` function per mode, so there is never any ambiguity +about which frame should show up wide. + +The code is in `examples/31-per-language-toolbelts/`. `./demo.sh build` +builds all three; each language directory's `demo.sh run [hot|alloc]` runs +it, and the top-level `README.md` documents the full toolbelt transcript +this chapter draws from. + +{% include excalidraw.html + file="31-flamegraph-reading" + alt="A schematic flamegraph with two labeled axes: y is stack depth, bottom the root frame runtime.main and top the leaf running at the instant a sample was taken; x is share of samples, explicitly not time order or call order. A stack of boxes narrows from a full-width root frame up through main/main.run, into a wide accent box for spin_hot/hotSpin covering about 99% of the row's width, into an equally wide inlined divisor-loop frame directly above it, topped by a leaf sliver. A callout box explains that color here just marks which language tab a frame came from, since real profilers usually color by module or at random. A footer note says the way to find the hot path is to scan for the widest unbroken column reaching toward the top." + caption="Figure 31.1 — reading a flamegraph: width is the share of samples, height is stack depth, and the widest column reaching toward the top is the function actually spending the cycles" %} + +> **Tools used** — `perf` record/report/stat, `clang-tidy`, GCC `-fanalyzer` +> (C++); `go tool pprof`, `go tool trace`, `dlv` (Go); `perf` record/report, +> `cargo flamegraph`, `cargo clippy`, `cargo fmt --check` (Rust) — all host. +> `perf`, `gdb`, and the Go/Rust toolchains are on `scripts/check-host.sh`; +> `dlv` and `cargo-flamegraph` are pulled on demand (`go install .../dlv`, +> `cargo install flamegraph`) as this chapter's transcripts show. + +## Why every language needs its own toolbelt + +`perf` is an *external* sampler: it attaches to any ELF binary from the +outside, interrupts it on a hardware or software event (by default a fixed +number of CPU cycles), and records the instruction pointer and call stack at +each interrupt. It has no idea what language produced the binary — it only +needs a symbol table and, for call graphs, either frame pointers or DWARF — +so `perf record`/`perf report`/`perf stat` work identically against the C++ +and Rust binaries in this chapter. That is also exactly why they *don't* +work as cleanly for Go: Go's runtime multiplexes goroutines onto a handful of +OS threads and moves them around, which external samplers can still see but +cannot label with the right goroutine-level context. Go's own answer is to +put the profiler *inside* the binary: `runtime/pprof` and `runtime/trace` are +libraries the program calls into, so writing a CPU profile, a heap profile, +or an execution trace is part of the program's own behavior, driven by flags +this chapter's Go build accepts and C++/Rust do not (`--cpuprofile`, +`--memprofile`, `--trace`). Rust sits in the middle: it is compiled to the +same kind of native ELF binary as C++, so `perf` attaches to it the same way, +but the Rust ecosystem also ships `cargo flamegraph`, a Rust program that +*wraps* `perf record` and then renders the result with `inferno`, a Rust +reimplementation of Brendan Gregg's original Perl `stackcollapse`/`flamegraph` +scripts. Three ecosystems, three different relationships between the +profiler and the profiled program — but the artifact each one hands back is +the same shape: a call stack with a weight attached, which is what a +flamegraph draws directly and what `--top` tabulates as text. + +A flamegraph earns its name from the shape, not from heat: each horizontal +row is one level of stack depth (the root frame along the bottom, the +function actually executing at the top), and the *width* of a box is the +fraction of samples in which that function was somewhere on the stack at +that depth. Frames are usually **sorted alphabetically within their parent**, +not by call order or wall-clock time, which is the detail that trips up +first-time readers expecting the x-axis to mean "when". Reading one is a +single scan: follow the tallest, widest column of boxes from the bottom to +the top, and the name at the top of that column is where the cycles are +actually going. Figure 31.1 draws that shape directly from this chapter's +own data — `spin_hot`/`hotSpin` is that column in every one of the three +profiles below. + +{% include excalidraw.html + file="31-toolbelt-map" + alt="One shared target box, app hot or alloc with an identical result across languages, feeding three columns. The C++ column lists perf record with call-graph fp as its accent entry naming spin_hot at 99.9%, then perf report/perf stat, clang-tidy, GCC -fanalyzer, and a dashed gdb box marked Chapter 28. The Go column lists runtime/pprof as its accent entry, cpuprofile into go tool pprof top naming main.hotSpin, then memprofile with alloc_space naming main.allocChurn, runtime/trace with go tool trace, delve, and a dashed go vet box. The Rust column lists perf record with call-graph fp as its accent entry naming spin_hot inlined near 100%, then cargo flamegraph producing a real SVG, cargo clippy and cargo fmt check, a dashed tokio-console box marked forward reference to Chapter 27 not applicable here since there is no async runtime, and a dashed gdb/rust-gdb box marked Chapter 28. A note explains the amber boxes are the CPU-profiling step this chapter cross-checks against each other, and the dashed boxes are noted but out of scope." + caption="Figure 31.2 — one shared target, three native toolbelts: the amber boxes are what this chapter's cross-check compares; the dashed boxes point to where gdb (Ch. 28) and tokio-console (Ch. 27) belong instead" %} + +## How the code works + +The two workloads are deliberately simple, because the point of this chapter +is the tool, not the algorithm — but the two `noinline` functions they route +through are the whole reason the profiles below are unambiguous, so they get +the tabs, verbatim from `examples/31-per-language-toolbelts/`: + +{% include codetabs.html langs="C++|Go|Rust" %} + +```cpp +// The CPU-bound target: trial division is deliberately unoptimized (checks +// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +// buys a profile with a single, unmistakable hot frame. +extern "C" [[gnu::noinline]] std::uint64_t spin_hot(long n) { + std::uint64_t count = 0; + for (long i = 2; i < n; ++i) { + bool prime = true; + for (long d = 2; d * d <= i; ++d) { + if (i % d == 0) { + prime = false; + break; + } + } + if (prime) { + ++count; + } + } + return count; +} +``` + +```go +// The CPU-bound target: trial division is deliberately unoptimized (checks +// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +// buys a profile with a single, unmistakable hot frame. //go:noinline keeps +// the compiler from folding it into run and hiding it from the sampler. +// +//go:noinline +func hotSpin(n int) uint64 { + var count uint64 + for i := 2; i < n; i++ { + prime := true + for d := 2; d*d <= i; d++ { + if i%d == 0 { + prime = false + break + } + } + if prime { + count++ + } + } + return count +} +``` + +```rust +/// The CPU-bound target: trial division is deliberately unoptimized (checks +/// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +/// buys a profile with a single, unmistakable hot frame. `#[inline(never)]` +/// keeps the compiler from folding it into `run` and hiding it from the +/// sampler. +#[inline(never)] +fn spin_hot(n: u64) -> u64 { + let mut count = 0u64; + for i in 2..n { + let mut prime = true; + let mut d = 2u64; + while d * d <= i { + if i % d == 0 { + prime = false; + break; + } + d += 1; + } + if prime { + count += 1; + } + } + count +} +``` + +All three are the same trial-division loop with the same invariant: an +`i` in `[2, n)` is prime only if no `d` in `[2, sqrt(i)]` divides it evenly, +found by walking `d` up from 2 and stopping the moment `d*d > i`. The loop is +intentionally the slow textbook version — no sieve, no memoized small primes, +no early-exit on evens — because a fast implementation would finish before a +sampling profiler collected enough hits to draw a clean picture, and the goal +here is a profile, not a fast prime counter. The `noinline` annotation on +each (`[[gnu::noinline]]`, `//go:noinline`, `#[inline(never)]`) is not +decoration: at `-O2`/release optimization, all three compilers are entitled +to fold a small, single-call-site function directly into its caller, which +would erase the frame boundary a profiler needs to name. `spin_hot` also has +`extern "C"` linkage in the C++ build specifically so `perf` and `gdb` print +the plain symbol `spin_hot` instead of an Itanium-mangled name — a detail +that matters once you are grepping profiler output for a function name, as +every transcript below does. + +The allocation-heavy sibling (`alloc_churn`/`allocChurn`/`alloc_churn`, not +shown in the tabs above but present in every `main.cpp`/`main.go`/`main.rs`) +does the opposite kind of work on purpose: `n` iterations round-robin +overwrite a 1,000-key string map, and every iteration allocates a fresh key +string and a fresh value string before inserting. Where `spin_hot` never +touches the heap, `alloc_churn` never does meaningful arithmetic — its job is +to be a clean single allocation site for a heap/allocation profiler to find, +the same way `spin_hot` is a clean single CPU sink for a cycle profiler. + +The wiring around both functions is intentionally thin. C++ threads +`std::expected` through argument parsing and the mode +dispatch, wraps the timed section in an RAII `ScopedTimer` whose destructor +writes the elapsed milliseconds regardless of how the scope is left, and +prints with `std::println`. Go's `run(cfg)` additionally opens the profiler +files the CLI's extra flags name — `pprof.StartCPUProfile`/`trace.Start` +before the timed call, `pprof.WriteHeapProfile` after it, each guarded by a +`defer` so a profile still flushes on any return path — and Rust threads +`anyhow::Result` through `parse_args`/`run` and returns `ExitCode` from +`main` rather than calling `std::process::exit`, so destructors still run on +every exit path. None of that wiring changes the algorithm; it only exists +so the same binary can be pointed at by three very different profilers +without recompiling it differently for each one. + +## Errors, three ways + +The CLI contract is the familiar shape from every chapter in this book: a +missing mode, an unknown mode, a non-positive or non-numeric `--n`, and an +unknown flag all print a one-line diagnostic (`app: missing mode`, +`app: unknown mode: bogus`, `app: not a positive integer: zero`, +`app: unknown argument: --bogus`) followed by +`usage: app [--n N]` on stderr and exit `2`. Go's usage banner +appends its three profiling-only flags (`--cpuprofile`, `--memprofile`, +`--trace`); `verify.lua`'s pattern match tolerates that suffix because it +only asserts the shared ` [--n N]` prefix. `--n` itself is parsed +with `std::from_chars` in C++, `strconv.Atoi` in Go, and `str::parse::` +in Rust — three ways of saying "reject anything that is not cleanly a +positive integer" — and every language rejects the same malformed inputs the +same way, which `verify.lua` checks per language before it ever touches a +profiler. + +## Concurrency lens: what each language's tooling gives you + +This is a pure-tooling chapter — `app` is single-threaded in every language, +so there is no race to hunt here the way Chapters 25 and 26 hunted one. What +still differs by language is how much of the *concurrency picture* each +toolbelt can show once a program does have more than one thread of control. +`perf record --call-graph` samples every thread in a process by default, so +it is already concurrency-aware for C++ and Rust: point it at `workq` from +Chapter 25 and the flamegraph would show each worker thread's stack +separately. Go's profilers go further because they are runtime-aware: a +`runtime/pprof` CPU profile attributes samples to the goroutine that was +running, and `go tool trace` is built specifically to show +goroutine-level concurrency — when one parks, when the scheduler hands its P +to another, when the GC stops the world — none of which a generic external +sampler can label without help. Rust's forward-referenced tool, +`tokio-console`, is the async-native version of that same idea: it shows +live per-task poll counts and waker state for an async runtime the way `go +tool trace` shows goroutines, which is exactly why it has nothing to display +against this chapter's synchronous `app` and belongs instead with Chapter +27's `chatterd`. + +## Build, run, observe + +```bash +[host]$ cd examples/31-per-language-toolbelts && ./demo.sh build +``` + +A hand run confirms the shared contract before any profiler gets involved — +`result` is the same number in all three languages, because it is the same +algorithm over the same input: + +```console +[host]$ ./cpp/build/release/app hot +app: mode=hot n=3000000 result=216816 ms=791 +[host]$ ./go/bin/app hot +app: mode=hot n=3000000 result=216816 ms=756 +[host]$ ./rust/target/release/app hot +app: mode=hot n=3000000 result=216816 ms=464 +[host]$ ./cpp/build/release/app alloc +app: mode=alloc n=200000 result=5000 ms=24 +[host]$ ./go/bin/app alloc +app: mode=alloc n=200000 result=5000 ms=37 +[host]$ ./rust/target/release/app alloc +app: mode=alloc n=200000 result=5000 ms=18 +``` + +Now the toolbelts, each run against the real built binary. + +**C++: `perf record`/`perf report`, `clang-tidy`, `-fanalyzer`.** + +```console +[host]$ perf record -o cpp.perf.data --call-graph fp -- ./cpp/build/release/app hot +app: mode=hot n=3000000 result=216816 ms=871 +[ perf record: Captured and wrote 0.214 MB cpp.perf.data (3461 samples) ] +[host]$ perf report -i cpp.perf.data --stdio +# Samples: 3K of event 'cpu/cycles/Pu' +# Event count (approx.): 3306382033 +# +# Children Self Command Shared Object Symbol +# ........ ........ ....... .................... ................ +# + 99.95% 99.95% app app [.] spin_hot + | + ---spin_hot +``` + +`spin_hot` is 99.95% of 3,461 samples — everything else (`ld-linux`'s +startup path) is the loader noise that ran before the timed section even +began. `clang-tidy` and GCC's `-fanalyzer` are the static complements: they +never run the binary at all, so they catch a different class of thing — +patterns that are *suspicious* in the source, not slow at runtime: + +```console +[host]$ clang-tidy -p cpp/build/release cpp/src/main.cpp +main.cpp:143:5: warning: an exception may be thrown in function 'main' which + should not throw exceptions [bugprone-exception-escape] +``` + +One warning, and it is a known, practical false positive: +`bugprone-exception-escape` traces a theoretical path where `std::format`'s +`format_error` could propagate out of `main`, reachable only if the format +string itself were malformed — ours are compile-time literals, so the path +is dead code the checker cannot prove unreachable statically. GCC's +`-fanalyzer` (its own CMake preset, `cmake --build --preset analyze`) is +stricter about interprocedural state and prints nothing at all against this +source — a clean run I reproduced with a fresh `-fanalyzer` build during +this chapter's verification pass. + +**Go: the `pprof` ecosystem, `go tool trace`, `delve`.** + +```console +[host]$ ./go/bin/app hot --cpuprofile cpu.prof +app: mode=hot n=3000000 result=216816 ms=770 +[host]$ go tool pprof --top --nodecount=5 ./go/bin/app cpu.prof + flat flat% sum% cum cum% + 760ms 98.70% 98.70% 770ms 100% main.hotSpin + 10ms 1.30% 100% 10ms 1.30% runtime.asyncPreempt + 0 0% 100% 770ms 100% main.main + 0 0% 100% 770ms 100% main.run + 0 0% 100% 770ms 100% runtime.main +``` + +`main.hotSpin` owns 98.70% of samples on its own line (`flat`) and 100% once +you count everything under it on the stack (`cum`) — `runtime.asyncPreempt` +is the Go scheduler's cooperative preemption check firing inside the tight +loop, not separate work. The heap profile needs one extra flag to be useful: + +```console +[host]$ ./go/bin/app alloc --memprofile mem.prof +app: mode=alloc n=200000 result=5000 ms=33 +[host]$ go tool pprof --top --cum --alloc_space --nodecount=6 ./go/bin/app mem.prof + flat flat% sum% cum cum% + 3604.31kB 53.98% 53.98% 6164.35kB 92.32% main.allocChurn + 0 0% 53.98% 6164.35kB 92.32% main.main + 0 0% 53.98% 6164.35kB 92.32% main.run + 0 0% 53.98% 6164.35kB 92.32% runtime.main + 2560.04kB 38.34% 92.32% 2560.04kB 38.34% fmt.Sprintf +``` + +By the time `pprof.WriteHeapProfile` runs, the churned map from the +just-returned `allocChurn` call is already garbage — the default *in-use* +view would show almost nothing live. `--alloc_space` switches to pprof's +cumulative-since-start counter, which remembers every byte `mallocgc` ever +handed out whether or not it was later freed, which is the view that +actually answers "where did the allocations happen": `main.allocChurn` +itself directly owns 53.98% (the map's own bucket storage), and +`fmt.Sprintf`, called twice per iteration to build the key and value +strings, owns another 38.34% underneath it. `go tool trace` opens an +interactive browser UI by default, but `-d=parsed` dumps the same event +stream as text, enough to confirm the tracer really captured `hotSpin` +executing: + +```console +[host]$ ./go/bin/app hot --n 800000 --trace trace.out +app: mode=hot n=800000 result=63951 ms=114 +[host]$ go tool trace -d=parsed trace.out | grep -B2 hotSpin | head -3 + runtime.asyncPreempt @ 0x484d8a + /usr/lib/golang/src/runtime/preempt_amd64.s:124 + main.hotSpin @ 0x4d272c +``` + +Delve is the interactive debugger this chapter ties back to Chapter 28's +`gdb` work: same idea (breakpoints, stack frames, live variables), native to +Go's runtime instead of DWARF-only. It was not preinstalled on this host — +`go install github.com/go-delve/delve/cmd/dlv@latest` pulled it from the Go +module proxy in a few seconds — and it scripts the same way `gdb --batch` +does, by piping a command list instead of typing at a TUI: + +```console +[host]$ echo -e "break main.hotSpin\ncontinue\nprint n\nbt\ncontinue\nquit" \ + | dlv --allow-non-terminal-interactive=true exec ./go/bin/app -- hot --n 500000 +(dlv) Breakpoint 1 set at 0x4d26e1 for main.hotSpin() ./go/main.go:100 +(dlv) > [Breakpoint 1] main.hotSpin() ./go/main.go:100 (hits goroutine(1):1 total:1) +(dlv) 500000 +(dlv) 0 0x00000000004d26e1 in main.hotSpin + at ./go/main.go:100 +1 0x00000000004d2f95 in main.run + at ./go/main.go:175 +``` + +The breakpoint lands exactly on `hotSpin`'s declaration, `print n` confirms +the argument delve stopped with (`500000`, matching `--n 500000`), and `bt` +names the same two frames `go tool pprof` and `go tool trace` already named +— three different Go tools, one function. + +**Rust: `perf record`/`perf report`, `cargo flamegraph`, `clippy`.** + +Rust compiles to a native ELF binary, so the same external `perf` from the +C++ section attaches to it unchanged: + +```console +[host]$ perf record -o rust.perf.data --call-graph fp -- ./rust/target/release/app hot +app: mode=hot n=3000000 result=216816 ms=476 +[ perf record: Captured and wrote 0.132 MB rust.perf.data (1895 samples) ] +[host]$ perf report -i rust.perf.data --stdio +# Samples: 1K of event 'cpu/cycles/Pu' +# Event count (approx.): 1946413493 + 99.98% 99.98% app app [.] app::spin_hot + | + ---0xcccccccccccccccc + spin_hot (inlined) +``` + +99.98% again, spelled `app::spin_hot` (Rust's own path-qualified symbol, +needing no `extern "C"` trick to stay readable) and reported as inlined at +release optimization — the frame boundary survives as debug-info metadata +even though the call itself was folded away. `cargo-flamegraph` was already +on this host (`cargo install flamegraph --locked` is what would put it there +otherwise), so I ran it for real rather than falling back to the tabular +report: + +```console +[host]$ cargo flamegraph --output rust-hot.svg -- hot +app: mode=hot n=3000000 result=216816 ms=476 +[ perf record: Woken up 118 times to write data ] +[ perf record: Captured and wrote 29.409 MB perf.data (480 samples) ] +writing flamegraph to "rust-hot.svg" +[host]$ grep -o 'title>[^<]*spin_hot[^<]*' rust-hot.svg +title>spin_hot (744,743,865 samples, 38.41%) +title>spin_hot (892,224,516 samples, 46.02%) +title>spin_hot (892,224,516 samples, 46.02%) +title>spin_hot (301,908,555 samples, 15.57%) +``` + +`inferno`, the Rust crate `cargo flamegraph` wraps, is a reimplementation of +Brendan Gregg's classic Perl `stackcollapse-perf.pl`/`flamegraph.pl` +pipeline; the SVG it wrote splits `spin_hot` across three adjacent frames +(different instruction addresses inside the same inlined loop get +symbolized to the same function name but folded as separate stack shapes) — +summed, `38.41 + 46.02 + 15.57 = 99.998%`, the same number `perf report` +gave directly, just rendered as boxes on a page instead of tabulated as +text. Where crates.io is unreachable, the `perf record`/`perf report --stdio` +pair above is the fallback: same data, no picture. + +`cargo clippy --release --all-targets` and `cargo fmt --check` are both +clean on this source — no findings, no diff. `tokio-console` is worth naming +and then setting aside: it attaches to a running **async runtime** and shows +live task/waker state, which only exists once a program has `tokio` tasks to +show. `app` has none — no `.await` anywhere in this chapter's source — so +`tokio-console` genuinely has nothing to display here; it belongs with +Chapter 27's async runtimes, where `chatterd`'s tokio tasks are the thing +worth watching live. + +> **CLion's built-in profiler** unverified — +> CLion wraps this same `perf record`/`perf report` pipeline (and, on +> supported platforms, its own sampler) behind a GUI flamegraph and a +> call-tree view, so a reader working in that IDE gets Figure 31.1's shape +> without leaving the editor. This chapter's transcripts are all terminal +> tools run directly, so the IDE integration itself was not exercised this +> session — confirm it against your own CLion install if that is your +> workflow. + +## Cross-check: three profilers, one function name + +The claim worth checking independently is not any single number — sample +counts vary run to run with host load — but that **three unrelated +profiling tools, built by three different projects, agree on which function +is hot**. `perf stat` gives the coarsest independent signal, a hardware +counter read that needs no symbolication at all: + +```console +[host]$ perf stat -e cycles,instructions -- ./cpp/build/release/app hot +app: mode=hot n=3000000 result=216816 ms=867 + + Performance counter stats for './cpp/build/release/app hot': + + 3,218,019,355 cycles:u + 3,185,629,752 instructions:u + + 0.870321798 seconds time elapsed +``` + +Roughly one instruction retired per cycle — consistent with a tight, +branch-heavy integer loop, not a program stalled on cache misses or memory. +Then the three per-language profilers, each pointed at the same `hot` +workload, each naming its own hot frame in its own vocabulary: + +| Profiler | Command | Named hot frame | Share | +|---|---|---|---| +| C++ `perf report --stdio` | `perf record --call-graph fp -- app hot` | `spin_hot` | 99.95% | +| Go `go tool pprof --top` | `app hot --cpuprofile cpu.prof` | `main.hotSpin` | 98.70% flat / 100% cum | +| Rust `perf report --stdio` | `perf record --call-graph fp -- app hot` | `app::spin_hot` | 99.98% | + +Three profilers with nothing in common under the hood — a kernel-level +hardware sampler reading raw ELF symbols, an in-process Go library sampling +its own goroutine stacks, the same kernel sampler again reading Rust's DWARF +— converge on the same name in every language's own spelling of it, at the +same overwhelming share of samples. That agreement is the cross-check: if +`spin_hot`/`hotSpin` were *not* actually where the time went, three +independently-implemented tools would have no reason to all point at the +same place. + +## What you learned + +- Reading a flamegraph is one scan regardless of language: **width is share + of samples, height is stack depth**, and the widest column reaching toward + the top names the hot path — `spin_hot`/`hotSpin` at 98.7–99.98% across + three unrelated profilers in this chapter's own run. +- `perf` is external and language-agnostic (it attaches to any ELF binary); + Go's `runtime/pprof`/`runtime/trace` are internal libraries the program + calls into, which is why Go's CLI carries `--cpuprofile`/`--memprofile`/ + `--trace` flags that C++ and Rust do not need. +- Static analyzers (`clang-tidy`, `-fanalyzer`, `cargo clippy`) answer a + different question than a profiler — "is this pattern suspicious" instead + of "is this slow" — and both classes of tool are worth running even on + code with no known defect. +- `--alloc_space` (not the default in-use view) is what makes a Go heap + profile useful once the allocating call has already returned and its + garbage is gone; `cargo flamegraph` gets the same collapse+render pipeline + Brendan Gregg's original Perl scripts do, reimplemented in Rust as + `inferno`. + +Next, the chapters ahead move from watching one program's own binary to +watching a whole running system from the outside, with the eBPF-based +tracing tools this book has deferred until now. + +--- + +

verified — every command, +number, and transcript above was run this session on the Fedora 44 reference +host (kernel 7.1.3-200.fc44, GCC 16.1.1, clang 22.1.8, go1.26.5, rustc +1.97.1, perf 7.1.3, gdb 17.2) against the binaries in +examples/31-per-language-toolbelts/. The plain hand runs +printed result=216816 (hot) and result=5000 (alloc) +identically in all three languages; perf record/perf +report --stdio named spin_hot at 99.95% (C++, 3,461 +samples) and app::spin_hot at 99.98% (Rust, 1,895 samples); +perf stat reported 3,218,019,355 cycles and +3,185,629,752 instructions on app hot; +go tool pprof --top named main.hotSpin at 98.70% +flat/100% cum on a real --cpuprofile capture and +main.allocChurn at 53.98% flat/92.32% cum ( +--alloc_space) on a real --memprofile capture; +go tool trace -d=parsed showed main.hotSpin in its +parsed event stream; clang-tidy reported exactly the one +bugprone-exception-escape warning at main.cpp:143 +and a fresh -fanalyzer build (cmake --build --preset +analyze) produced zero diagnostics; dlv was installed +live with go install .../dlv@latest and its scripted session +broke at main.hotSpin, printed n=500000, and +backtraced through main.run; cargo flamegraph was +already present, ran for real, and its output SVG contained +spin_hot frames summing to 99.998% of samples; +cargo clippy --release --all-targets and cargo fmt +--check were both clean. The CLion profiler-integration callout is +unverified as marked — no CLion session was run this session.

diff --git a/assets/diagrams/28-remote-debug-topology.excalidraw b/assets/diagrams/28-remote-debug-topology.excalidraw new file mode 100644 index 0000000..d5bd4d2 --- /dev/null +++ b/assets/diagrams/28-remote-debug-topology.excalidraw @@ -0,0 +1,1424 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r326373800", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 1000, + "height": 230, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 309669599, + "versionNonce": 1785924820, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t887807120", + "type": "text", + "x": 32, + "y": 26, + "width": 198.00000000000003, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 745735389, + "versionNonce": 1546624571, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "host (this session \u2014 verified)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "host (this session \u2014 verified)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1912516448", + "type": "rectangle", + "x": 20, + "y": 280, + "width": 1000, + "height": 230, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1618886611, + "versionNonce": 886121251, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1620993483", + "type": "text", + "x": 32, + "y": 286, + "width": 343.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 315975688, + "versionNonce": 1850543581, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-target VM \u2014 documented, not run this session", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-target VM \u2014 documented, not run this session", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1818191008", + "type": "rectangle", + "x": 50, + "y": 60, + "width": 210, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 187042612, + "versionNonce": 474022816, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t23964814", + "type": "text", + "x": 60, + "y": 68, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 743212446, + "versionNonce": 484811311, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "app crash", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "app crash", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1748403483", + "type": "text", + "x": 60, + "y": 88, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 300447834, + "versionNonce": 1299073187, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SIGSEGV, exit 139", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SIGSEGV, exit 139", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1110522314", + "type": "rectangle", + "x": 50, + "y": 140, + "width": 210, + "height": 70, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 760202443, + "versionNonce": 459498445, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t929915165", + "type": "text", + "x": 60, + "y": 148, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1658118299, + "versionNonce": 789797396, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systemd-coredump", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systemd-coredump", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t682424260", + "type": "text", + "x": 60, + "y": 168, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1399953088, + "versionNonce": 986199210, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core_pattern='|...systemd-coredump'", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core_pattern='|...systemd-coredump'", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t553211812", + "type": "text", + "x": 60, + "y": 183, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 180537805, + "versionNonce": 220364516, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "coredumpctl dump -> core.app", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "coredumpctl dump -> core.app", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1120529574", + "type": "rectangle", + "x": 300, + "y": 100, + "width": 210, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 959877560, + "versionNonce": 1762322208, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1316385509", + "type": "text", + "x": 310, + "y": 108, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1910166918, + "versionNonce": 708163266, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdb app core.app", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdb app core.app", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t111584548", + "type": "text", + "x": 310, + "y": 128, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 159930814, + "versionNonce": 1692295633, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bt full names every frame", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bt full names every frame", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r423925736", + "type": "rectangle", + "x": 590, + "y": 60, + "width": 220, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1421457648, + "versionNonce": 1244916441, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t602466662", + "type": "text", + "x": 600, + "y": 68, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 548586219, + "versionNonce": 1069102505, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdb (client)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdb (client)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1253389110", + "type": "text", + "x": 600, + "y": 88, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 144251873, + "versionNonce": 1439135196, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "target remote", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "target remote", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t589198926", + "type": "text", + "x": 600, + "y": 103, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 376460299, + "versionNonce": 1396401584, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "systems-target:2345", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "systems-target:2345", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1671080167", + "type": "rectangle", + "x": 830, + "y": 60, + "width": 170, + "height": 90, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 583337070, + "versionNonce": 228733307, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t4406568", + "type": "text", + "x": 840, + "y": 68, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 71858233, + "versionNonce": 1014824362, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "CLion", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "CLion", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t540610133", + "type": "text", + "x": 840, + "y": 88, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1372172454, + "versionNonce": 1929803965, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Remote GDB Server", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Remote GDB Server", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t207975198", + "type": "text", + "x": 840, + "y": 103, + "width": 60.50000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 36430782, + "versionNonce": 793641679, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "run config", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "run config", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1952357489", + "type": "text", + "x": 840, + "y": 118, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 673742019, + "versionNonce": 1699171952, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(unverified)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(unverified)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1903331748", + "type": "rectangle", + "x": 590, + "y": 320, + "width": 220, + "height": 70, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1621982626, + "versionNonce": 136132331, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t156930015", + "type": "text", + "x": 600, + "y": 328, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1798927090, + "versionNonce": 1954048232, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdbserver :2345", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdbserver :2345", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1516623530", + "type": "text", + "x": 600, + "y": 348, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1987413787, + "versionNonce": 1506376469, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "--once app crash", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "--once app crash", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1390529694", + "type": "rectangle", + "x": 590, + "y": 420, + "width": 220, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1024263507, + "versionNonce": 291251575, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t123849642", + "type": "text", + "x": 600, + "y": 428, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1497234519, + "versionNonce": 1932520708, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "app (crash target)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "app (crash target)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1637633489", + "type": "text", + "x": 600, + "y": 448, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 313180586, + "versionNonce": 1833529600, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "child process, PTRACE'd", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "child process, PTRACE'd", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a853457864", + "type": "arrow", + "x": 155, + "y": 120, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1600927829, + "versionNonce": 424870864, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1662512311", + "type": "arrow", + "x": 260, + "y": 165, + "width": 40, + "height": 25, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1215787635, + "versionNonce": 689439404, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 40, + -25 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t382430110", + "type": "text", + "x": 280.0, + "y": 138.5, + "width": 48.400000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1927495124, + "versionNonce": 741249102, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "core.app", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "core.app", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1862521267", + "type": "arrow", + "x": 700, + "y": 130, + "width": 0, + "height": 190, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1202970521, + "versionNonce": 1373790944, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 190 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t830108902", + "type": "text", + "x": 700.0, + "y": 211.0, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 555009023, + "versionNonce": 1930298700, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdb remote serial protocol, TCP:2345", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdb remote serial protocol, TCP:2345", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1427555367", + "type": "arrow", + "x": 830, + "y": 100, + "width": 20, + "height": 5, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1239506157, + "versionNonce": 205353920, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -20, + -5 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a431701102", + "type": "arrow", + "x": 700, + "y": 390, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1598274450, + "versionNonce": 1457821927, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1038007051", + "type": "text", + "x": 50, + "y": 225, + "width": 538.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1681056868, + "versionNonce": 899193709, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1569622889", + "type": "text", + "x": 50, + "y": 240, + "width": 459.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 655747158, + "versionNonce": 770186368, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "catches it, coredumpctl extracts core.app, gdb opens the core file directly.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "catches it, coredumpctl extracts core.app, gdb opens the core file directly.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1289868818", + "type": "text", + "x": 590, + "y": 225, + "width": 514.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1289099483, + "versionNonce": 417412572, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t710451804", + "type": "text", + "x": 590, + "y": 240, + "width": 490.05, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1428685785, + "versionNonce": 1308971993, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "TCP with 'target remote'. No systems-target VM was up this session; not run here.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "TCP with 'target remote'. No systems-target VM was up this session; not run here.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1886562230", + "type": "text", + "x": 590, + "y": 490, + "width": 375.37500000000006, + "height": 13.125, + "angle": 0, + "strokeColor": "#666666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 843711235, + "versionNonce": 1291512914, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdbserver is preinstalled on the lab VM image (not on this host).", + "fontSize": 10.5, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdbserver is preinstalled on the lab VM image (not on this host).", + "lineHeight": 1.25, + "baseline": 10.5 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/28-remote-debug-topology.svg b/assets/diagrams/28-remote-debug-topology.svg new file mode 100644 index 0000000..921fb2d --- /dev/null +++ b/assets/diagrams/28-remote-debug-topology.svg @@ -0,0 +1,46 @@ + + + +host (this session — verified) + +systems-target VM — documented, not run this session + +app crash +SIGSEGV, exit 139 + +systemd-coredump +core_pattern='|...systemd-coredump' +coredumpctl dump -> core.app + +gdb app core.app +bt full names every frame + +gdb (client) +target remote +systems-target:2345 + +CLion +Remote GDB Server +run config +(unverified) + +gdbserver :2345 +--once app crash + +app (crash target) +child process, PTRACE'd + + +core.app +core.app + +gdb remote serial protocol, TCP:2345 +gdb remote serial protocol, TCP:2345 + + +Left: the local path this chapter's evidence is built on -- app crashes, systemd-coredump +catches it, coredumpctl extracts core.app, gdb opens the core file directly. +Right: the remote path -- gdbserver attaches on the VM, the host's gdb drives it over +TCP with 'target remote'. No systems-target VM was up this session; not run here. +gdbserver is preinstalled on the lab VM image (not on this host). + \ No newline at end of file diff --git a/assets/diagrams/28-stack-frame-anatomy.excalidraw b/assets/diagrams/28-stack-frame-anatomy.excalidraw new file mode 100644 index 0000000..c58ec6e --- /dev/null +++ b/assets/diagrams/28-stack-frame-anatomy.excalidraw @@ -0,0 +1,1468 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1641464857", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 660, + "height": 560, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 240616298, + "versionNonce": 66402568, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1110119193", + "type": "text", + "x": 32, + "y": 26, + "width": 402.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1114219789, + "versionNonce": 830076611, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "call stack at the moment of SIGSEGV (gdb bt full on core.app)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "call stack at the moment of SIGSEGV (gdb bt full on core.app)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1102604333", + "type": "rectangle", + "x": 700, + "y": 20, + "width": 320, + "height": 560, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1123446777, + "versionNonce": 113706131, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1237744847", + "type": "text", + "x": 712, + "y": 26, + "width": 442.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 712777891, + "versionNonce": 1393692427, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "DWARF .debug_line cross-check\n(readelf/objdump --dwarf=decodedline)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "DWARF .debug_line cross-check\n(readelf/objdump --dwarf=decodedline)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1736069188", + "type": "rectangle", + "x": 60, + "y": 60, + "width": 580, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 985109138, + "versionNonce": 1202869554, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1339820313", + "type": "text", + "x": 70, + "y": 68, + "width": 284.90000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 642287268, + "versionNonce": 1105028127, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#0 std::println (crash PC)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#0 std::println (crash PC)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1230392254", + "type": "text", + "x": 70, + "y": 88, + "width": 363.00000000000006, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1526422811, + "versionNonce": 1833369702, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "/usr/include/c++/16/print:164 -- reads *child.exit_slot here", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "/usr/include/c++/16/print:164 -- reads *child.exit_slot here", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r228219481", + "type": "rectangle", + "x": 60, + "y": 150, + "width": 580, + "height": 70, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 553247539, + "versionNonce": 1476470481, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1445244924", + "type": "text", + "x": 70, + "y": 158, + "width": 300.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1177624501, + "versionNonce": 1020810364, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#1 (anonymous namespace)::deref(child)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#1 (anonymous namespace)::deref(child)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1462615252", + "type": "text", + "x": 70, + "y": 178, + "width": 278.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 947183247, + "versionNonce": 1148458295, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:52 child = {pid=4242, exit_slot=0x0}", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:52 child = {pid=4242, exit_slot=0x0}", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1042903518", + "type": "rectangle", + "x": 60, + "y": 240, + "width": 580, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1806430453, + "versionNonce": 1731095235, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t572805765", + "type": "text", + "x": 70, + "y": 248, + "width": 354.20000000000005, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1042865112, + "versionNonce": 1090469323, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#2 (anonymous namespace)::handle_child(child)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#2 (anonymous namespace)::handle_child(child)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1372629617", + "type": "text", + "x": 70, + "y": 268, + "width": 260.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 997700157, + "versionNonce": 1050925264, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:56 return addr 0x00000000004016b2", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:56 return addr 0x00000000004016b2", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r89734019", + "type": "rectangle", + "x": 60, + "y": 330, + "width": 580, + "height": 80, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1931476791, + "versionNonce": 1347973870, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t564891827", + "type": "text", + "x": 70, + "y": 338, + "width": 446.6, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 667929432, + "versionNonce": 1088513741, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#3 (anonymous namespace)::run_supervisor(inject_bug=true)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#3 (anonymous namespace)::run_supervisor(inject_bug=true)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1558961529", + "type": "text", + "x": 70, + "y": 358, + "width": 260.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1185262648, + "versionNonce": 1938703082, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:65 return addr 0x000000000040177c", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:65 return addr 0x000000000040177c", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t611404502", + "type": "text", + "x": 70, + "y": 373, + "width": 326.70000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1086314140, + "versionNonce": 1826159886, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "child={pid=4242, exit_slot=0x0}, guard=", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "child={pid=4242, exit_slot=0x0}, guard=", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1307436166", + "type": "rectangle", + "x": 60, + "y": 440, + "width": 580, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 508088969, + "versionNonce": 1612076236, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t212327065", + "type": "text", + "x": 70, + "y": 448, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1591463561, + "versionNonce": 1066521082, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#4 main(argc, argv)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#4 main(argc, argv)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t156374866", + "type": "text", + "x": 70, + "y": 468, + "width": 266.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1929658036, + "versionNonce": 1056222971, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:104 return addr 0x0000000000400964", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:104 return addr 0x0000000000400964", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1722781656", + "type": "rectangle", + "x": 730, + "y": 150, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1850538677, + "versionNonce": 958332864, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1854820507", + "type": "text", + "x": 740, + "y": 158, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1120432001, + "versionNonce": 1207331969, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:52 -> 0x401680", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:52 -> 0x401680", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1502336016", + "type": "text", + "x": 740, + "y": 178, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1968800901, + "versionNonce": 542319685, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "exact match: gdb's own", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "exact match: gdb's own", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t338525129", + "type": "text", + "x": 740, + "y": 193, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1042763347, + "versionNonce": 1472403581, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "breakpoint address", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "breakpoint address", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r139608967", + "type": "rectangle", + "x": 730, + "y": 240, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1901883208, + "versionNonce": 1543253628, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1293306776", + "type": "text", + "x": 740, + "y": 248, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1141631417, + "versionNonce": 323895811, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:56 -> 0x4016b2", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:56 -> 0x4016b2", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1715959067", + "type": "text", + "x": 740, + "y": 268, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 980600380, + "versionNonce": 1531986570, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "exact match: #2's", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "exact match: #2's", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t951414320", + "type": "text", + "x": 740, + "y": 283, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 155773768, + "versionNonce": 190895639, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "return address", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "return address", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1083341568", + "type": "rectangle", + "x": 730, + "y": 350, + "width": 260, + "height": 60, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 652655206, + "versionNonce": 1483357540, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t891375213", + "type": "text", + "x": 740, + "y": 358, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 345516595, + "versionNonce": 1113158309, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.cpp:65 -> 0x401772", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.cpp:65 -> 0x401772", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t352356057", + "type": "text", + "x": 740, + "y": 378, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 415149635, + "versionNonce": 486806038, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "close, not exact:", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "close, not exact:", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1242501787", + "type": "text", + "x": 740, + "y": 393, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1162806487, + "versionNonce": 1803530634, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "#3 is a return addr", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "#3 is a return addr", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1500388273", + "type": "arrow", + "x": 350, + "y": 130, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 545555730, + "versionNonce": 51593832, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1184377010", + "type": "arrow", + "x": 350, + "y": 220, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 570035317, + "versionNonce": 980128325, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a652756066", + "type": "arrow", + "x": 350, + "y": 310, + "width": 0, + "height": 20, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1020273569, + "versionNonce": 1805892539, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 20 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a93567720", + "type": "arrow", + "x": 350, + "y": 410, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 363820446, + "versionNonce": 1699047447, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a129631433", + "type": "arrow", + "x": 640, + "y": 185, + "width": 90, + "height": 5, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 738366073, + "versionNonce": 825172863, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 90, + -5 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1553568448", + "type": "arrow", + "x": 640, + "y": 275, + "width": 90, + "height": 5, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1068045478, + "versionNonce": 718646352, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 90, + -5 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1962205663", + "type": "arrow", + "x": 640, + "y": 370, + "width": 90, + "height": 10, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1884769422, + "versionNonce": 1812684068, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 90, + 10 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1410581754", + "type": "text", + "x": 60, + "y": 20, + "width": 392.70000000000005, + "height": 13.125, + "angle": 0, + "strokeColor": "#666666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 514287622, + "versionNonce": 1983328153, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stack grows toward lower addresses (down this column = older frames)", + "fontSize": 10.5, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stack grows toward lower addresses (down this column = older frames)", + "lineHeight": 1.25, + "baseline": 10.5 + }, + { + "id": "t1114504778", + "type": "text", + "x": 60, + "y": 525, + "width": 490.05, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 366322948, + "versionNonce": 196798374, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Every frame function is [[gnu::noinline]] so -O2 cannot fold the call chain away.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Every frame function is [[gnu::noinline]] so -O2 cannot fold the call chain away.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1147181084", + "type": "text", + "x": 60, + "y": 545, + "width": 508.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1385892006, + "versionNonce": 729949112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ShutdownGuard's destructor (main.cpp) never runs: SIGSEGV does not unwind the stack.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ShutdownGuard's destructor (main.cpp) never runs: SIGSEGV does not unwind the stack.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/28-stack-frame-anatomy.svg b/assets/diagrams/28-stack-frame-anatomy.svg new file mode 100644 index 0000000..fa00b87 --- /dev/null +++ b/assets/diagrams/28-stack-frame-anatomy.svg @@ -0,0 +1,46 @@ + + + +call stack at the moment of SIGSEGV (gdb bt full on core.app) + +DWARF .debug_line cross-check +(readelf/objdump --dwarf=decodedline) + +#0 std::println<int,int> (crash PC) +/usr/include/c++/16/print:164 -- reads *child.exit_slot here + +#1 (anonymous namespace)::deref(child) +main.cpp:52 child = {pid=4242, exit_slot=0x0} + +#2 (anonymous namespace)::handle_child(child) +main.cpp:56 return addr 0x00000000004016b2 + +#3 (anonymous namespace)::run_supervisor(inject_bug=true) +main.cpp:65 return addr 0x000000000040177c +child={pid=4242, exit_slot=0x0}, guard=<optimized out> + +#4 main(argc, argv) +main.cpp:104 return addr 0x0000000000400964 + +main.cpp:52 -> 0x401680 +exact match: gdb's own +breakpoint address + +main.cpp:56 -> 0x4016b2 +exact match: #2's +return address + +main.cpp:65 -> 0x401772 +close, not exact: +#3 is a return addr + + + + + + + +stack grows toward lower addresses (down this column = older frames) +Every frame function is [[gnu::noinline]] so -O2 cannot fold the call chain away. +ShutdownGuard's destructor (main.cpp) never runs: SIGSEGV does not unwind the stack. + \ No newline at end of file diff --git a/assets/diagrams/29-shadow-memory.excalidraw b/assets/diagrams/29-shadow-memory.excalidraw new file mode 100644 index 0000000..67f8689 --- /dev/null +++ b/assets/diagrams/29-shadow-memory.excalidraw @@ -0,0 +1,1233 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r544058912", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 980, + "height": 190, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1871281396, + "versionNonce": 785825928, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1940008201", + "type": "text", + "x": 32, + "y": 26, + "width": 468.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 83362951, + "versionNonce": 1281132081, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Application memory -- real addresses from this chapter's ASan `uaf` run", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Application memory -- real addresses from this chapter's ASan `uaf` run", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1787612733", + "type": "rectangle", + "x": 20, + "y": 240, + "width": 980, + "height": 190, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 339571128, + "versionNonce": 855023781, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1849839623", + "type": "text", + "x": 32, + "y": 246, + "width": 468.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1263769162, + "versionNonce": 1803186700, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Shadow memory -- 1 shadow byte encodes the state of 8 application bytes", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Shadow memory -- 1 shadow byte encodes the state of 8 application bytes", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1034719379", + "type": "rectangle", + "x": 60, + "y": 70, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 560712151, + "versionNonce": 1798916655, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t887706797", + "type": "text", + "x": 70, + "y": 78, + "width": 92.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1815178148, + "versionNonce": 1434507800, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "left redzone", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "left redzone", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1781741959", + "type": "text", + "x": 70, + "y": 98, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 130709127, + "versionNonce": 1797834319, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0x7b29be3dff80..ffff", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0x7b29be3dff80..ffff", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1271515812", + "type": "text", + "x": 70, + "y": 113, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1505281724, + "versionNonce": 381058872, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "never allocated to this block", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "never allocated to this block", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r863622960", + "type": "rectangle", + "x": 340, + "y": 70, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1676758434, + "versionNonce": 1957394727, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t326650522", + "type": "text", + "x": 350, + "y": 78, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 972708742, + "versionNonce": 1167300096, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "int* p (4 bytes)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "int* p (4 bytes)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t774800766", + "type": "text", + "x": 350, + "y": 98, + "width": 84.7, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 914048028, + "versionNonce": 252642646, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0x7b29be3e0010", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0x7b29be3e0010", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1670383057", + "type": "text", + "x": 350, + "y": 113, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1797772719, + "versionNonce": 535009559, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "new'd, then delete'd -- *p=7 UB here", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "new'd, then delete'd -- *p=7 UB here", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r906084337", + "type": "rectangle", + "x": 620, + "y": 70, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1564830594, + "versionNonce": 1945151336, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t412495369", + "type": "text", + "x": 630, + "y": 78, + "width": 100.10000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 637609424, + "versionNonce": 178374008, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "right redzone", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "right redzone", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1701298894", + "type": "text", + "x": 630, + "y": 98, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 439537800, + "versionNonce": 309248636, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0x7b29be3e0018..007f", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0x7b29be3e0018..007f", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t852294374", + "type": "text", + "x": 630, + "y": 113, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1014946264, + "versionNonce": 607320997, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "padding around the block", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "padding around the block", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1135404874", + "type": "rectangle", + "x": 60, + "y": 290, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1269019354, + "versionNonce": 1155120286, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t295483215", + "type": "text", + "x": 70, + "y": 298, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1621716773, + "versionNonce": 455027313, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shadow byte: fa", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shadow byte: fa", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t124607293", + "type": "text", + "x": 70, + "y": 318, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1522289218, + "versionNonce": 369216206, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"Heap left redzone\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"Heap left redzone\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1359222515", + "type": "text", + "x": 70, + "y": 333, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1674941794, + "versionNonce": 608311177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "covers 0x...ff80-ff87 .. -0007", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "covers 0x...ff80-ff87 .. -0007", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1599563954", + "type": "rectangle", + "x": 340, + "y": 290, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1015001772, + "versionNonce": 1059383588, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1562942564", + "type": "text", + "x": 350, + "y": 298, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 755147974, + "versionNonce": 1328004961, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shadow byte: fd", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shadow byte: fd", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t70775384", + "type": "text", + "x": 350, + "y": 318, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1760174700, + "versionNonce": 98418377, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"Freed heap region\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"Freed heap region\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1601192794", + "type": "text", + "x": 350, + "y": 333, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1210937230, + "versionNonce": 714969249, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the exact byte the write to *p trips", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the exact byte the write to *p trips", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1407906064", + "type": "rectangle", + "x": 620, + "y": 290, + "width": 260, + "height": 70, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 198006594, + "versionNonce": 451528682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t359061673", + "type": "text", + "x": 630, + "y": 298, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1374110478, + "versionNonce": 1841904609, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shadow byte: fa", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shadow byte: fa", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t511477672", + "type": "text", + "x": 630, + "y": 318, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 78135707, + "versionNonce": 1635181083, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"Heap left redzone\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"Heap left redzone\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t809776027", + "type": "text", + "x": 630, + "y": 333, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1343613319, + "versionNonce": 381865967, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(next block's guard)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(next block's guard)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r629133836", + "type": "rectangle", + "x": 60, + "y": 370, + "width": 820, + "height": 40, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1941661699, + "versionNonce": 810059354, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t628373501", + "type": "text", + "x": 70, + "y": 378, + "width": 831.6, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1364559590, + "versionNonce": 70047168, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "a1383587477", + "type": "arrow", + "x": 190, + "y": 140, + "width": 0, + "height": 150, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1701031949, + "versionNonce": 904528505, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 150 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a26307959", + "type": "arrow", + "x": 470, + "y": 140, + "width": 0, + "height": 150, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1029355745, + "versionNonce": 181707044, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 150 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t572120009", + "type": "text", + "x": 470.0, + "y": 201.0, + "width": 72.60000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1448840536, + "versionNonce": 1702273013, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": ">>3 + offset", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": ">>3 + offset", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a47520700", + "type": "arrow", + "x": 750, + "y": 140, + "width": 0, + "height": 150, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1159309765, + "versionNonce": 1685543790, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 150 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t382940612", + "type": "text", + "x": 60, + "y": 438, + "width": 1101.1000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 749069591, + "versionNonce": 1623862015, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Real shadow legend from this session's ASan run: 00 addressable, 01-07 partially addressable, fa heap left/right redzone, fd freed heap region, f1-f8 stack states, f9 global redzone.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Real shadow legend from this session's ASan run: 00 addressable, 01-07 partially addressable, fa heap left/right redzone, fd freed heap region, f1-f8 stack states, f9 global redzone.", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t31986229", + "type": "text", + "x": 60, + "y": 456, + "width": 1246.3000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 202026895, + "versionNonce": 1497185338, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "The inline check is why sanitizer overhead stays close to native: this chapter's TSan race run took 0.046s and native 0.002s, versus valgrind's binary-translation memcheck at 1.0s for the same tiny program.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "The inline check is why sanitizer overhead stays close to native: this chapter's TSan race run took 0.046s and native 0.002s, versus valgrind's binary-translation memcheck at 1.0s for the same tiny program.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/29-shadow-memory.svg b/assets/diagrams/29-shadow-memory.svg new file mode 100644 index 0000000..7e5b288 --- /dev/null +++ b/assets/diagrams/29-shadow-memory.svg @@ -0,0 +1,40 @@ + + + +Application memory -- real addresses from this chapter's ASan `uaf` run + +Shadow memory -- 1 shadow byte encodes the state of 8 application bytes + +left redzone +0x7b29be3dff80..ffff +never allocated to this block + +int* p (4 bytes) +0x7b29be3e0010 +new'd, then delete'd -- *p=7 UB here + +right redzone +0x7b29be3e0018..007f +padding around the block + +shadow byte: fa +"Heap left redzone" +covers 0x...ff80-ff87 .. -0007 + +shadow byte: fd +"Freed heap region" +the exact byte the write to *p trips + +shadow byte: fa +"Heap left redzone" +(next block's guard) + +shadow_addr = (app_addr >> 3) + offset -- one comparison before every load/store, not a full CPU emulation + + +>>3 + offset +>>3 + offset + +Real shadow legend from this session's ASan run: 00 addressable, 01-07 partially addressable, fa heap left/right redzone, fd freed heap region, f1-f8 stack states, f9 global redzone. +The inline check is why sanitizer overhead stays close to native: this chapter's TSan race run took 0.046s and native 0.002s, versus valgrind's binary-translation memcheck at 1.0s for the same tiny program. + \ No newline at end of file diff --git a/assets/diagrams/29-tool-coverage-matrix.excalidraw b/assets/diagrams/29-tool-coverage-matrix.excalidraw new file mode 100644 index 0000000..c94d003 --- /dev/null +++ b/assets/diagrams/29-tool-coverage-matrix.excalidraw @@ -0,0 +1,2322 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1877305764", + "type": "rectangle", + "x": 0, + "y": 0, + "width": 1210, + "height": 650, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fbfbfb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1648937952, + "versionNonce": 246469572, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "r1052263784", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 150, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1078676451, + "versionNonce": 1903759051, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t647117195", + "type": "text", + "x": 30, + "y": 28, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 460607581, + "versionNonce": 1082084382, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "defect", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "defect", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r620783819", + "type": "rectangle", + "x": 170, + "y": 20, + "width": 340, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1269428872, + "versionNonce": 1764422844, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1020259232", + "type": "text", + "x": 180, + "y": 28, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 553427090, + "versionNonce": 1316446356, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "C++ (no protection compiled in)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "C++ (no protection compiled in)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1949192437", + "type": "rectangle", + "x": 510, + "y": 20, + "width": 340, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 219653045, + "versionNonce": 1351099562, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t875545816", + "type": "text", + "x": 520, + "y": 28, + "width": 184.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1635698095, + "versionNonce": 976964682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go (GC + runtime checks)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go (GC + runtime checks)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1801387675", + "type": "rectangle", + "x": 850, + "y": 20, + "width": 340, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 318361058, + "versionNonce": 1167957500, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t175960607", + "type": "text", + "x": 860, + "y": 28, + "width": 161.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1102016670, + "versionNonce": 153287104, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Rust (borrow checker)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Rust (borrow checker)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1375845200", + "type": "rectangle", + "x": 20, + "y": 70, + "width": 150, + "height": 98, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1778385773, + "versionNonce": 96050388, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t66485200", + "type": "text", + "x": 30, + "y": 78, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 233663471, + "versionNonce": 620552421, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "leak", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "leak", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1958054890", + "type": "rectangle", + "x": 170, + "y": 70, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 837024907, + "versionNonce": 1415659709, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1211530011", + "type": "text", + "x": 180, + "y": 78, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 842175735, + "versionNonce": 286476677, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind memcheck: CAUGHT", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind memcheck: CAUGHT", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1877655744", + "type": "text", + "x": 180, + "y": 98, + "width": 199.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 769610823, + "versionNonce": 639888250, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"4,096 bytes ... definitely lost\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"4,096 bytes ... definitely lost\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1376751588", + "type": "rectangle", + "x": 510, + "y": 70, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 334439597, + "versionNonce": 86677601, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1364643655", + "type": "text", + "x": 520, + "y": 78, + "width": 254.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1222300560, + "versionNonce": 549086175, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "self-detected: leaked=1 goroutine", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "self-detected: leaked=1 goroutine", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t602294289", + "type": "text", + "x": 520, + "y": 98, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1265331620, + "versionNonce": 1096999142, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind: CONFUSED (630 errors,", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind: CONFUSED (630 errors,", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t257745365", + "type": "text", + "x": 520, + "y": 113, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 748566801, + "versionNonce": 943069903, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "0 real signal -- see below)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "0 real signal -- see below)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1992065991", + "type": "rectangle", + "x": 850, + "y": 70, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1422896646, + "versionNonce": 11093157, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1629864486", + "type": "text", + "x": 860, + "y": 78, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 742184496, + "versionNonce": 1153412048, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind: CAUGHT (65,536B lost)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind: CAUGHT (65,536B lost)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1132028579", + "type": "text", + "x": 860, + "y": 98, + "width": 199.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1423062753, + "versionNonce": 1516717935, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "+ miri (nightly): \"memory leaked\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "+ miri (nightly): \"memory leaked\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r372000580", + "type": "rectangle", + "x": 20, + "y": 178, + "width": 150, + "height": 98, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 459232997, + "versionNonce": 415363182, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1042745739", + "type": "text", + "x": 30, + "y": 186, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1066974500, + "versionNonce": 1611802766, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uninit", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uninit", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r916801216", + "type": "rectangle", + "x": 170, + "y": 178, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1589697812, + "versionNonce": 10059342, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1646601639", + "type": "text", + "x": 180, + "y": 186, + "width": 192.50000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 889636850, + "versionNonce": 1635856598, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind memcheck: CAUGHT", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind memcheck: CAUGHT", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1166417469", + "type": "text", + "x": 180, + "y": 206, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1871778357, + "versionNonce": 1117593516, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"uninitialised value(s)\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"uninitialised value(s)\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1063071355", + "type": "text", + "x": 180, + "y": 221, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1201528204, + "versionNonce": 1383759347, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ASan alone: MISS (needs MSan)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ASan alone: MISS (needs MSan)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1436186307", + "type": "rectangle", + "x": 510, + "y": 178, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 258467556, + "versionNonce": 1148712493, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1333048839", + "type": "text", + "x": 520, + "y": 186, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 524922198, + "versionNonce": 1176169080, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: every value zeroed", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: every value zeroed", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t529213862", + "type": "text", + "x": 520, + "y": 206, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1596411788, + "versionNonce": 1628257304, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "exit 64, no tool needed", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "exit 64, no tool needed", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r816110819", + "type": "rectangle", + "x": 850, + "y": 178, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 947080709, + "versionNonce": 733046528, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t696162806", + "type": "text", + "x": 860, + "y": 186, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 162481795, + "versionNonce": 1863309438, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: no unsafe value", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: no unsafe value", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t209804847", + "type": "text", + "x": 860, + "y": 206, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1543284287, + "versionNonce": 235539178, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "exit 64, no tool needed", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "exit 64, no tool needed", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1978322345", + "type": "rectangle", + "x": 20, + "y": 286, + "width": 150, + "height": 98, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 907597731, + "versionNonce": 953993974, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t541212816", + "type": "text", + "x": 30, + "y": 294, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1041704106, + "versionNonce": 366900514, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uaf", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uaf", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r887768375", + "type": "rectangle", + "x": 170, + "y": 286, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 279230519, + "versionNonce": 1703602234, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1548787032", + "type": "text", + "x": 180, + "y": 294, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 11964677, + "versionNonce": 977781472, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind: CAUGHT (Invalid write,", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind: CAUGHT (Invalid write,", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1017399843", + "type": "text", + "x": 180, + "y": 314, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1153891353, + "versionNonce": 1903769027, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"freed\") + ASan: CAUGHT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"freed\") + ASan: CAUGHT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t440555561", + "type": "text", + "x": 180, + "y": 329, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 269952763, + "versionNonce": 482765415, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"heap-use-after-free\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"heap-use-after-free\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r606434224", + "type": "rectangle", + "x": 510, + "y": 286, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1082873785, + "versionNonce": 619964146, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t910692268", + "type": "text", + "x": 520, + "y": 294, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1890553892, + "versionNonce": 690819043, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: GC won't free a", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: GC won't free a", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t540778531", + "type": "text", + "x": 520, + "y": 314, + "width": 139.15, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1010930499, + "versionNonce": 1609195666, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "live reference; exit 64", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "live reference; exit 64", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1260251142", + "type": "rectangle", + "x": 850, + "y": 286, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 441246984, + "versionNonce": 1088032156, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t704782592", + "type": "text", + "x": 860, + "y": 294, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 884212897, + "versionNonce": 607664607, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: borrow checker;", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: borrow checker;", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t379393523", + "type": "text", + "x": 860, + "y": 314, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 420993369, + "versionNonce": 1501205029, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "unsafe+miri: CAUGHT (below)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "unsafe+miri: CAUGHT (below)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r296201772", + "type": "rectangle", + "x": 20, + "y": 394, + "width": 150, + "height": 98, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1880129952, + "versionNonce": 1332920342, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t453538940", + "type": "text", + "x": 30, + "y": 402, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 366482995, + "versionNonce": 1597335468, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "overflow", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "overflow", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r764649998", + "type": "rectangle", + "x": 170, + "y": 394, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1414371864, + "versionNonce": 1360572834, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1749650120", + "type": "text", + "x": 180, + "y": 402, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 817723526, + "versionNonce": 767070649, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "valgrind: CAUGHT (\"0 bytes after", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "valgrind: CAUGHT (\"0 bytes after", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1734281073", + "type": "text", + "x": 180, + "y": 422, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1097596456, + "versionNonce": 716479090, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "a block\") + ASan: CAUGHT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "a block\") + ASan: CAUGHT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1776898924", + "type": "text", + "x": 180, + "y": 437, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 387692966, + "versionNonce": 1333209096, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"heap-buffer-overflow\"", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"heap-buffer-overflow\"", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1099740185", + "type": "rectangle", + "x": 510, + "y": 394, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 609037983, + "versionNonce": 45329767, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1329402464", + "type": "text", + "x": 520, + "y": 402, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 124628090, + "versionNonce": 1062648218, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: slice/array bounds", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: slice/array bounds", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1857824616", + "type": "text", + "x": 520, + "y": 422, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1163324480, + "versionNonce": 1348928699, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "check panics; exit 64", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "check panics; exit 64", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r573233283", + "type": "rectangle", + "x": 850, + "y": 394, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1969147878, + "versionNonce": 416840552, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1969183473", + "type": "text", + "x": 860, + "y": 402, + "width": 215.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 566565047, + "versionNonce": 1617389991, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: bounds-checked or", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: bounds-checked or", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1327473277", + "type": "text", + "x": 860, + "y": 422, + "width": 145.20000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1739552911, + "versionNonce": 1879772959, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "rejected at compile time", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "rejected at compile time", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r928553723", + "type": "rectangle", + "x": 20, + "y": 502, + "width": 150, + "height": 98, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 976628272, + "versionNonce": 1257687298, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t947538784", + "type": "text", + "x": 30, + "y": 510, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 493240014, + "versionNonce": 275148951, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "race", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "race", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1587247833", + "type": "rectangle", + "x": 170, + "y": 502, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1708383961, + "versionNonce": 1635020812, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1782102004", + "type": "text", + "x": 180, + "y": 510, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1419824591, + "versionNonce": 1076993039, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "memcheck: not its job (untested)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "memcheck: not its job (untested)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1415268780", + "type": "text", + "x": 180, + "y": 530, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 608819816, + "versionNonce": 1813590614, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "helgrind: CAUGHT (\"Possible", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "helgrind: CAUGHT (\"Possible", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t629081433", + "type": "text", + "x": 180, + "y": 545, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1446309357, + "versionNonce": 1489221852, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "data race\") + TSan: CAUGHT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "data race\") + TSan: CAUGHT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1699662085", + "type": "rectangle", + "x": 510, + "y": 502, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1847484649, + "versionNonce": 1427196420, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t406441714", + "type": "text", + "x": 520, + "y": 510, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 425007734, + "versionNonce": 203112290, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "go -race: CAUGHT", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "go -race: CAUGHT", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t771140059", + "type": "text", + "x": 520, + "y": 530, + "width": 175.45000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1549221340, + "versionNonce": 1910627048, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "\"WARNING: DATA RACE\", exit 66", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "\"WARNING: DATA RACE\", exit 66", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r710860434", + "type": "rectangle", + "x": 850, + "y": 502, + "width": 340, + "height": 98, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1518944969, + "versionNonce": 359044311, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t274998165", + "type": "text", + "x": 860, + "y": 510, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 433895100, + "versionNonce": 1001427142, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prevented: Send/Sync rules", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prevented: Send/Sync rules", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t164518977", + "type": "text", + "x": 860, + "y": 530, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 324647094, + "versionNonce": 1588461313, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "reject the unsync'd share", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "reject the unsync'd share", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t570015135", + "type": "text", + "x": 20, + "y": 620, + "width": 1024.65, + "height": 14.375, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 969375978, + "versionNonce": 1735094743, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "amber = a tool run this session caught the defect \u00b7 dark = the language prevents it outright, no tool needed \u00b7 plain = a real blind spot this session measured", + "fontSize": 11.5, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "amber = a tool run this session caught the defect \u00b7 dark = the language prevents it outright, no tool needed \u00b7 plain = a real blind spot this session measured", + "lineHeight": 1.25, + "baseline": 11.5 + }, + { + "id": "t120759823", + "type": "text", + "x": 20, + "y": 638, + "width": 695.75, + "height": 13.75, + "angle": 0, + "strokeColor": "#666666", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 787218117, + "versionNonce": 946160421, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\" -- none of this is asserted from memory.", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Every cell is backed by a command in \"Build, run, observe\" / \"Cross-check\" -- none of this is asserted from memory.", + "lineHeight": 1.25, + "baseline": 11 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/29-tool-coverage-matrix.svg b/assets/diagrams/29-tool-coverage-matrix.svg new file mode 100644 index 0000000..1527430 --- /dev/null +++ b/assets/diagrams/29-tool-coverage-matrix.svg @@ -0,0 +1,74 @@ + + + + +defect + +C++ (no protection compiled in) + +Go (GC + runtime checks) + +Rust (borrow checker) + +leak + +valgrind memcheck: CAUGHT +"4,096 bytes ... definitely lost" + +self-detected: leaked=1 goroutine +valgrind: CONFUSED (630 errors, +0 real signal -- see below) + +valgrind: CAUGHT (65,536B lost) ++ miri (nightly): "memory leaked" + +uninit + +valgrind memcheck: CAUGHT +"uninitialised value(s)" +ASan alone: MISS (needs MSan) + +prevented: every value zeroed +exit 64, no tool needed + +prevented: no unsafe value +exit 64, no tool needed + +uaf + +valgrind: CAUGHT (Invalid write, +"freed") + ASan: CAUGHT +"heap-use-after-free" + +prevented: GC won't free a +live reference; exit 64 + +prevented: borrow checker; +unsafe+miri: CAUGHT (below) + +overflow + +valgrind: CAUGHT ("0 bytes after +a block") + ASan: CAUGHT +"heap-buffer-overflow" + +prevented: slice/array bounds +check panics; exit 64 + +prevented: bounds-checked or +rejected at compile time + +race + +memcheck: not its job (untested) +helgrind: CAUGHT ("Possible +data race") + TSan: CAUGHT + +go -race: CAUGHT +"WARNING: DATA RACE", exit 66 + +prevented: Send/Sync rules +reject the unsync'd share +amber = a tool run this session caught the defect · dark = the language prevents it outright, no tool needed · plain = a real blind spot this session measured +Every cell is backed by a command in "Build, run, observe" / "Cross-check" -- none of this is asserted from memory. + \ No newline at end of file diff --git a/assets/diagrams/30-ebpf-tooling-stack.excalidraw b/assets/diagrams/30-ebpf-tooling-stack.excalidraw new file mode 100644 index 0000000..3492142 --- /dev/null +++ b/assets/diagrams/30-ebpf-tooling-stack.excalidraw @@ -0,0 +1,2471 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r889586152", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 1160, + "height": 90, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1593333648, + "versionNonce": 33407335, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1329355475", + "type": "text", + "x": 32, + "y": 26, + "width": 151.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1752355980, + "versionNonce": 1727264931, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel probe primitives", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel probe primitives", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r530502448", + "type": "rectangle", + "x": 20, + "y": 130, + "width": 1160, + "height": 170, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 201825200, + "versionNonce": 1653236552, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1616790358", + "type": "text", + "x": 32, + "y": 136, + "width": 376.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 882729578, + "versionNonce": 898471669, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "eBPF-based tooling (this chapter, live on systems-target)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "eBPF-based tooling (this chapter, live on systems-target)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1945088781", + "type": "rectangle", + "x": 20, + "y": 320, + "width": 1160, + "height": 90, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1122949582, + "versionNonce": 819431524, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1619878170", + "type": "text", + "x": 32, + "y": 326, + "width": 270.6, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1766943416, + "versionNonce": 963668679, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "sibling technology (Ch. 45 covers deeper)", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "sibling technology (Ch. 45 covers deeper)", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1621989384", + "type": "rectangle", + "x": 20, + "y": 430, + "width": 1160, + "height": 110, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 426684623, + "versionNonce": 1199202204, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1300588738", + "type": "text", + "x": 32, + "y": 436, + "width": 217.8, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 757617299, + "versionNonce": 1427084986, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "target binaries under observation", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "target binaries under observation", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1217545443", + "type": "rectangle", + "x": 50, + "y": 55, + "width": 210, + "height": 44, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 944217614, + "versionNonce": 603418030, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1707653059", + "type": "text", + "x": 60, + "y": 63, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 111939809, + "versionNonce": 1826113646, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kprobe / kretprobe", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kprobe / kretprobe", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t537224998", + "type": "text", + "x": 60, + "y": 83, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 201524053, + "versionNonce": 507786177, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel functions", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel functions", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r230376830", + "type": "rectangle", + "x": 280, + "y": 55, + "width": 210, + "height": 44, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1109247556, + "versionNonce": 950547392, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1322612543", + "type": "text", + "x": 290, + "y": 63, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 96882453, + "versionNonce": 1754864493, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uprobe / uretprobe", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uprobe / uretprobe", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t138108210", + "type": "text", + "x": 290, + "y": 83, + "width": 114.95, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1777790320, + "versionNonce": 616060659, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "userspace functions", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "userspace functions", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1921175110", + "type": "rectangle", + "x": 510, + "y": 55, + "width": 210, + "height": 44, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1256368038, + "versionNonce": 429545710, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t94138712", + "type": "text", + "x": 520, + "y": 63, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 744474064, + "versionNonce": 1886244721, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tracepoint", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tracepoint", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1336540820", + "type": "text", + "x": 520, + "y": 83, + "width": 102.85000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 420217084, + "versionNonce": 1983797077, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stable kernel ABI", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stable kernel ABI", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1129575772", + "type": "rectangle", + "x": 740, + "y": 55, + "width": 210, + "height": 44, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1159232326, + "versionNonce": 763818538, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1099637864", + "type": "text", + "x": 750, + "y": 63, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1436454461, + "versionNonce": 1556206417, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "perf_event", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "perf_event", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t245368294", + "type": "text", + "x": 750, + "y": 83, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 598254954, + "versionNonce": 586690867, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "timer / PMU sampling", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "timer / PMU sampling", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r978372932", + "type": "rectangle", + "x": 970, + "y": 55, + "width": 180, + "height": 44, + "angle": 0, + "strokeColor": "#888888", + "backgroundColor": "#f4f4f4", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 127826569, + "versionNonce": 384139372, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1188022591", + "type": "text", + "x": 980, + "y": 63, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 45013036, + "versionNonce": 1759333477, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "BTF / CO-RE", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "BTF / CO-RE", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1327774354", + "type": "text", + "x": 980, + "y": 83, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 532632371, + "versionNonce": 162196902, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "struct layout w/o headers", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "struct layout w/o headers", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1223473594", + "type": "rectangle", + "x": 50, + "y": 160, + "width": 330, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1318943024, + "versionNonce": 1303195729, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1189960346", + "type": "text", + "x": 60, + "y": 168, + "width": 69.30000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1902440061, + "versionNonce": 970698086, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bcc-tools", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bcc-tools", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t460047993", + "type": "text", + "x": 60, + "y": 188, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 827867338, + "versionNonce": 1652662450, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "execsnoop/opensnoop/biolatency/", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "execsnoop/opensnoop/biolatency/", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t314939759", + "type": "text", + "x": 60, + "y": 203, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1803623059, + "versionNonce": 463707251, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "offcputime/funccount/profile", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "offcputime/funccount/profile", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1462394268", + "type": "rectangle", + "x": 400, + "y": 160, + "width": 300, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 253736548, + "versionNonce": 999665921, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t494211406", + "type": "text", + "x": 410, + "y": 168, + "width": 61.60000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1426747478, + "versionNonce": 1769843988, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bpftrace", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bpftrace", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1711790827", + "type": "text", + "x": 410, + "y": 188, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1803778036, + "versionNonce": 1173904203, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one-liners + scripts,", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one-liners + scripts,", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t875790971", + "type": "text", + "x": 410, + "y": 203, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1805436926, + "versionNonce": 1964150639, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "probe:target /filter/ { action }", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "probe:target /filter/ { action }", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r706623541", + "type": "rectangle", + "x": 720, + "y": 160, + "width": 220, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 108440125, + "versionNonce": 751635119, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t564810929", + "type": "text", + "x": 730, + "y": 168, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 444916260, + "versionNonce": 807887676, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bpftool", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bpftool", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1245135468", + "type": "text", + "x": 730, + "y": 188, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 128983277, + "versionNonce": 1701222290, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "prog show / map show,", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "prog show / map show,", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1824102886", + "type": "text", + "x": 730, + "y": 203, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 691470218, + "versionNonce": 1325261267, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inspect what's loaded", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inspect what's loaded", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r914810587", + "type": "rectangle", + "x": 960, + "y": 160, + "width": 190, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 289800738, + "versionNonce": 1921218108, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1777739666", + "type": "text", + "x": 970, + "y": 168, + "width": 53.900000000000006, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1694636092, + "versionNonce": 519018691, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "c++filt", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "c++filt", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t560313249", + "type": "text", + "x": 970, + "y": 188, + "width": 108.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1833694475, + "versionNonce": 1179923932, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "demangle the exact", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "demangle the exact", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1077517753", + "type": "text", + "x": 970, + "y": 203, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 388347801, + "versionNonce": 1696712372, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "symbol you attach to", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "symbol you attach to", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1392461052", + "type": "rectangle", + "x": 50, + "y": 240, + "width": 1090, + "height": 40, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 351570785, + "versionNonce": 1520016784, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t67588642", + "type": "text", + "x": 60, + "y": 248, + "width": 1008.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1846859214, + "versionNonce": 1287228043, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "libbpf loads a small verified BPF program per probe; the kernel runs it in-kernel on each hit and maps the result back to userspace", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "libbpf loads a small verified BPF program per probe; the kernel runs it in-kernel on each hit and maps the result back to userspace", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1352865200", + "type": "rectangle", + "x": 50, + "y": 355, + "width": 300, + "height": 55, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1436734793, + "versionNonce": 63005819, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1601303266", + "type": "text", + "x": 60, + "y": 363, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 67569618, + "versionNonce": 830698455, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SystemTap (stap)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SystemTap (stap)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t22683001", + "type": "text", + "x": 60, + "y": 383, + "width": 133.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1740700384, + "versionNonce": 1567016383, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "kernel-module backend:", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "kernel-module backend:", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t589849612", + "type": "text", + "x": 60, + "y": 398, + "width": 163.35000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1797616110, + "versionNonce": 1944321161, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "needs matching kernel-devel", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "needs matching kernel-devel", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r443513350", + "type": "rectangle", + "x": 380, + "y": 355, + "width": 300, + "height": 55, + "angle": 0, + "strokeColor": "#2f6db5", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 650709243, + "versionNonce": 1083372144, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t334464463", + "type": "text", + "x": 390, + "y": 363, + "width": 169.4, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 220795468, + "versionNonce": 1269608185, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "stap --runtime=dyninst", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "stap --runtime=dyninst", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1139057701", + "type": "text", + "x": 390, + "y": 383, + "width": 157.3, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 454915369, + "versionNonce": 1481150779, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "userspace binary patching,", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "userspace binary patching,", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1509980651", + "type": "text", + "x": 390, + "y": 398, + "width": 96.80000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 638178509, + "versionNonce": 1748427669, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no kernel module", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no kernel module", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1925722164", + "type": "rectangle", + "x": 60, + "y": 465, + "width": 340, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 613995853, + "versionNonce": 408224485, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1741312053", + "type": "text", + "x": 70, + "y": 473, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 739308547, + "versionNonce": 206184392, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "C++23: busy_hash", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "C++23: busy_hash", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1220267607", + "type": "text", + "x": 70, + "y": 493, + "width": 121.00000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1172738009, + "versionNonce": 1716401526, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "extern \"C\", SysV ABI", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "extern \"C\", SysV ABI", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t534249963", + "type": "text", + "x": 70, + "y": 508, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 361964860, + "versionNonce": 222583955, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "arg0 == RDI (correct)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "arg0 == RDI (correct)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r409984143", + "type": "rectangle", + "x": 430, + "y": 465, + "width": 340, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1425657298, + "versionNonce": 1592741560, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t679605478", + "type": "text", + "x": 440, + "y": 473, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 517184601, + "versionNonce": 1650754576, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Rust: cmd_work", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Rust: cmd_work", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t813513751", + "type": "text", + "x": 440, + "y": 493, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1795699295, + "versionNonce": 120422131, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "_RNvCs...cmd_work (v0 mangling)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "_RNvCs...cmd_work (v0 mangling)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1900345430", + "type": "text", + "x": 440, + "y": 508, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1701826023, + "versionNonce": 1935865825, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "SysV ABI, arg0 == RDI (correct)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "SysV ABI, arg0 == RDI (correct)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1681840542", + "type": "rectangle", + "x": 800, + "y": 465, + "width": 340, + "height": 55, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1168107473, + "versionNonce": 1741506838, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1690376801", + "type": "text", + "x": 810, + "y": 473, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1692473117, + "versionNonce": 741518823, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go: main.cmdWork", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go: main.cmdWork", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1299304068", + "type": "text", + "x": 810, + "y": 493, + "width": 127.05000000000001, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1512211950, + "versionNonce": 758207132, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "ABIInternal registers", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "ABIInternal registers", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1946634212", + "type": "text", + "x": 810, + "y": 508, + "width": 235.95000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1491668927, + "versionNonce": 723857409, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "arg0/RDI == 1 (wrong); RAX == 9 (right)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "arg0/RDI == 1 (wrong); RAX == 9 (right)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a1518556580", + "type": "arrow", + "x": 155, + "y": 99, + "width": 0, + "height": 61, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 814231924, + "versionNonce": 1807105799, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a209947409", + "type": "arrow", + "x": 385, + "y": 99, + "width": 115, + "height": 61, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 955907769, + "versionNonce": 1832424832, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 115, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1403581601", + "type": "arrow", + "x": 615, + "y": 99, + "width": 65, + "height": 61, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 808549254, + "versionNonce": 829948440, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -65, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1719648722", + "type": "arrow", + "x": 845, + "y": 99, + "width": 15, + "height": 61, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 295274441, + "versionNonce": 232661733, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -15, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a998265955", + "type": "arrow", + "x": 1060, + "y": 99, + "width": 5, + "height": 61, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1008291408, + "versionNonce": 690687187, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -5, + 61 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a166950383", + "type": "arrow", + "x": 200, + "y": 280, + "width": 0, + "height": 75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 426435855, + "versionNonce": 1144829492, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 75 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a306563577", + "type": "arrow", + "x": 230, + "y": 465, + "width": 360, + "height": 245, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 894683188, + "versionNonce": 1332115957, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 360, + -245 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a68430036", + "type": "arrow", + "x": 600, + "y": 465, + "width": 50, + "height": 245, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 383371516, + "versionNonce": 1428357894, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -50, + -245 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a539474616", + "type": "arrow", + "x": 970, + "y": 465, + "width": 420, + "height": 245, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 733081830, + "versionNonce": 356691848, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + -420, + -245 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1491338044", + "type": "text", + "x": 760.0, + "y": 328.5, + "width": 78.65, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1492239304, + "versionNonce": 251604112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "arg0 mismatch", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "arg0 mismatch", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t36729944", + "type": "text", + "x": 50, + "y": 545, + "width": 1157.4750000000001, + "height": 14.375, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1034406791, + "versionNonce": 1741850906, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Every amber box above ran live against systems-target this session; stap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, dyninst attached without erroring.", + "fontSize": 11.5, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Every amber box above ran live against systems-target this session; stap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, dyninst attached without erroring.", + "lineHeight": 1.25, + "baseline": 11.5 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/30-ebpf-tooling-stack.svg b/assets/diagrams/30-ebpf-tooling-stack.svg new file mode 100644 index 0000000..d14201b --- /dev/null +++ b/assets/diagrams/30-ebpf-tooling-stack.svg @@ -0,0 +1,76 @@ + + + +kernel probe primitives + +eBPF-based tooling (this chapter, live on systems-target) + +sibling technology (Ch. 45 covers deeper) + +target binaries under observation + +kprobe / kretprobe +kernel functions + +uprobe / uretprobe +userspace functions + +tracepoint +stable kernel ABI + +perf_event +timer / PMU sampling + +BTF / CO-RE +struct layout w/o headers + +bcc-tools +execsnoop/opensnoop/biolatency/ +offcputime/funccount/profile + +bpftrace +one-liners + scripts, +probe:target /filter/ { action } + +bpftool +prog show / map show, +inspect what's loaded + +c++filt +demangle the exact +symbol you attach to + +libbpf loads a small verified BPF program per probe; the kernel runs it in-kernel on each hit and maps the result back to userspace + +SystemTap (stap) +kernel-module backend: +needs matching kernel-devel + +stap --runtime=dyninst +userspace binary patching, +no kernel module + +C++23: busy_hash +extern "C", SysV ABI +arg0 == RDI (correct) + +Rust: cmd_work +_RNvCs...cmd_work (v0 mangling) +SysV ABI, arg0 == RDI (correct) + +Go: main.cmdWork +ABIInternal registers +arg0/RDI == 1 (wrong); RAX == 9 (right) + + + + + + + + + +arg0 mismatch +arg0 mismatch +Every amber box above ran live against systems-target this session; stap's kernel-module path failed on a real kernel-devel/running-kernel mismatch, dyninst attached without erroring. + \ No newline at end of file diff --git a/assets/diagrams/30-uprobe-attachment.excalidraw b/assets/diagrams/30-uprobe-attachment.excalidraw new file mode 100644 index 0000000..a132da0 --- /dev/null +++ b/assets/diagrams/30-uprobe-attachment.excalidraw @@ -0,0 +1,1736 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1754098421", + "type": "rectangle", + "x": 20, + "y": 20, + "width": 1140, + "height": 490, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1583202044, + "versionNonce": 349993142, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "r1956138614", + "type": "rectangle", + "x": 40, + "y": 50, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 275273923, + "versionNonce": 886798148, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t847720020", + "type": "text", + "x": 50, + "y": 58, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1522403789, + "versionNonce": 1398773431, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "C++23 binary (app)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "C++23 binary (app)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t132363940", + "type": "text", + "x": 50, + "y": 78, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1713977070, + "versionNonce": 1874744189, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "nm: t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "nm: t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r757417764", + "type": "rectangle", + "x": 40, + "y": 130, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1552620513, + "versionNonce": 1187777204, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1592487249", + "type": "text", + "x": 50, + "y": 138, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 983475079, + "versionNonce": 450850914, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "c++filt (auto Itanium ABI)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "c++filt (auto Itanium ABI)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t987857793", + "type": "text", + "x": 50, + "y": 158, + "width": 235.95000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 10288797, + "versionNonce": 926688393, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "-> (anonymous namespace)::cmd_work(int)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "-> (anonymous namespace)::cmd_work(int)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1010574982", + "type": "rectangle", + "x": 40, + "y": 210, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 369654337, + "versionNonce": 922785865, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t565207477", + "type": "text", + "x": 50, + "y": 218, + "width": 361.90000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 431051904, + "versionNonce": 452324387, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uprobe:app:_ZN12_GLOBAL__N_1L8cmd_workEi.isra.0", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uprobe:app:_ZN12_GLOBAL__N_1L8cmd_workEi.isra.0", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1738581943", + "type": "text", + "x": 50, + "y": 238, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1499984478, + "versionNonce": 1157660511, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "attach target is the MANGLED string", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "attach target is the MANGLED string", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1965534206", + "type": "rectangle", + "x": 40, + "y": 290, + "width": 340, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1222669917, + "versionNonce": 1699157665, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1261547165", + "type": "text", + "x": 50, + "y": 298, + "width": 261.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1241299517, + "versionNonce": 1005956203, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fired: seconds_arg=5 (--seconds 5)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fired: seconds_arg=5 (--seconds 5)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t173005726", + "type": "text", + "x": 50, + "y": 318, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 260768910, + "versionNonce": 441360910, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "arg0==RDI is correct: SysV x86-64 ABI", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "arg0==RDI is correct: SysV x86-64 ABI", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1121188611", + "type": "rectangle", + "x": 420, + "y": 50, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1959440368, + "versionNonce": 1356766795, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1482149366", + "type": "text", + "x": 430, + "y": 58, + "width": 130.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1042916993, + "versionNonce": 1206005769, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Rust binary (app)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Rust binary (app)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t885813197", + "type": "text", + "x": 430, + "y": 78, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 920429240, + "versionNonce": 218749629, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "nm: t _RNvCs6FzKXswKZCc_3app8cmd_work", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "nm: t _RNvCs6FzKXswKZCc_3app8cmd_work", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1474437593", + "type": "rectangle", + "x": 420, + "y": 130, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 446448860, + "versionNonce": 240267415, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1787512226", + "type": "text", + "x": 430, + "y": 138, + "width": 231.00000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 209044707, + "versionNonce": 1847704033, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "c++filt (auto-detects Rust v0)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "c++filt (auto-detects Rust v0)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1410974172", + "type": "text", + "x": 430, + "y": 158, + "width": 205.70000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 825327829, + "versionNonce": 436567840, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "-> app[4db24ee1e6ed4506]::cmd_work", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "-> app[4db24ee1e6ed4506]::cmd_work", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1065197268", + "type": "rectangle", + "x": 420, + "y": 210, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 885907292, + "versionNonce": 1219958086, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1059537249", + "type": "text", + "x": 430, + "y": 218, + "width": 323.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1895707063, + "versionNonce": 600921453, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uprobe:app:_RNvCs6FzKXswKZCc_3app8cmd_work", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uprobe:app:_RNvCs6FzKXswKZCc_3app8cmd_work", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1216978632", + "type": "text", + "x": 430, + "y": 238, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 966912513, + "versionNonce": 184329965, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "attach target is the MANGLED string", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "attach target is the MANGLED string", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1403478398", + "type": "rectangle", + "x": 420, + "y": 290, + "width": 340, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1491501512, + "versionNonce": 906350814, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t400267280", + "type": "text", + "x": 430, + "y": 298, + "width": 261.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1034200950, + "versionNonce": 1839232616, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fired: seconds_arg=7 (--seconds 7)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fired: seconds_arg=7 (--seconds 7)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1151099042", + "type": "text", + "x": 430, + "y": 318, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1704188117, + "versionNonce": 1632592338, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "arg0==RDI is correct: SysV x86-64 ABI", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "arg0==RDI is correct: SysV x86-64 ABI", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r240611804", + "type": "rectangle", + "x": 800, + "y": 50, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1131249876, + "versionNonce": 489901900, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1964879859", + "type": "text", + "x": 810, + "y": 58, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1067157508, + "versionNonce": 1190182434, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go binary (app)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go binary (app)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1735980169", + "type": "text", + "x": 810, + "y": 78, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1088527976, + "versionNonce": 5792331, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "nm: T main.cmdWork (never mangled)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "nm: T main.cmdWork (never mangled)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r862673389", + "type": "rectangle", + "x": 800, + "y": 130, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1764798659, + "versionNonce": 1694862538, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t703606078", + "type": "text", + "x": 810, + "y": 138, + "width": 161.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 981252059, + "versionNonce": 1346651120, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "no demangling needed;", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "no demangling needed;", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1378250094", + "type": "text", + "x": 810, + "y": 158, + "width": 217.8, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 379816349, + "versionNonce": 551176123, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "package-qualified name IS the symbol", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "package-qualified name IS the symbol", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r978017236", + "type": "rectangle", + "x": 800, + "y": 210, + "width": 340, + "height": 50, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1476685171, + "versionNonce": 1968166007, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t670363807", + "type": "text", + "x": 810, + "y": 218, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 617149996, + "versionNonce": 581552661, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "uprobe:app:main.cmdWork", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "uprobe:app:main.cmdWork", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1214002371", + "type": "text", + "x": 810, + "y": 238, + "width": 181.50000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1045209074, + "versionNonce": 1806221682, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "plain, unmangled attach target", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "plain, unmangled attach target", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1874714109", + "type": "rectangle", + "x": 800, + "y": 290, + "width": 340, + "height": 60, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 117538039, + "versionNonce": 1138719962, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1534056229", + "type": "text", + "x": 810, + "y": 298, + "width": 177.10000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1576011080, + "versionNonce": 769011179, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "fired with --seconds 9:", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "fired with --seconds 9:", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t992372469", + "type": "text", + "x": 810, + "y": 318, + "width": 187.55, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1680507688, + "versionNonce": 1121067639, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "RDI(arg0)=1 WRONG RAX=9 RIGHT", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "RDI(arg0)=1 WRONG RAX=9 RIGHT", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r684409042", + "type": "rectangle", + "x": 40, + "y": 400, + "width": 1100, + "height": 90, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 49802162, + "versionNonce": 1979318144, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t167950310", + "type": "text", + "x": 50, + "y": 408, + "width": 592.9000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 128581071, + "versionNonce": 347771222, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go's ABIInternal (1.17+) passes integer args in RAX,RBX,RCX,RDI,RSI,R8,R9,...", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go's ABIInternal (1.17+) passes integer args in RAX,RBX,RCX,RDI,RSI,R8,R9,...", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t970376882", + "type": "text", + "x": 50, + "y": 428, + "width": 381.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1828806705, + "versionNonce": 1382756112, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "not the SysV RDI,RSI,RDX,... order bpftrace's arg0/arg1 assumes", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "not the SysV RDI,RSI,RDX,... order bpftrace's arg0/arg1 assumes", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1937755755", + "type": "text", + "x": 50, + "y": 443, + "width": 538.45, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 468466170, + "versionNonce": 656728356, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "goroutine stacks also grow by copying, so a captured stack address can move under a probe", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "goroutine stacks also grow by copying, so a captured stack address can move under a probe", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a438046888", + "type": "arrow", + "x": 210, + "y": 100, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 449451399, + "versionNonce": 1353320032, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1373296296", + "type": "arrow", + "x": 210, + "y": 180, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1549615284, + "versionNonce": 634518804, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1750337447", + "type": "arrow", + "x": 210, + "y": 260, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1639899922, + "versionNonce": 1618255226, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1552158325", + "type": "arrow", + "x": 590, + "y": 100, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 820813757, + "versionNonce": 1140255335, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a633919358", + "type": "arrow", + "x": 590, + "y": 180, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 632908165, + "versionNonce": 1385924214, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a439584910", + "type": "arrow", + "x": 590, + "y": 260, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 762165031, + "versionNonce": 1229498226, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a1104889453", + "type": "arrow", + "x": 970, + "y": 100, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1791760123, + "versionNonce": 855125513, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a246281700", + "type": "arrow", + "x": 970, + "y": 180, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1144781034, + "versionNonce": 58741643, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a467384221", + "type": "arrow", + "x": 970, + "y": 260, + "width": 0, + "height": 30, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1174341852, + "versionNonce": 1054404856, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 30 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/30-uprobe-attachment.svg b/assets/diagrams/30-uprobe-attachment.svg new file mode 100644 index 0000000..acfbf43 --- /dev/null +++ b/assets/diagrams/30-uprobe-attachment.svg @@ -0,0 +1,53 @@ + + + + +C++23 binary (app) +nm: t _ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 + +c++filt (auto Itanium ABI) +-> (anonymous namespace)::cmd_work(int) + +uprobe:app:_ZN12_GLOBAL__N_1L8cmd_workEi.isra.0 +attach target is the MANGLED string + +fired: seconds_arg=5 (--seconds 5) +arg0==RDI is correct: SysV x86-64 ABI + +Rust binary (app) +nm: t _RNvCs6FzKXswKZCc_3app8cmd_work + +c++filt (auto-detects Rust v0) +-> app[4db24ee1e6ed4506]::cmd_work + +uprobe:app:_RNvCs6FzKXswKZCc_3app8cmd_work +attach target is the MANGLED string + +fired: seconds_arg=7 (--seconds 7) +arg0==RDI is correct: SysV x86-64 ABI + +Go binary (app) +nm: T main.cmdWork (never mangled) + +no demangling needed; +package-qualified name IS the symbol + +uprobe:app:main.cmdWork +plain, unmangled attach target + +fired with --seconds 9: +RDI(arg0)=1 WRONG RAX=9 RIGHT + +Go's ABIInternal (1.17+) passes integer args in RAX,RBX,RCX,RDI,RSI,R8,R9,... +not the SysV RDI,RSI,RDX,... order bpftrace's arg0/arg1 assumes +goroutine stacks also grow by copying, so a captured stack address can move under a probe + + + + + + + + + + \ No newline at end of file diff --git a/assets/diagrams/31-flamegraph-reading.excalidraw b/assets/diagrams/31-flamegraph-reading.excalidraw new file mode 100644 index 0000000..2f48ed6 --- /dev/null +++ b/assets/diagrams/31-flamegraph-reading.excalidraw @@ -0,0 +1,1056 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1342027046", + "type": "rectangle", + "x": 30, + "y": 40, + "width": 1120, + "height": 430, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1891182095, + "versionNonce": 1349770702, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1114294625", + "type": "text", + "x": 42, + "y": 46, + "width": 376.20000000000005, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1105643368, + "versionNonce": 797994939, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "one flamegraph: each box is a stack frame sampled by perf", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "one flamegraph: each box is a stack frame sampled by perf", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1679648468", + "type": "rectangle", + "x": 30, + "y": 480, + "width": 230, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1328428637, + "versionNonce": 585415125, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t693240986", + "type": "text", + "x": 40, + "y": 488, + "width": 115.50000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 714944571, + "versionNonce": 842678581, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "y = stack depth", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "y = stack depth", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t558825819", + "type": "text", + "x": 40, + "y": 508, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1386588300, + "versionNonce": 1903486150, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bottom: root top: leaf (running now)", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bottom: root top: leaf (running now)", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r3641925", + "type": "rectangle", + "x": 300, + "y": 480, + "width": 230, + "height": 56, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1127025188, + "versionNonce": 1005676419, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1858843151", + "type": "text", + "x": 310, + "y": 488, + "width": 154.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1745982271, + "versionNonce": 1078513578, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "x = share of samples", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "x = share of samples", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t619874292", + "type": "text", + "x": 310, + "y": 508, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1089939092, + "versionNonce": 693847700, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "NOT time order or call order", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "NOT time order or call order", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1310277838", + "type": "rectangle", + "x": 570, + "y": 480, + "width": 290, + "height": 56, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1676923547, + "versionNonce": 1498230013, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1658065085", + "type": "text", + "x": 580, + "y": 488, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 342469872, + "versionNonce": 942870248, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "widest box = the hot function", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "widest box = the hot function", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t120163902", + "type": "text", + "x": 580, + "y": 508, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 521414714, + "versionNonce": 1371431073, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spin_hot / hotSpin at ~99% of samples", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spin_hot / hotSpin at ~99% of samples", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r168925630", + "type": "rectangle", + "x": 900, + "y": 480, + "width": 250, + "height": 56, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 900646667, + "versionNonce": 397386988, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1714993724", + "type": "text", + "x": 910, + "y": 488, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1020712380, + "versionNonce": 513800403, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "color here = which language tab", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "color here = which language tab", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t904808210", + "type": "text", + "x": 910, + "y": 508, + "width": 260.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 855683249, + "versionNonce": 1458459369, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "real tools often color per-module or random", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "real tools often color per-module or random", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1496362393", + "type": "rectangle", + "x": 40, + "y": 400, + "width": 1100, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 367896032, + "versionNonce": 507166890, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t740374388", + "type": "text", + "x": 50, + "y": 408, + "width": 492.80000000000007, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1191169823, + "versionNonce": 644425156, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "_start / runtime.main (100% \u2014 every sample passes through here)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "_start / runtime.main (100% \u2014 every sample passes through here)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r893012982", + "type": "rectangle", + "x": 40, + "y": 340, + "width": 1080, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 979608840, + "versionNonce": 1978541002, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t698908698", + "type": "text", + "x": 50, + "y": 348, + "width": 184.8, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1857831139, + "versionNonce": 1420416034, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main / main.run (99.9%)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main / main.run (99.9%)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1251401244", + "type": "rectangle", + "x": 40, + "y": 280, + "width": 18, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1603024662, + "versionNonce": 1725790034, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t572025249", + "type": "text", + "x": 50, + "y": 288, + "width": 40, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 687901196, + "versionNonce": 657026946, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r1710173968", + "type": "rectangle", + "x": 70, + "y": 280, + "width": 1030, + "height": 50, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 766127377, + "versionNonce": 543470318, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1240652776", + "type": "text", + "x": 80, + "y": 288, + "width": 446.6, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 612610173, + "versionNonce": 461328454, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "spin_hot / hotSpin (99.2% \u2014 the wide box IS the hot path)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "spin_hot / hotSpin (99.2% \u2014 the wide box IS the hot path)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r192753395", + "type": "rectangle", + "x": 70, + "y": 220, + "width": 1030, + "height": 50, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 474037466, + "versionNonce": 1459942077, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1890608095", + "type": "text", + "x": 80, + "y": 228, + "width": 523.6, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 265376439, + "versionNonce": 823971518, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inner divisor loop (i % d == 0) \u2014 inlined, same width as its caller", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inner divisor loop (i % d == 0) \u2014 inlined, same width as its caller", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r789012829", + "type": "rectangle", + "x": 70, + "y": 160, + "width": 140, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1855115536, + "versionNonce": 1036182386, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1398915770", + "type": "text", + "x": 80, + "y": 168, + "width": 138.60000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1464231097, + "versionNonce": 443341425, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "leaf @ sample tick", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "leaf @ sample tick", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "r581788511", + "type": "rectangle", + "x": 230, + "y": 160, + "width": 870, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1498985293, + "versionNonce": 202243535, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t321971763", + "type": "text", + "x": 240, + "y": 168, + "width": 585.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 372002333, + "versionNonce": 1514239051, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "(rest of spin_hot's samples, other instants \u2014 same function, same box below)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "(rest of spin_hot's samples, other instants \u2014 same function, same box below)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "a961835177", + "type": "arrow", + "x": 155, + "y": 156, + "width": 0, + "height": 56, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 591920451, + "versionNonce": 1100165600, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + -56 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1845385661", + "type": "text", + "x": 155.0, + "y": 114.0, + "width": 381.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1502683285, + "versionNonce": 1805024348, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "read UP one column: this is the call stack open at that instant", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "read UP one column: this is the call stack open at that instant", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "t1249730954", + "type": "text", + "x": 590, + "y": 15, + "width": 280.5, + "height": 18.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1315039622, + "versionNonce": 1519529216, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Figure 31.1 \u2014 reading a flamegraph", + "fontSize": 15, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Figure 31.1 \u2014 reading a flamegraph", + "lineHeight": 1.25, + "baseline": 15 + }, + { + "id": "t1726708790", + "type": "text", + "x": 40, + "y": 586, + "width": 772.2, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1606847653, + "versionNonce": 1497303463, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Finding the hot path: scan for the widest unbroken column reaching toward the top \u2014 that name is where the cycles go.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Finding the hot path: scan for the widest unbroken column reaching toward the top \u2014 that name is where the cycles go.", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/31-flamegraph-reading.svg b/assets/diagrams/31-flamegraph-reading.svg new file mode 100644 index 0000000..2006923 --- /dev/null +++ b/assets/diagrams/31-flamegraph-reading.svg @@ -0,0 +1,36 @@ + + + +one flamegraph: each box is a stack frame sampled by perf + +y = stack depth +bottom: root top: leaf (running now) + +x = share of samples +NOT time order or call order + +widest box = the hot function +spin_hot / hotSpin at ~99% of samples + +color here = which language tab +real tools often color per-module or random + +_start / runtime.main (100% — every sample passes through here) + +main / main.run (99.9%) + + + +spin_hot / hotSpin (99.2% — the wide box IS the hot path) + +inner divisor loop (i % d == 0) — inlined, same width as its caller + +leaf @ sample tick + +(rest of spin_hot's samples, other instants — same function, same box below) + +read UP one column: this is the call stack open at that instant +read UP one column: this is the call stack open at that instant +Figure 31.1 — reading a flamegraph +Finding the hot path: scan for the widest unbroken column reaching toward the top — that name is where the cycles go. + \ No newline at end of file diff --git a/assets/diagrams/31-toolbelt-map.excalidraw b/assets/diagrams/31-toolbelt-map.excalidraw new file mode 100644 index 0000000..6d00ca3 --- /dev/null +++ b/assets/diagrams/31-toolbelt-map.excalidraw @@ -0,0 +1,2003 @@ +{ + "type": "excalidraw", + "version": 2, + "source": "https://excalidraw.com", + "elements": [ + { + "id": "r1255782213", + "type": "rectangle", + "x": 20, + "y": 30, + "width": 360, + "height": 640, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#eef4fb", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 299209805, + "versionNonce": 1248040726, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1217288629", + "type": "text", + "x": 32, + "y": 36, + "width": 40, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 999150578, + "versionNonce": 1448702416, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "C++", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "C++", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r94207849", + "type": "rectangle", + "x": 410, + "y": 30, + "width": 360, + "height": 640, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fafafa", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 989188769, + "versionNonce": 1658585768, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1871389426", + "type": "text", + "x": 422, + "y": 36, + "width": 40, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1061529758, + "versionNonce": 1292326757, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Go", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Go", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1464988738", + "type": "rectangle", + "x": 800, + "y": 30, + "width": 360, + "height": 640, + "angle": 0, + "strokeColor": "#dddddd", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1919667749, + "versionNonce": 694938205, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t560172677", + "type": "text", + "x": 812, + "y": 36, + "width": 40, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 375558258, + "versionNonce": 1504557779, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Rust", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Rust", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "r1885004040", + "type": "rectangle", + "x": 415, + "y": 40, + "width": 350, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#111111", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1511981199, + "versionNonce": 18689664, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t485361471", + "type": "text", + "x": 425, + "y": 48, + "width": 231.00000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#ffffff", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1353451830, + "versionNonce": 1259625771, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "shared target: app ", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "shared target: app ", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1228160840", + "type": "text", + "x": 425, + "y": 68, + "width": 229.9, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1680486603, + "versionNonce": 1505180445, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "byte-identical result across all three", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "byte-identical result across all three", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1814982090", + "type": "rectangle", + "x": 40, + "y": 130, + "width": 320, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1582530477, + "versionNonce": 68702765, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t334328421", + "type": "text", + "x": 50, + "y": 138, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1735430819, + "versionNonce": 1932355256, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "perf record --call-graph fp", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "perf record --call-graph fp", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1043328514", + "type": "text", + "x": 50, + "y": 158, + "width": 254.10000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 352373036, + "versionNonce": 1180762392, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "external sampler, any ELF \u2014 spin_hot 99.9%", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "external sampler, any ELF \u2014 spin_hot 99.9%", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r675689995", + "type": "rectangle", + "x": 40, + "y": 210, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1352876214, + "versionNonce": 425154206, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t105335509", + "type": "text", + "x": 50, + "y": 218, + "width": 238.70000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 869257936, + "versionNonce": 599726105, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "perf report --stdio / perf stat", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "perf report --stdio / perf stat", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1232264494", + "type": "text", + "x": 50, + "y": 238, + "width": 235.95000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1036544807, + "versionNonce": 1582682590, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cycles, instructions, flamegraph source", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cycles, instructions, flamegraph source", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r48380440", + "type": "rectangle", + "x": 40, + "y": 280, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1800963072, + "versionNonce": 1969397479, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t948376426", + "type": "text", + "x": 50, + "y": 288, + "width": 77.0, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 344163389, + "versionNonce": 1402985095, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "clang-tidy", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "clang-tidy", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t527840978", + "type": "text", + "x": 50, + "y": 308, + "width": 242.00000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1101151767, + "versionNonce": 1106675256, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "bugprone-/modernize-/performance- checks", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "bugprone-/modernize-/performance- checks", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r478368794", + "type": "rectangle", + "x": 40, + "y": 340, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1918445969, + "versionNonce": 438480475, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t21590050", + "type": "text", + "x": 50, + "y": 348, + "width": 107.80000000000001, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1221008182, + "versionNonce": 1707881236, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "GCC -fanalyzer", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "GCC -fanalyzer", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1896303051", + "type": "text", + "x": 50, + "y": 368, + "width": 260.15000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 483342178, + "versionNonce": 727248440, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "interprocedural static analysis, own preset", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "interprocedural static analysis, own preset", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r578382749", + "type": "rectangle", + "x": 40, + "y": 400, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 811877612, + "versionNonce": 744350717, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1555197836", + "type": "text", + "x": 50, + "y": 408, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1689055659, + "versionNonce": 1792880119, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdb (Chapter 28)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdb (Chapter 28)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t953462728", + "type": "text", + "x": 50, + "y": 428, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 774495091, + "versionNonce": 1507352761, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the same binary, post-mortem", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the same binary, post-mortem", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1911144787", + "type": "rectangle", + "x": 430, + "y": 130, + "width": 320, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1348380191, + "versionNonce": 1222085920, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t495562268", + "type": "text", + "x": 440, + "y": 138, + "width": 200.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1297084099, + "versionNonce": 1270143560, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "runtime/pprof (in-process)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "runtime/pprof (in-process)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1623459425", + "type": "text", + "x": 440, + "y": 158, + "width": 296.45000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 405730861, + "versionNonce": 610476173, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "--cpuprofile -> go tool pprof --top: main.hotSpin", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "--cpuprofile -> go tool pprof --top: main.hotSpin", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1357384417", + "type": "rectangle", + "x": 430, + "y": 210, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1782934036, + "versionNonce": 1343586716, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1012776494", + "type": "text", + "x": 440, + "y": 218, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1612576474, + "versionNonce": 713407463, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "--memprofile, --alloc_space", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "--memprofile, --alloc_space", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1922467326", + "type": "text", + "x": 440, + "y": 238, + "width": 266.20000000000005, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1298562884, + "versionNonce": 1688790696, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "main.allocChurn, since-start allocation view", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "main.allocChurn, since-start allocation view", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r554731821", + "type": "rectangle", + "x": 430, + "y": 280, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 150405162, + "versionNonce": 1926014396, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t33705765", + "type": "text", + "x": 440, + "y": 288, + "width": 223.3, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1222835033, + "versionNonce": 1733367633, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "runtime/trace + go tool trace", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "runtime/trace + go tool trace", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t83436325", + "type": "text", + "x": 440, + "y": 308, + "width": 193.60000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1341471512, + "versionNonce": 406628599, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "per-goroutine execution timeline", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "per-goroutine execution timeline", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1538528905", + "type": "rectangle", + "x": 430, + "y": 340, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 465764505, + "versionNonce": 556460872, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1564642824", + "type": "text", + "x": 440, + "y": 348, + "width": 84.7, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1552886623, + "versionNonce": 298568521, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "delve (dlv)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "delve (dlv)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t127280096", + "type": "text", + "x": 440, + "y": 368, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 537175528, + "versionNonce": 1395090194, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "scripted breakpoint on main.hotSpin", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "scripted breakpoint on main.hotSpin", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1632779374", + "type": "rectangle", + "x": 430, + "y": 400, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1569471747, + "versionNonce": 415369833, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1800456197", + "type": "text", + "x": 440, + "y": 408, + "width": 46.2, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1281040152, + "versionNonce": 1932556429, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "go vet", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "go vet", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t300229610", + "type": "text", + "x": 440, + "y": 428, + "width": 151.25, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1537239168, + "versionNonce": 1880858327, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "static correctness checks", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "static correctness checks", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r1535264532", + "type": "rectangle", + "x": 820, + "y": 130, + "width": 320, + "height": 60, + "angle": 0, + "strokeColor": "#e8870c", + "backgroundColor": "#fff8ef", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1557520396, + "versionNonce": 1156218669, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1765188635", + "type": "text", + "x": 830, + "y": 138, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 275986453, + "versionNonce": 434035976, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "perf record --call-graph fp", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "perf record --call-graph fp", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t470777069", + "type": "text", + "x": 830, + "y": 158, + "width": 290.40000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 35861134, + "versionNonce": 318881105, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "external sampler again \u2014 spin_hot inlined, ~100%", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "external sampler again \u2014 spin_hot inlined, ~100%", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r734756497", + "type": "rectangle", + "x": 820, + "y": 210, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 1254044374, + "versionNonce": 1626801286, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t319009674", + "type": "text", + "x": 830, + "y": 218, + "width": 123.20000000000002, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 778951647, + "versionNonce": 1675269316, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cargo flamegraph", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cargo flamegraph", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1605118055", + "type": "text", + "x": 830, + "y": 238, + "width": 284.35, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1911677823, + "versionNonce": 1255821460, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "inferno collapse+render, real SVG with spin_hot", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "inferno collapse+render, real SVG with spin_hot", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r32505096", + "type": "rectangle", + "x": 820, + "y": 280, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 880211325, + "versionNonce": 1143080092, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t357762969", + "type": "text", + "x": 830, + "y": 288, + "width": 246.40000000000003, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1949859800, + "versionNonce": 1747193439, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "cargo clippy / cargo fmt --check", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "cargo clippy / cargo fmt --check", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t653882997", + "type": "text", + "x": 830, + "y": 308, + "width": 211.75000000000003, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 795835368, + "versionNonce": 519045606, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "lints + style, clean on this source", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "lints + style, clean on this source", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r405267489", + "type": "rectangle", + "x": 820, + "y": 340, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 302225269, + "versionNonce": 995497776, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1785464365", + "type": "text", + "x": 830, + "y": 348, + "width": 269.5, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1461771901, + "versionNonce": 763005399, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "tokio-console (forward ref, Ch. 27)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "tokio-console (forward ref, Ch. 27)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1180129360", + "type": "text", + "x": 830, + "y": 368, + "width": 223.85000000000002, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 678560312, + "versionNonce": 698592396, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "not applicable: no async runtime here", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "not applicable: no async runtime here", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "r808754739", + "type": "rectangle", + "x": 820, + "y": 400, + "width": 320, + "height": 50, + "angle": 0, + "strokeColor": "#999999", + "backgroundColor": "#ffffff", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "dashed", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 3 + }, + "seed": 774009776, + "versionNonce": 515726222, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false + }, + { + "id": "t1558068003", + "type": "text", + "x": 830, + "y": 408, + "width": 207.9, + "height": 17.5, + "angle": 0, + "strokeColor": "#111111", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1561172281, + "versionNonce": 1873041767, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "gdb / rust-gdb (Chapter 28)", + "fontSize": 14, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "gdb / rust-gdb (Chapter 28)", + "lineHeight": 1.25, + "baseline": 14 + }, + { + "id": "t1274216371", + "type": "text", + "x": 830, + "y": 428, + "width": 169.4, + "height": 13.75, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 102859351, + "versionNonce": 815299309, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "the same binary, post-mortem", + "fontSize": 11, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "the same binary, post-mortem", + "lineHeight": 1.25, + "baseline": 11 + }, + { + "id": "a966953282", + "type": "arrow", + "x": 200, + "y": 90, + "width": 0, + "height": 38, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 1342801637, + "versionNonce": 1049183265, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 38 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a415682665", + "type": "arrow", + "x": 590, + "y": 90, + "width": 0, + "height": 38, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 925800911, + "versionNonce": 1295889530, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 38 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "a411666146", + "type": "arrow", + "x": 980, + "y": 90, + "width": 0, + "height": 38, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 2, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": { + "type": 2 + }, + "seed": 638796038, + "versionNonce": 1388092812, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "points": [ + [ + 0, + 0 + ], + [ + 0, + 38 + ] + ], + "lastCommittedPoint": null, + "startBinding": null, + "endBinding": null, + "startArrowhead": null, + "endArrowhead": "arrow" + }, + { + "id": "t1157283713", + "type": "text", + "x": 590, + "y": 12, + "width": 396.00000000000006, + "height": 18.75, + "angle": 0, + "strokeColor": "#b8650a", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1092557651, + "versionNonce": 1467863172, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Figure 31.2 \u2014 one target, three native toolbelts", + "fontSize": 15, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Figure 31.2 \u2014 one target, three native toolbelts", + "lineHeight": 1.25, + "baseline": 15 + }, + { + "id": "t1030153522", + "type": "text", + "x": 40, + "y": 460, + "width": 1181.4, + "height": 15.0, + "angle": 0, + "strokeColor": "#1d1d1d", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1022210251, + "versionNonce": 863227375, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Amber boxes are the CPU-profiling step this chapter cross-checks against each other: perf report / go tool pprof / perf report (or cargo flamegraph), all naming the same function.", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Amber boxes are the CPU-profiling step this chapter cross-checks against each other: perf report / go tool pprof / perf report (or cargo flamegraph), all naming the same function.", + "lineHeight": 1.25, + "baseline": 12 + }, + { + "id": "t421756848", + "type": "text", + "x": 40, + "y": 490, + "width": 1102.2, + "height": 15.0, + "angle": 0, + "strokeColor": "#555555", + "backgroundColor": "transparent", + "fillStyle": "solid", + "strokeWidth": 1, + "strokeStyle": "solid", + "roughness": 1, + "opacity": 100, + "groupIds": [], + "frameId": null, + "roundness": null, + "seed": 1417057689, + "versionNonce": 284448318, + "isDeleted": false, + "boundElements": [], + "updated": 1, + "link": null, + "locked": false, + "text": "Dashed boxes are noted but out of scope here: gdb belongs to Chapter 28, tokio-console to the async runtimes of Chapter 27 (this program has no async runtime to show).", + "fontSize": 12, + "fontFamily": 1, + "textAlign": "left", + "verticalAlign": "top", + "containerId": null, + "originalText": "Dashed boxes are noted but out of scope here: gdb belongs to Chapter 28, tokio-console to the async runtimes of Chapter 27 (this program has no async runtime to show).", + "lineHeight": 1.25, + "baseline": 12 + } + ], + "appState": { + "viewBackgroundColor": "#ffffff", + "gridSize": null + }, + "files": {} +} \ No newline at end of file diff --git a/assets/diagrams/31-toolbelt-map.svg b/assets/diagrams/31-toolbelt-map.svg new file mode 100644 index 0000000..f16a9df --- /dev/null +++ b/assets/diagrams/31-toolbelt-map.svg @@ -0,0 +1,63 @@ + + + +C++ + +Go + +Rust + +shared target: app <hot|alloc> +byte-identical result across all three + +perf record --call-graph fp +external sampler, any ELF — spin_hot 99.9% + +perf report --stdio / perf stat +cycles, instructions, flamegraph source + +clang-tidy +bugprone-/modernize-/performance- checks + +GCC -fanalyzer +interprocedural static analysis, own preset + +gdb (Chapter 28) +the same binary, post-mortem + +runtime/pprof (in-process) +--cpuprofile -> go tool pprof --top: main.hotSpin + +--memprofile, --alloc_space +main.allocChurn, since-start allocation view + +runtime/trace + go tool trace +per-goroutine execution timeline + +delve (dlv) +scripted breakpoint on main.hotSpin + +go vet +static correctness checks + +perf record --call-graph fp +external sampler again — spin_hot inlined, ~100% + +cargo flamegraph +inferno collapse+render, real SVG with spin_hot + +cargo clippy / cargo fmt --check +lints + style, clean on this source + +tokio-console (forward ref, Ch. 27) +not applicable: no async runtime here + +gdb / rust-gdb (Chapter 28) +the same binary, post-mortem + + + +Figure 31.2 — one target, three native toolbelts +Amber boxes are the CPU-profiling step this chapter cross-checks against each other: perf report / go tool pprof / perf report (or cargo flamegraph), all naming the same function. +Dashed boxes are noted but out of scope here: gdb belongs to Chapter 28, tokio-console to the async runtimes of Chapter 27 (this program has no async runtime to show). + \ No newline at end of file diff --git a/assets/diagrams/README.md b/assets/diagrams/README.md index 6e18c6b..abc8e57 100644 --- a/assets/diagrams/README.md +++ b/assets/diagrams/README.md @@ -37,6 +37,8 @@ here as you add them: | `20-ipc-matrix` | 20 | The coordination ladder as measured on this host — futex, eventfd, pipe, POSIX mq, UNIX socket, and the 1 ms poll loop, with bench p50/p99 and the voluntary context-switch evidence | | `13-pid-reuse-race-pidfd` | 13 | The five-step pid-reuse race timeline — spawn, crash, reap, recycle, kill(2) hitting a stranger — above the four-step pidfd lane that pins identity to an fd, annotated with the real fdinfo Pid: and strace evidence from pmon v2 | | `18-pipe-internals` | 18 | One live pipe as the kernel sees it — write end fd 1 and read end fd 5 around the 16-slot ring of pages, with the measured 65536 B capacity, the F_SETPIPE_SZ page rounding, and what PIPE_BUF atomicity does and does not promise | +| `28-remote-debug-topology` | 28 | The local core-dump path (app crash → systemd-coredump → coredumpctl → gdb on core.app, verified this session) beside the gdbserver/`target remote`/CLion remote path on systems-target, documented but not run in this lane | +| `28-stack-frame-anatomy` | 28 | The five real frames gdb's `bt full` names on `app`'s core — main → run_supervisor → handle_child → deref → std::println — beside the exact `.debug_line` addresses readelf/objdump report for lines 52 and 56 | | `18-log-topology` | 18 | pmon v4's full log pipeline — child stdio through two O_CLOEXEC pipes into the supervisor's poll loop and the framed log, then tail splicing file→FIFO to a detachable cat reader, annotated with the run's real fds, inodes, flags, and splice return values | | `25-futex-fast-slow-path` | 25 | pthread mutex as a futex — the uncontended fast path (userspace CAS 0→1, no syscall) beside the contended slow path (FUTEX_WAIT parks the loser on a kernel wait queue keyed by (mm, &word), FUTEX_WAKE releases it), annotated with the measured 5,186→488,818 futex-call and 455→123,986 voluntary-context-switch scaling | | `25-mpmc-queue` | 25 | workq's bounded MPMC job queue shared three ways — P producers push, one mutex + not_full/not_empty condvars (a buffered channel in Go) bound capacity K, C consumers pop into per-consumer accumulators XOR-folded after join into the P/C/cap-independent checksum 42b3746c6cee7465 | @@ -54,3 +56,9 @@ here as you add them: | `27-tokio-task-lifecycle` | 27 | tokio's task/Waker cycle for the Rust async engine — `tokio::spawn`ed `async fn` tasks polled by 16 `tokio-rt-worker` threads (= cores), `.await` returning `Poll::Pending` and registering a Waker with the mio reactor, `epoll_wait` calling that Waker on fd readiness to re-enqueue the task, and the re-poll to `Poll::Ready` — 20k tasks over 16 threads | | `22-connection-state-machine` | 22 | chatterd v1's epoll engine per-connection state machine — recv() drained to EAGAIN into inbuf, take_frame reassembling whole frames, broadcast appending to every conn's outbuf, and the send()/EAGAIN backpressure loop that arms EPOLLOUT and resumes flush() on the next epoll_wait wakeup | | `22-netpoller-vs-epoll` | 22 | the chapter's key comparison — C++/Rust one thread over a hand-written epoll interest list (4 epoll_wait calls, explicit outbuf+EPOLLOUT backpressure) beside Go's goroutine-per-conn with the runtime netpoller hidden underneath (165 edge-triggered epoll_pwait calls, ~7 OS threads, blocking-channel backpressure) | +| `31-flamegraph-reading` | 31 | a schematic flamegraph annotated with its two axes — x = share of samples (not time order), y = stack depth, bottom the root frame, top the leaf running at each sample tick — with the widest column labeled as spin_hot/hotSpin, the real hot path this chapter's three profilers agree on | +| `31-toolbelt-map` | 31 | one shared `app ` target feeding three native toolbelts — C++'s perf/clang-tidy/-fanalyzer, Go's runtime/pprof+go tool trace+delve, Rust's perf/cargo-flamegraph+clippy — with gdb (Ch. 28) and tokio-console (Ch. 27, not applicable here) noted as dashed, out-of-scope boxes | +| `29-tool-coverage-matrix` | 29 | the five bugfarm defects (leak, uninit, uaf, overflow, race) down the rows against C++/Go/Rust across the columns, each cell backed by a command run this session — valgrind memcheck and ASan both catching uaf/overflow, valgrind's 630-error confusion on the Go binary, Rust's borrow checker preventing four of five outright, and helgrind+TSan+`go -race` all catching the same race | +| `29-shadow-memory` | 29 | ASan's shadow-memory model — one shadow byte per 8 application bytes via `(app_addr >> 3) + offset` — with the real freed-block/redzone shadow bytes (`fa`/`fd`) this chapter's `uaf` run produced, and the measured 0.002s native vs 0.046s TSan vs 1.0s valgrind memcheck gap that the inline-check design explains | +| `30-ebpf-tooling-stack` | 30 | the layered eBPF observation stack on systems-target — kprobe/uprobe/tracepoint/perf_event/BTF at the bottom, bcc-tools/bpftrace/bpftool/c++filt above it (every box exercised live this session), SystemTap's kernel-module vs dyninst backends beside it, and the three target binaries' uprobe arg0 outcome (correct for C++/Rust's SysV ABI, wrong for Go's ABIInternal) | +| `30-uprobe-attachment` | 30 | one uprobe attach walked across all three binaries — C++'s and Rust's mangled `cmd_work` symbols demangled by `c++filt` (Itanium and Rust v0 both auto-detected) then attached by the mangled string, beside Go's plain unmangled `main.cmdWork` — with the real, measured RDI-vs-RAX argument-register mismatch that only the Go column hits | diff --git a/examples/28-gdb-and-remote-debugging/README.md b/examples/28-gdb-and-remote-debugging/README.md new file mode 100644 index 0000000..63a3f59 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/README.md @@ -0,0 +1,183 @@ +# 28-gdb-and-remote-debugging — bugfarm scenario 1: a crashing pmon + +Chapter 28 is the first of the "bugfarm" examples: `app` is deliberately +dual-use. `app run` is a normal, well-behaved program. `app crash` is a +debugging TARGET — it walks the identical three/four-frame call chain but +feeds it a value that was never filled in, and goes straight into the bug +class this chapter is about: reading through a pointer/reference/`Option` +that a concurrent-looking event handler assumed — wrongly — was already +populated. + +``` +app run # normal operation, exit 0 +app crash # deliberate crash, debugging target +``` + +## Behavior + +Both subcommands walk the same call chain: + +``` +main -> run_supervisor -> handle_child -> deref +``` + +`run_supervisor` fakes a supervisor that just reaped child pid 4242. In +`run` mode the reaper filled in the child's exit-status slot before +`handle_child` looks at it. In `crash` mode, `handle_child` runs as if it +fired *before* the reaper got there — the slot was never populated — which +is exactly the shape of race a real signal-driven supervisor (see chapter +12's `pmon`) can hit if a `SIGCHLD` notification and the code that consumes +it get out of step. + +- `app run` prints: + ``` + pmon: supervisor starting + pmon: child 4242 exited status=0 + pmon: supervisor exiting cleanly + ``` + and exits 0. +- `app crash` prints only `pmon: supervisor starting`, then hits the bug. + **What happens next is different in every language — that difference is + the point of this chapter:** + +| | mechanism | exit | core? | name the frame with | +|---|---|---|---|---| +| C++ | null pointer deref → **SIGSEGV** | 139 | yes (via systemd-coredump) | `gdb --batch -ex run -ex bt` | +| Go | nil pointer deref → runtime catches SIGSEGV, converts to **panic** | 2 | no | the goroutine trace in the panic output | +| Rust | `Option::unwrap()` on `None` → **panic** | 101 | no | `RUST_BACKTRACE=1` | + +- No/unknown subcommand prints a two-line usage block on stderr and exits 2 + (identical across all three). + +### Why C++ gets a core and Go/Rust don't + +C++ has no runtime standing between your bug and the kernel: a null-pointer +read simply generates a hardware page fault, the kernel delivers `SIGSEGV`, +and the default disposition for an unhandled `SIGSEGV` is "dump core, then +terminate." Go and Rust both interpose a runtime between your code and the +raw fault. Go's runtime installs its own `SIGSEGV` handler, recognizes the +fault came from ordinary code (not from inside the runtime itself), and +converts it into a normal — if unrecovered — panic; there is no +`sigaction(SA_RESETHAND)` back to the default disposition, so no core. +Rust's `.unwrap()` calls `panic!` directly: no signal is even involved, +just the ordinary unwind-and-abort (here, unwind-and-`exit(101)`) machinery. +`gdb` still attaches and runs both binaries fine, but there is no signal to +catch — asking gdb for a backtrace after the fact gets you nothing useful, +which is why the Go/Rust checks below read the runtime's own trace instead. + +One more thing worth noticing once you're inside gdb on the C++ build: the +`ShutdownGuard` RAII type in `cpp/src/main.cpp`, whose destructor prints the +"exiting cleanly" line, never fires on the crash path. RAII buys you +cleanup on every C++-level exit (`return`, an exception, `std::exit` via a +`terminate` handler) — it buys you nothing against a hardware fault, because +a signal doesn't unwind the stack. Frame `#0` in the release backtrace below +even lands a level *inside* `std::println`'s own formatting machinery, +rather than exactly on `*child.exit_slot`: at `-O2`, forming the reference +argument doesn't fault, only reading through it during formatting does. + +## Demo contract + +Each language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary (builds first if needed) +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh` instead of running locally. + +All three builds keep debug info in an otherwise optimized binary: the C++ +`release` CMake preset is `RelWithDebInfo` (`-O2 -g`), Go's toolchain always +emits DWARF, and Rust's `Cargo.toml` sets `[profile.release] debug = true`. +Every frame function (`deref`, `handle_child`, `run_supervisor`) is marked +noinline (`[[gnu::noinline]]` / `//go:noinline` / `#[inline(never)]`) so +optimization can't fold the call chain away — Rust's `handle_child` also +needs a `std::hint::black_box(())` after the call, or the optimizer turns it +into a tail call and erases its own frame from the backtrace. + +Try it: + +```sh +./demo.sh build +./demo.sh cpp run crash # exit 139, no output about "exiting cleanly" +./demo.sh go run crash # exit 2, a goroutine trace +RUST_BACKTRACE=1 ./demo.sh rust run crash # exit 101, a frame-named backtrace +``` + +## Reproducing the debugging session by hand + +```sh +cd cpp && cmake --preset release && cmake --build --preset release + +# a) it crashes, and drops a core (this host routes core_pattern through +# systemd-coredump, so "the core file" means coredumpctl's copy, not a +# bare `core` in the cwd): +ws=$(mktemp -d); cd "$ws" +( ulimit -c unlimited; /path/to/cpp/build/release/app crash & echo $! >pid; wait $! ) +coredumpctl info "$(cat pid)" # confirms SIGSEGV + a stored core +coredumpctl dump "$(cat pid)" -o core.app # extracts it as a real file + +# b) a scripted gdb session names the frame: +gdb --batch -ex 'set pagination off' -ex 'set debuginfod enabled off' \ + -ex run -ex bt --args /path/to/cpp/build/release/app crash +# ... +# Program received signal SIGSEGV, Segmentation fault. +# #1 (anonymous namespace)::deref (child=...) at src/main.cpp:52 +# #2 (anonymous namespace)::handle_child (child=...) at src/main.cpp:56 +# #3 (anonymous namespace)::run_supervisor (inject_bug=true) at src/main.cpp:65 +# #4 main (argc=..., argv=...) at src/main.cpp:104 +``` + +```sh +cd go && go build -o bin/app . +./bin/app crash +# panic: runtime error: invalid memory address or nil pointer dereference +# [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=...] +# goroutine 1 [running]: +# main.deref(...) +# main.handleChild(...) +# main.runSupervisor(0x1) +# main.main() +``` + +```sh +cd rust && cargo build --release +RUST_BACKTRACE=1 ./target/release/app crash +# thread 'main' panicked at src/main.rs:38:25: +# called `Option::unwrap()` on a `None` value +# ... +# 5: app::deref +# 6: app::handle_child +# 7: app::run_supervisor +# 8: app::main +``` + +## Verification + +`verify.lua` (run per language with `LSP_LANG` set) asserts: + +- `run` is identical across languages: exit 0, the three status lines. +- bare / unknown subcommand: exit 2, usage block — identical across languages. +- `crash`, per language: + - **cpp** — exit 139 (SIGSEGV); the RAII cleanup line never appears; under + `ulimit -c unlimited` in a scratch temp dir, `coredumpctl` finds and + extracts a core for the crashed pid; a scripted + `gdb --batch -ex run -ex bt` names `deref`, `handle_child`, and + `run_supervisor` in the backtrace. + - **go** — exit 2 (not 139, no core); the panic message names the + standard nil-pointer-dereference text; the goroutine trace names + `main.deref`, `main.handleChild`, `main.runSupervisor`. + - **rust** — exit 101 (not 139, no core) either way; with + `RUST_BACKTRACE=1`, the backtrace names `app::deref`, + `app::handle_child`, `app::run_supervisor`; without it, those names are + absent (the backtrace is opt-in) but the exit code is unchanged. + +Run it standalone from this directory: + +```sh +LSP_LANG=cpp EXAMPLE_DIR="$(pwd)" REPO_ROOT=../.. lua verify.lua +``` + +or through the orchestrator: + +```sh +python3 ../../scripts/test-all-examples.py --only 28-gdb-and-remote-debugging +``` diff --git a/examples/28-gdb-and-remote-debugging/cpp/.clang-format b/examples/28-gdb-and-remote-debugging/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/28-gdb-and-remote-debugging/cpp/.clang-tidy b/examples/28-gdb-and-remote-debugging/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/28-gdb-and-remote-debugging/cpp/CMakeLists.txt b/examples/28-gdb-and-remote-debugging/cpp/CMakeLists.txt new file mode 100644 index 0000000..465b8f7 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(gdb_and_remote_debugging CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/28-gdb-and-remote-debugging/cpp/CMakePresets.json b/examples/28-gdb-and-remote-debugging/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/28-gdb-and-remote-debugging/cpp/demo.sh b/examples/28-gdb-and-remote-debugging/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp b/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp new file mode 100644 index 0000000..220f470 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/cpp/src/main.cpp @@ -0,0 +1,105 @@ +// app — bugfarm scenario 1: a crashing pmon-like supervisor +// (chapter 28: gdb and remote debugging). +// +// app run -- normal operation: reap a child, report its exit, exit 0 +// app crash -- the SAME call path, fed a child record whose exit-status +// slot was never populated (as if handle_child() ran before +// the reaper filled it in) -> a null pointer dereference +// three frames down: +// +// main -> run_supervisor -> handle_child -> deref +// +// This binary is deliberately dual-use: a normal program in "run" mode, a +// debugging TARGET in "crash" mode. It is built with debug info (the +// "release" CMake preset is RelWithDebInfo = -O2 -g) so gdb can name every +// frame even though this is not a debug build; each frame function is +// marked noinline so -O2 cannot fold the call chain away. +// +// One more thing to notice once you're inside gdb: ShutdownGuard below never +// gets to print its message on the crash path. SIGSEGV is not a C++ +// exception -- the kernel delivers a signal, the default disposition +// terminates the process immediately, and nothing unwinds the stack. RAII +// buys you cleanup on every *C++-level* exit path (return, throw, exit via +// exceptions); it buys you nothing against a hardware fault. + +#include +#include +#include +#include +#include +#include + +namespace { + +// Runs at scope exit only if control leaves run_supervisor the ordinary way. +class ShutdownGuard { + public: + ~ShutdownGuard() { std::println("pmon: supervisor exiting cleanly"); } +}; + +// A child the supervisor is reporting on. exit_slot is filled in by the +// reaper once the wait status is known; handle_child reads through it to +// print the status line. +struct ChildRecord { + int pid = 0; + const int* exit_slot = nullptr; +}; + +[[gnu::noinline]] void deref(const ChildRecord& child) { + // BUG: no null check before the read. In "crash" mode exit_slot is + // nullptr because handle_child ran before the reaper recorded the exit + // status -- exactly the kind of race a real supervisor can hit. + std::println("pmon: child {} exited status={}", child.pid, + *child.exit_slot); +} + +[[gnu::noinline]] void handle_child(const ChildRecord& child) { deref(child); } + +[[gnu::noinline]] int run_supervisor(bool inject_bug) { + std::println("pmon: supervisor starting"); + const int reaped_status = 0; + ChildRecord child{.pid = 4242, + .exit_slot = inject_bug ? nullptr : &reaped_status}; + + if (inject_bug) { + handle_child(child); // crash path: exit_slot stays null + return 0; // unreachable + } + ShutdownGuard guard; // only ever constructed on the clean path + handle_child(child); + return 0; +} + +enum class Mode { kRun, kCrash }; + +[[nodiscard]] std::expected +parse_mode(std::string_view sub) { + if (sub == "run") { + return Mode::kRun; + } + if (sub == "crash") { + return Mode::kCrash; + } + return std::unexpected("unknown subcommand: " + std::string(sub)); +} + +[[noreturn]] void usage() { + std::println(stderr, "usage: app run"); + std::println(stderr, " app crash"); + std::exit(2); +} + +} // namespace + +int main(int argc, char** argv) { + std::setvbuf(stdout, nullptr, _IOLBF, 0); // line-buffered even into a file + + if (argc != 2) { + usage(); + } + const auto mode = parse_mode(argv[1]); + if (!mode) { + usage(); + } + return run_supervisor(*mode == Mode::kCrash); +} diff --git a/examples/28-gdb-and-remote-debugging/demo.sh b/examples/28-gdb-and-remote-debugging/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/28-gdb-and-remote-debugging/go/demo.sh b/examples/28-gdb-and-remote-debugging/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/28-gdb-and-remote-debugging/go/go.mod b/examples/28-gdb-and-remote-debugging/go/go.mod new file mode 100644 index 0000000..91cc952 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/28-gdb-and-remote-debugging/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/28-gdb-and-remote-debugging/go/main.go b/examples/28-gdb-and-remote-debugging/go/main.go new file mode 100644 index 0000000..6e75820 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/go/main.go @@ -0,0 +1,83 @@ +// app -- bugfarm scenario 1: a crashing pmon-like supervisor +// (chapter 28: gdb and remote debugging). +// +// app run -- normal operation: reap a child, report its exit, exit 0 +// app crash -- the SAME call path, fed a child record whose exit-status +// pointer was never populated (as if handleChild ran before +// the reaper filled it in) -> a nil pointer dereference +// three frames down: +// +// main -> runSupervisor -> handleChild -> deref +// +// This binary is deliberately dual-use: a normal program in "run" mode, a +// debugging TARGET in "crash" mode. +// +// Go does not core-dump a nil dereference the way C++ does. The runtime +// itself catches the hardware SIGSEGV, converts it to a runtime.Error, and +// panics. Left unrecovered, that panic unwinds to main, prints +// "panic: runtime error: invalid memory address or nil pointer dereference" +// plus a full goroutine trace naming every frame below, and the process +// exits with status 2 -- not 139, and there is no core file for gdb to load. +// A goroutine trace (or dlv) is the tool for this crash, not gdb + core. +package main + +import ( + "fmt" + "os" +) + +// childRecord is a child the supervisor is reporting on. exitSlot is filled +// in by the reaper once the wait status is known; handleChild reads through +// it to print the status line. +type childRecord struct { + pid int + exitSlot *int +} + +//go:noinline +func deref(child childRecord) { + // BUG: no nil check before the read. In "crash" mode exitSlot is nil + // because handleChild ran before the reaper recorded the exit status -- + // exactly the kind of race a real supervisor can hit. + fmt.Printf("pmon: child %d exited status=%d\n", child.pid, *child.exitSlot) +} + +//go:noinline +func handleChild(child childRecord) { + deref(child) +} + +//go:noinline +func runSupervisor(injectBug bool) int { + fmt.Println("pmon: supervisor starting") + reapedStatus := 0 + child := childRecord{pid: 4242} + if injectBug { + handleChild(child) // crash path: exitSlot stays nil + return 0 // unreachable + } + child.exitSlot = &reapedStatus + handleChild(child) + fmt.Println("pmon: supervisor exiting cleanly") + return 0 +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: app run") + fmt.Fprintln(os.Stderr, " app crash") + os.Exit(2) +} + +func main() { + if len(os.Args) != 2 { + usage() + } + switch os.Args[1] { + case "run": + os.Exit(runSupervisor(false)) + case "crash": + os.Exit(runSupervisor(true)) + default: + usage() + } +} diff --git a/examples/28-gdb-and-remote-debugging/rust/Cargo.lock b/examples/28-gdb-and-remote-debugging/rust/Cargo.lock new file mode 100644 index 0000000..dabc337 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "gdb-and-remote-debugging" +version = "0.1.0" diff --git a/examples/28-gdb-and-remote-debugging/rust/Cargo.toml b/examples/28-gdb-and-remote-debugging/rust/Cargo.toml new file mode 100644 index 0000000..2e9eaea --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/rust/Cargo.toml @@ -0,0 +1,16 @@ +[package] +name = "gdb-and-remote-debugging" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] + +# Keep debug info in the release binary: gdb needs line/function info even +# though this is an optimized build, same as the CMake "release" preset +# (RelWithDebInfo) and `go build`'s always-on DWARF. +[profile.release] +debug = true diff --git a/examples/28-gdb-and-remote-debugging/rust/demo.sh b/examples/28-gdb-and-remote-debugging/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/28-gdb-and-remote-debugging/rust/rust-toolchain.toml b/examples/28-gdb-and-remote-debugging/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/28-gdb-and-remote-debugging/rust/src/main.rs b/examples/28-gdb-and-remote-debugging/rust/src/main.rs new file mode 100644 index 0000000..63adac0 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/rust/src/main.rs @@ -0,0 +1,104 @@ +//! app -- bugfarm scenario 1: a crashing pmon-like supervisor +//! (chapter 28: gdb and remote debugging). +//! +//! app run -- normal operation: reap a child, report its exit, exit 0 +//! app crash -- the SAME call path, fed a child record whose exit-status +//! slot was never populated (as if handle_child ran before +//! the reaper filled it in) -> Option::unwrap() on a None +//! three frames down: +//! +//! main -> run_supervisor -> handle_child -> deref +//! +//! This binary is deliberately dual-use: a normal program in "run" mode, a +//! debugging TARGET in "crash" mode. +//! +//! Rust's answer to the C++ null-pointer bug class: there is no raw pointer +//! to leave dangling, so the equivalent defect is an `Option` that should +//! have been filled in by now and was not. `.unwrap()` on that `None` panics +//! rather than segfaulting -- with `panic = "unwind"` (this crate's default) +//! gdb sees the unwinder's machinery, not a clean SIGSEGV, so +//! `RUST_BACKTRACE=1` is the idiomatic tool here, printing every frame below +//! by name. Each frame function below is `#[inline(never)]` so release +//! optimizations cannot fold the call chain away. + +#[derive(Clone, Copy)] +struct ChildRecord { + pid: u32, + exit_slot: Option, +} + +#[inline(never)] +fn deref(child: &ChildRecord) { + // BUG: unwrap() with no check. In "crash" mode exit_slot is None because + // handle_child ran before the reaper recorded the exit status -- exactly + // the kind of race a real supervisor can hit. + println!( + "pmon: child {} exited status={}", + child.pid, + child.exit_slot.unwrap() + ); +} + +#[inline(never)] +fn handle_child(child: &ChildRecord) { + deref(child); + // A no-op after the call: without it, `deref` is the last thing this + // function does and the optimizer turns the call into a tail jump, + // reusing this frame and erasing `handle_child` from the backtrace. + std::hint::black_box(()); +} + +#[inline(never)] +fn run_supervisor(inject_bug: bool) -> i32 { + println!("pmon: supervisor starting"); + let child = ChildRecord { + pid: 4242, + exit_slot: if inject_bug { None } else { Some(0) }, + }; + if inject_bug { + handle_child(&child); // crash path: exit_slot stays None + return 0; // unreachable + } + handle_child(&child); + println!("pmon: supervisor exiting cleanly"); + 0 +} + +enum Mode { + Run, + Crash, +} + +fn parse_mode(s: &str) -> Result { + match s { + "run" => Ok(Mode::Run), + "crash" => Ok(Mode::Crash), + other => Err(format!("unknown subcommand: {other}")), + } +} + +fn usage() -> ! { + eprintln!("usage: app run"); + eprintln!(" app crash"); + std::process::exit(2); +} + +/// Parses argv down to a `Mode` using `?`; any usage error is handled by the +/// caller, which prints the same two-line usage block as the other two +/// languages regardless of which parse step failed. +fn try_parse(args: &[String]) -> Result { + if args.len() != 2 { + return Err("wrong number of arguments".to_string()); + } + let mode = parse_mode(&args[1])?; + Ok(mode) +} + +fn main() { + let args: Vec = std::env::args().collect(); + let Ok(mode) = try_parse(&args) else { + usage(); + }; + let code = run_supervisor(matches!(mode, Mode::Crash)); + std::process::exit(code); +} diff --git a/examples/28-gdb-and-remote-debugging/verify.lua b/examples/28-gdb-and-remote-debugging/verify.lua new file mode 100644 index 0000000..a75ec78 --- /dev/null +++ b/examples/28-gdb-and-remote-debugging/verify.lua @@ -0,0 +1,167 @@ +-- Verify bugfarm scenario 1: `app run` is a normal, dual-use program; +-- `app crash` is a debugging TARGET whose call chain is identical across +-- languages (main -> run_supervisor -> handle_child -> deref) but whose +-- observable crash mechanism is NOT -- that difference is the chapter: +-- * C++: a real SIGSEGV, exit 139, a systemd-coredump core recoverable +-- with coredumpctl, and a scripted `gdb --batch -ex run -ex bt` backtrace +-- naming every frame. +-- * Go: the runtime turns the same class of bug into a panic, not a core. +-- Exit 2, and the goroutine trace (not gdb) names every frame. +-- * Rust: `.unwrap()` on a `None` panics too. Exit 101, and +-- RUST_BACKTRACE=1 (not gdb) names every frame. +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set. Plain Lua 5.4 (checks.lua has no deps). + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local bins = { + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +} +local bin = bins[lang] + +local example_dir = os.getenv("EXAMPLE_DIR") +if example_dir == nil or example_dir == "" then + example_dir = checks.run("pwd").out:gsub("%s+$", "") +end +local abs_bin = example_dir .. "/" .. bin + +-- Boolean assertion via the checks pattern API (same trick as ch12's verify). +local function expect_true(cond, label) + checks.expect_match(cond and "yes" or "no", "yes", label) +end + +-- Ensure the binary exists (the orchestrator builds before verifying; a +-- standalone run should fail loudly here instead of deep in a scenario). +local r = checks.run("test -x " .. bin) +checks.expect_exit(r, 0, lang .. ": built binary " .. bin .. " exists") + +-- 1. Usage: bare invocation and an unknown subcommand both exit 2 with the +-- identical two-line usage block, across all three languages. +r = checks.run("./" .. bin) +checks.expect_exit(r, 2, lang .. ": bare invocation exits 2") +checks.expect_match(r.out, "usage: app run", lang .. ": prints usage") +r = checks.run("./" .. bin .. " bogus") +checks.expect_exit(r, 2, lang .. ": unknown subcommand exits 2") + +-- 2. "run" is the normal program: identical dual-use output, exit 0. +r = checks.run("./" .. bin .. " run") +checks.expect_exit(r, 0, lang .. ": run exits 0") +checks.expect_match(r.out, "pmon: supervisor starting", lang .. ": run starts") +checks.expect_match(r.out, "pmon: child 4242 exited status=0", + lang .. ": run reports the reaped child") +checks.expect_match(r.out, "pmon: supervisor exiting cleanly", + lang .. ": run reaches its normal cleanup line") + +-- 3. "crash" -- each language's real crash story. +if lang == "cpp" then + -- 3a. A real SIGSEGV: exit 139, and the RAII ShutdownGuard's destructor + -- never runs (a signal is not stack unwinding). + r = checks.run("./" .. bin .. " crash") + checks.expect_exit(r, 139, lang .. ": crash dies to SIGSEGV (exit 139)") + checks.expect_match(r.out, "pmon: supervisor starting", + lang .. ": crash gets as far as starting") + expect_true(not r.out:find("supervisor exiting cleanly", 1, true), + lang .. ": RAII guard's destructor does not run on SIGSEGV") + + -- 3b. Core-enabling wrapper: ulimit -c unlimited, run in a temp dir, and + -- confirm a core was actually dropped. This host's core_pattern pipes + -- through systemd-coredump (`cat /proc/sys/kernel/core_pattern` + -- starts with '|'), so "drops a core file" means coredumpctl can find + -- and extract one into the temp dir -- not a bare `core` file + -- appearing there on its own. + local ws = checks.run("mktemp -d").out:gsub("%s+$", "") + assert(ws:match("^/"), "mktemp -d must yield an absolute path") + r = checks.run(string.format([[ + cd %s + ( ulimit -c unlimited; %s crash >/dev/null 2>&1 & echo $! >pidfile; wait $! ) + code=$? + pid=$(cat pidfile) + echo "child_exit=$code" + found=0 + for i in $(seq 1 100); do + if coredumpctl info "$pid" >/dev/null 2>&1; then found=1; break; fi + sleep 0.1 + done + echo "coredump_found=$found" + if [ "$found" = "1" ] && coredumpctl dump "$pid" -o core.app >/dev/null 2>&1 \ + && [ -s core.app ]; then + echo "core_file=yes" + else + echo "core_file=no" + fi + ]], ws, abs_bin)) + checks.expect_match(r.out, "child_exit=139", + lang .. ": crash under ulimit -c unlimited still dies to SIGSEGV") + checks.expect_match(r.out, "coredump_found=1", + lang .. ": systemd-coredump recorded the crash") + checks.expect_match(r.out, "core_file=yes", + lang .. ": a core file was dropped in the temp dir") + checks.run("rm -rf " .. ws) + + -- 3c. A scripted gdb session on the crash names the expected frames. + r = checks.run(string.format( + "gdb --batch -ex 'set pagination off' -ex 'set debuginfod enabled off' " .. + "-ex run -ex bt --args %s crash", abs_bin)) + checks.expect_match(r.out, "SIGSEGV", lang .. ": gdb reports the SIGSEGV") + checks.expect_match(r.out, "deref", lang .. ": gdb backtrace names deref") + checks.expect_match(r.out, "handle_child", + lang .. ": gdb backtrace names handle_child") + checks.expect_match(r.out, "run_supervisor", + lang .. ": gdb backtrace names run_supervisor") +elseif lang == "go" then + -- Go does not core the same way: the runtime catches the hardware SIGSEGV + -- itself and turns it into an ordinary (unrecovered) panic. Assert the + -- panic's goroutine trace names every frame instead of a gdb backtrace. + r = checks.run("./" .. bin .. " crash") + checks.expect_exit(r, 2, lang .. ": nil-deref panic exits 2 (not 139, no core)") + checks.expect_match(r.out, "pmon: supervisor starting", + lang .. ": crash gets as far as starting") + checks.expect_match(r.out, + "panic: runtime error: invalid memory address or nil pointer dereference", + lang .. ": panics with the standard nil-deref message") + checks.expect_match(r.out, "main%.deref", + lang .. ": goroutine trace names deref") + checks.expect_match(r.out, "main%.handleChild", + lang .. ": goroutine trace names handleChild") + checks.expect_match(r.out, "main%.runSupervisor", + lang .. ": goroutine trace names runSupervisor") + expect_true(not r.out:find("supervisor exiting cleanly", 1, true), + lang .. ": no code past the panic point runs") +elseif lang == "rust" then + -- Rust's equivalent bug class has no raw pointer to leave dangling: it is + -- an Option that should have been filled in and was not. unwrap() panics + -- (exit 101, no core); RUST_BACKTRACE=1 names every frame, opt-in. + r = checks.run("RUST_BACKTRACE=1 ./" .. bin .. " crash") + checks.expect_exit(r, 101, lang .. ": unwrap-on-None panics, exit 101 (not 139, no core)") + checks.expect_match(r.out, "pmon: supervisor starting", + lang .. ": crash gets as far as starting") + checks.expect_match(r.out, "panicked at src/main.rs", + lang .. ": panics with a source location") + checks.expect_match(r.out, "app::deref", lang .. ": backtrace names deref") + checks.expect_match(r.out, "app::handle_child", + lang .. ": backtrace names handle_child") + checks.expect_match(r.out, "app::run_supervisor", + lang .. ": backtrace names run_supervisor") + expect_true(not r.out:find("supervisor exiting cleanly", 1, true), + lang .. ": no code past the panic point runs") + + -- Without RUST_BACKTRACE, the same panic and exit code happen, but no + -- frame names are printed -- documents that the backtrace is opt-in. + r = checks.run("./" .. bin .. " crash") + checks.expect_exit(r, 101, + lang .. ": panic exit code is unchanged without a backtrace") + expect_true(not r.out:find("app::deref", 1, true), + lang .. ": RUST_BACKTRACE=1 is required to see frame names") +end + +checks.finish() diff --git a/examples/29-valgrind-sanitizers-miri/README.md b/examples/29-valgrind-sanitizers-miri/README.md new file mode 100644 index 0000000..2290d1a --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/README.md @@ -0,0 +1,177 @@ +# 29-valgrind-sanitizers-miri — bugfarm + +Chapter 29 is a menu of five seeded defects, one program each in C++, Go, +and Rust: + +``` +app +``` + +Each subcommand runs a small routine that exhibits exactly one memory or +concurrency bug — *where the language lets it*. This is not a program with +a normal happy path: the point of the chapter is that a plain run of most +of these subcommands looks fine (or crashes somewhere unhelpful, far from +the real bug) and it takes the right tool, pointed at the right binary, to +actually name the defect. + +## What's expressible in each language + +Memory safety isn't a spectrum here so much as a cliff: C++ has none of it +compiled in, Go trades manual memory management for a GC and runtime +bounds/nil checks, and Rust's borrow checker rejects three of these five +bugs before the program can exist at all. + +| bug | C++ | Go | Rust | +|---|---|---|---| +| `leak` | heap alloc, never freed | goroutine blocked forever (leaked) | `Box::leak` (safe API, opts out of reclamation) | +| `uaf` | use-after-free | — (GC won't free a live reference) | — (borrow checker) | +| `uninit` | read of indeterminate value | — (every value is zeroed) | — (no way to name an uninitialised value in safe code) | +| `overflow` | heap buffer overflow, one past the end | — (slice/array access panics, checked) | — (bounds-checked, or rejected at compile time for a fixed index) | +| `race` | data race on a shared int | data race on a shared int (allowed; GC keeps it memory-safe) | — (`Send`/`Sync` rules reject the unsynchronized sharing) | + +Where a bug isn't expressible, the subcommand still exists (so the CLI +shape is identical across all three) — it just prints a fixed note and +exits **64** (`EX_USAGE`) instead of attempting anything: + +- Go: `go: prevented by the runtime — see chapter` +- Rust: `rust: prevented at compile time — see chapter` + +That message *is* the finding for those cells, not a cop-out: it's the +same "no bug to reproduce" result you'd get chasing the equivalent C++ bug +in that language, just arrived at differently (a runtime check vs. a +rejected compilation) — and it's reproducible non-interactively, which is +this chapter's whole bar. + +Bad usage (no subcommand, or an unrecognized one) prints +`usage: app ` on stderr and exits 2, same +shape as every other chapter's `app`. + +## Catching each C++ bug: the tool table + +C++ is the only language that expresses all five, so it's the one where +"run it under the matching tool" actually means something for every +subcommand: + +| bug | tool | build | signature this chapter asserts | exit | +|---|---|---|---|---| +| `leak` | valgrind memcheck | plain (`build/release`) | `definitely lost` | 1 | +| `uninit` | valgrind memcheck | plain (`build/release`) | `Conditional jump or move depends on uninitialised value(s)` | 1 | +| `uaf` | ASan (+UBSan) | `cmake --preset asan` | `heap-use-after-free` | 1 | +| `overflow` | ASan (+UBSan) | `cmake --preset asan` | `heap-buffer-overflow` | 1 | +| `race` | TSan | `cmake --preset tsan` | `WARNING: ThreadSanitizer: data race` | 66 | + +The `asan` preset was already in this book's CMake template (`-O1 -g +-fno-omit-frame-pointer -fsanitize=address,undefined`); this chapter adds +a matching `tsan` preset. Neither is part of the `demo.sh` contract — +build them directly: + +```console +$ cd cpp +$ cmake --preset asan && cmake --build --preset asan +$ cmake --preset tsan && cmake --build --preset tsan +$ ./build/asan/app uaf +==12345==ERROR: AddressSanitizer: heap-use-after-free on address ... +$ ./build/tsan/app race +WARNING: ThreadSanitizer: data race (pid=...) +``` + +For Rust, valgrind confirms the one expressible bug (`leak`), and — where +available — miri adds a second, independent confirmation (see below). + +## A gotcha worth knowing: the optimizer can eat the bug + +Building `leak`/`uaf`/`uninit`/`overflow` at `-O2` with no barrier, GCC +proved on this host that the allocation was never otherwise observed and +deleted the whole thing — `malloc`, the write, and all — before valgrind +ever saw an allocation to lose. Same story in Rust release builds: a +`Box::leak`'d `Vec` whose only observable use is `.len()` (a +compile-time-known constant) can be fully constant-folded away, allocation +included. Every `do_*` function in `cpp/src/main.cpp` routes its pointer +through a `touch()` helper (`asm volatile("" : : "r,m"(value) : "memory")` +— the same trick as benchmark libraries' `DoNotOptimize`), and the Rust +`leak()` does the same with `std::hint::black_box`. Pull either barrier out +and re-run valgrind: the leak silently disappears. This is itself a small +lesson about why "it didn't crash" is not evidence of "there's no bug" — +the compiler is allowed to make dead code disappear, and an unobserved bug +is dead code. + +Separately: a plain (uninstrumented) run of C++ `overflow` can abort with +a glibc `Fatal glibc error: malloc.c:... assertion failed` — the +one-past-the-end write corrupts the next heap chunk's bookkeeping, and +`delete[]`'s consistency check catches the corruption far from where it +happened. That crash, when it happens, *is* the natural (if unhelpful) +consequence of the bug; it's why the ASan build exists. + +## miri: available on nightly, not on the pinned toolchain + +`rust-toolchain.toml` pins `1.97.1`, and `rustup component add miri +--toolchain 1.97.1` fails on this host (`component 'miri' ... is +unavailable for download for channel '1.97.1'`) — miri tracks nightly, not +stable. Where a `nightly` toolchain with the `miri` component happens to be +installed, `verify.lua` runs it as a **bonus, informational-only** check +(`cargo +nightly miri run -- leak`, expecting `memory leaked` in its +output) and prints what it found; where it isn't installed, it prints why +it's skipping and moves on. Neither path can fail the example — the +valgrind check above is what's actually asserted for Rust's `leak`. This +is exactly the "degrade gracefully" case: miri is a nice extra sample of +"a completely different technique (symbolic interpretation instead of +runtime instrumentation) finds the same bug," not a required tool for this +chapter. + +## Try it + +```console +$ ./demo.sh cpp run leak +bugfarm: leak: allocated 4096 bytes on the heap, never freed (intentional) +$ valgrind --leak-check=full --error-exitcode=1 ./cpp/build/release/app leak +==...== 4,096 bytes in 1 blocks are definitely lost in loss record 1 of 1 + +$ ./demo.sh go run leak +bugfarm: leak: goroutines before=1 after=2 leaked=1 + +$ ./demo.sh go run uaf +go: prevented by the runtime — see chapter # exit 64 + +$ ./demo.sh rust run leak +bugfarm: leak: leaked 65536 bytes via Box::leak, never reclaimed (intentional) +$ valgrind --leak-check=full --show-leak-kinds=all --errors-for-leak-kinds=all \ + --error-exitcode=1 ./rust/target/release/app leak +==...== 65,536 bytes in 1 blocks are definitely lost in loss record 2 of 2 + +$ ./demo.sh rust run race +rust: prevented at compile time — see chapter # exit 64 +``` + +(PIDs and valgrind's `==NNNNN==` prefixes vary by run; the signature +strings and exit codes are what's stable and what `verify.lua` checks.) + +## Demo contract + +Each language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh build` — build only (the plain, uninstrumented binary) +- `./demo.sh run [args]` — run the built binary +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh` instead of running locally. + +The top-level `demo.sh` dispatches: `./demo.sh cpp run leak` execs +`cpp/demo.sh run leak`. The binary is always named `app`. Because this +chapter's whole point is tool-assisted detection, the *instrumented* +builds (C++ ASan/TSan presets, the Go `-race` binary) live outside that +contract — see the tables above for how to build and run them directly. + +There's no sensible default action for a bare `./demo.sh` (no subcommand) +here — unlike most chapters, this one has no "just run it" happy path — so +running with no subcommand prints the usage line and exits 2. That's +intentional, not an oversight. + +## Verification + +`verify.lua` (run per language with `LSP_LANG` set) builds the plain +binary, checks the shared bad-usage/unknown-bug behavior, then — per +language — builds whatever instrumented variant that language's real bugs +need and asserts the detecting tool exits nonzero (66 for both race +detectors' default halt code, 1 otherwise) with the expected signature +string in its output, and that every "prevented" subcommand exits 64 with +its fixed message. Rust's miri check is informational only, per the +"degrade gracefully" rule for a pinned toolchain that doesn't ship miri. diff --git a/examples/29-valgrind-sanitizers-miri/cpp/.clang-format b/examples/29-valgrind-sanitizers-miri/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/29-valgrind-sanitizers-miri/cpp/.clang-tidy b/examples/29-valgrind-sanitizers-miri/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/29-valgrind-sanitizers-miri/cpp/CMakeLists.txt b/examples/29-valgrind-sanitizers-miri/cpp/CMakeLists.txt new file mode 100644 index 0000000..855ec3f --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(bugfarm CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/29-valgrind-sanitizers-miri/cpp/CMakePresets.json b/examples/29-valgrind-sanitizers-miri/cpp/CMakePresets.json new file mode 100644 index 0000000..3f7a1c4 --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/CMakePresets.json @@ -0,0 +1,73 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "tsan", + "inherits": "_base", + "displayName": "TSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=thread", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=thread" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "tsan", "configurePreset": "tsan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/29-valgrind-sanitizers-miri/cpp/demo.sh b/examples/29-valgrind-sanitizers-miri/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp b/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp new file mode 100644 index 0000000..b79f9c8 --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/cpp/src/main.cpp @@ -0,0 +1,153 @@ +// bugfarm — chapter 29: valgrind, sanitizers, and miri. +// +// app +// +// Each subcommand runs a tiny routine exhibiting exactly one seeded memory +// or concurrency defect. C++ gives no compile-time protection against any +// of the five, so this binary expresses all of them; run it under the +// matching tool to see each one caught: +// +// leak -> valgrind --leak-check=full ("definitely lost") +// uninit -> valgrind ("uninitialised value(s)") +// uaf -> ASan build ("heap-use-after-free") +// overflow -> ASan build ("heap-buffer-overflow") +// race -> TSan build ("data race") +// +// Plain (uninstrumented) runs are deliberately left un-hardened: leak and +// uninit stay silent, uaf usually "works" by accident, race just produces +// a wrong count -- and overflow can abort with a glibc heap-corruption +// message when its write clobbers the next chunk's bookkeeping. That +// unpredictability *is* the lesson: these bugs hide from a normal run and +// need a tool that watches memory, not eyeballs on stdout. + +#include +#include +#include +#include +#include +#include +#include + +namespace { + +enum class Bug { leak, uaf, uninit, overflow, race }; + +[[nodiscard]] std::expected parse_bug(std::string_view arg) { + if (arg == "leak") { + return Bug::leak; + } + if (arg == "uaf") { + return Bug::uaf; + } + if (arg == "uninit") { + return Bug::uninit; + } + if (arg == "overflow") { + return Bug::overflow; + } + if (arg == "race") { + return Bug::race; + } + return std::unexpected(arg); +} + +// Defeats the optimizer's ability to prove an allocation is never +// otherwise used and fold the whole thing away. Observed on this host at +// -O2 without this barrier: an unused `malloc`/`new` (and the memory +// pattern written into it) is deleted outright by GCC, taking the "bug" +// with it before valgrind ever sees an allocation. Same trick as +// benchmark libraries' DoNotOptimize. +template inline void touch(T const& value) { + asm volatile("" : : "r,m"(value) : "memory"); +} + +int counter = 0; // race: shared, unsynchronized on purpose + +void bump() { + for (int i = 0; i < 200'000; ++i) { + ++counter; + } +} + +int do_leak() { + int* buf = new int[1024]; + for (int i = 0; i < 1024; ++i) { + buf[i] = i; + } + touch(buf); // keep the allocation "used"; never delete[] it + std::println("bugfarm: leak: allocated {} bytes on the heap, never freed (intentional)", + 1024 * sizeof(int)); + return 0; +} + +int do_uaf() { + int* p = new int(42); + delete p; + touch(p); + *p = 7; // heap-use-after-free: write through a dangling pointer + std::println("bugfarm: uaf: wrote through a freed pointer, read back {}", *p); + return 0; +} + +int do_uninit() { + int* p = new int; // `new int` with no initializer: value is indeterminate + touch(p); + if (*p == 0x1234) { + std::println("bugfarm: uninit: buf had 0x1234 (unlikely)"); + } else { + std::println("bugfarm: uninit: read of an uninitialised int: {}", *p); + } + delete p; + return 0; +} + +int do_overflow() { + int* arr = new int[10]; + for (int i = 0; i < 10; ++i) { + arr[i] = i; + } + touch(arr); + arr[10] = 99; // heap-buffer-overflow: one past the allocated end + std::println("bugfarm: overflow: wrote arr[10]={} (one past the allocated end)", arr[10]); + delete[] arr; + return 0; +} + +int do_race() { + { + std::jthread t1(bump); // RAII joins both on scope exit -- lifetime + std::jthread t2(bump); // is safe; the *data* race on `counter` is not + } + std::println("bugfarm: race: counter={} (expected 400000; a wrong value is the benign symptom)", + counter); + return 0; +} + +constexpr std::string_view kUsage = "usage: app "; + +} // namespace + +int main(int argc, char** argv) { + if (argc != 2) { + std::println(stderr, "{}", kUsage); + return 2; + } + const auto bug = parse_bug(argv[1]); + if (!bug) { + std::println(stderr, "app: unknown bug: {}\n{}", bug.error(), kUsage); + return 2; + } + switch (*bug) { + case Bug::leak: + return do_leak(); + case Bug::uaf: + return do_uaf(); + case Bug::uninit: + return do_uninit(); + case Bug::overflow: + return do_overflow(); + case Bug::race: + return do_race(); + } + std::unreachable(); +} diff --git a/examples/29-valgrind-sanitizers-miri/demo.sh b/examples/29-valgrind-sanitizers-miri/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/29-valgrind-sanitizers-miri/go/demo.sh b/examples/29-valgrind-sanitizers-miri/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/29-valgrind-sanitizers-miri/go/go.mod b/examples/29-valgrind-sanitizers-miri/go/go.mod new file mode 100644 index 0000000..680a9c5 --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/29-valgrind-sanitizers-miri/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/29-valgrind-sanitizers-miri/go/main.go b/examples/29-valgrind-sanitizers-miri/go/main.go new file mode 100644 index 0000000..595873a --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/go/main.go @@ -0,0 +1,107 @@ +// bugfarm — chapter 29: valgrind, sanitizers, and miri. +// +// app +// +// Go is memory-safe by construction: there's no dangling pointer to chase +// after free (the GC never reclaims memory a live reference still points +// at), no read of uninitialised memory (every value starts zeroed), and no +// out-of-bounds write that silently corrupts a neighbor (a slice/array +// access past its bound panics immediately -- a runtime check, not +// something a background tool has to catch after the fact). So uaf, +// uninit, and overflow print a "prevented by the runtime" note and exit +// 64: that absence of a bug *is* the language's story here. +// +// What Go can still get wrong: +// +// leak -- a goroutine that blocks forever. There's no valgrind +// equivalent watching the Go heap for this; the way you notice +// is runtime.NumGoroutine() not going back down. +// race -- unsynchronized access to a shared variable from goroutines. +// Build with `go build -race` and the runtime's race detector +// prints "WARNING: DATA RACE" and the process exits 66. +package main + +import ( + "fmt" + "os" + "runtime" + "sync" + "time" +) + +const usage = "usage: app " +const prevented = "go: prevented by the runtime — see chapter" + +func parseBug(arg string) (string, error) { + switch arg { + case "leak", "uaf", "uninit", "overflow", "race": + return arg, nil + default: + return "", fmt.Errorf("unknown bug: %s", arg) + } +} + +// leak: spawns a goroutine that blocks forever on a channel receive nobody +// will ever send to. Nothing external is watching for this; we notice it +// the same way a real operator would, by the goroutine count never coming +// back down. +func leak() int { + before := runtime.NumGoroutine() + block := make(chan struct{}) + go func() { + <-block // never sent to: this goroutine (and its stack) leaks forever + }() + time.Sleep(50 * time.Millisecond) // let the scheduler actually start it + after := runtime.NumGoroutine() + leaked := after - before + fmt.Printf("bugfarm: leak: goroutines before=%d after=%d leaked=%d\n", before, after, leaked) + if leaked < 1 { + fmt.Fprintln(os.Stderr, "bugfarm: leak: expected goroutine leak was not observed") + return 1 + } + return 0 +} + +var counter int // race: shared, unsynchronized on purpose + +func bump(wg *sync.WaitGroup) { + defer wg.Done() + for i := 0; i < 100_000; i++ { + counter++ + } +} + +// race: two goroutines increment a shared int with no lock or atomic. Under +// a plain build this just risks a wrong final count; built with -race, the +// runtime instruments every access and reports the conflicting pair. +func race() int { + var wg sync.WaitGroup + wg.Add(2) + go bump(&wg) + go bump(&wg) + wg.Wait() + fmt.Printf("bugfarm: race: counter=%d (expected 200000; a wrong value is the benign symptom)\n", + counter) + return 0 +} + +func main() { + if len(os.Args) != 2 { + fmt.Fprintln(os.Stderr, usage) + os.Exit(2) + } + bug, err := parseBug(os.Args[1]) + if err != nil { + fmt.Fprintf(os.Stderr, "app: %v\n%s\n", err, usage) + os.Exit(2) + } + switch bug { + case "leak": + os.Exit(leak()) + case "race": + os.Exit(race()) + default: // uaf, uninit, overflow: not expressible in safe Go + fmt.Fprintln(os.Stderr, prevented) + os.Exit(64) + } +} diff --git a/examples/29-valgrind-sanitizers-miri/rust/Cargo.lock b/examples/29-valgrind-sanitizers-miri/rust/Cargo.lock new file mode 100644 index 0000000..d2db2be --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/rust/Cargo.lock @@ -0,0 +1,7 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "bugfarm" +version = "0.1.0" diff --git a/examples/29-valgrind-sanitizers-miri/rust/Cargo.toml b/examples/29-valgrind-sanitizers-miri/rust/Cargo.toml new file mode 100644 index 0000000..2fcab3b --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/rust/Cargo.toml @@ -0,0 +1,10 @@ +[package] +name = "bugfarm" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] diff --git a/examples/29-valgrind-sanitizers-miri/rust/demo.sh b/examples/29-valgrind-sanitizers-miri/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/29-valgrind-sanitizers-miri/rust/rust-toolchain.toml b/examples/29-valgrind-sanitizers-miri/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/29-valgrind-sanitizers-miri/rust/src/main.rs b/examples/29-valgrind-sanitizers-miri/rust/src/main.rs new file mode 100644 index 0000000..92f63ce --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/rust/src/main.rs @@ -0,0 +1,97 @@ +//! bugfarm — chapter 29: valgrind, sanitizers, and miri. +//! +//! app +//! +//! Safe Rust's compile-time guarantees rule out four of the five seeded +//! defects outright: the borrow checker won't let a use-after-free or a +//! data race compile, and there's no way to name an uninitialised value or +//! index past a slice's bound without either a panic (bounds check) or an +//! explicit `unsafe` block. So `uaf`, `uninit`, `overflow`, and `race` each +//! print a fixed note and exit 64 -- the compile-time rejection *is* the +//! finding, reproduced here as a documented outcome rather than a crash. +//! +//! What safe Rust can still do is leak memory on purpose: `Box::leak` +//! consumes a `Box` and hands back a `'static` reference, deliberately +//! opting the allocation out of Rust's normal drop-based reclamation. +//! Nothing frees it -- valgrind's memcheck reports the same "definitely +//! lost" block it would for an equivalent C/C++ leak. + +use std::env; +use std::hint::black_box; +use std::process::ExitCode; + +const USAGE: &str = "usage: app "; +const PREVENTED: &str = "rust: prevented at compile time — see chapter"; + +#[derive(Clone, Copy)] +enum Bug { + Leak, + Uaf, + Uninit, + Overflow, + Race, +} + +fn parse_bug(arg: &str) -> Result { + match arg { + "leak" => Ok(Bug::Leak), + "uaf" => Ok(Bug::Uaf), + "uninit" => Ok(Bug::Uninit), + "overflow" => Ok(Bug::Overflow), + "race" => Ok(Bug::Race), + other => Err(format!("unknown bug: {other}")), + } +} + +// leak: Box::leak is a safe API whose entire purpose is to opt an +// allocation out of automatic reclamation -- the program exits normally, +// but the 64 KiB Vec (and the Box that briefly held it) are never freed. +// +// black_box on the pointer is load-bearing, not decoration: without it, +// LLVM can see that nothing observable depends on the buffer's *contents* +// (only its statically-known length gets printed) and optimizes the +// entire allocation away, leak and all -- confirmed on this host by +// valgrind reporting zero bytes lost until the barrier was added. +fn leak() { + let leaked: &'static mut Vec = Box::leak(Box::new(vec![0xAAu8; 65536])); + leaked[0] = 0xAA; // touch it so the write isn't provably dead + black_box(leaked.as_ptr()); + println!( + "bugfarm: leak: leaked {} bytes via Box::leak, never reclaimed (intentional)", + leaked.len() + ); +} + +/// Returns `Ok(())` on the expressible bug (leak), or `Err(64)` for a bug +/// safe Rust prevents outright, or `Err(2)` for a usage error. +fn run(args: &[String]) -> Result<(), i32> { + if args.len() != 1 { + return Err(2); + } + let bug = parse_bug(&args[0]).map_err(|e| { + eprintln!("app: {e}"); + 2 + })?; + match bug { + Bug::Leak => { + leak(); + Ok(()) + } + Bug::Uaf | Bug::Uninit | Bug::Overflow | Bug::Race => { + eprintln!("{PREVENTED}"); + Err(64) + } + } +} + +fn main() -> ExitCode { + let args: Vec = env::args().skip(1).collect(); + match run(&args) { + Ok(()) => ExitCode::SUCCESS, + Err(2) => { + eprintln!("{USAGE}"); + ExitCode::from(2) + } + Err(code) => ExitCode::from(code as u8), + } +} diff --git a/examples/29-valgrind-sanitizers-miri/verify.lua b/examples/29-valgrind-sanitizers-miri/verify.lua new file mode 100644 index 0000000..6abb734 --- /dev/null +++ b/examples/29-valgrind-sanitizers-miri/verify.lua @@ -0,0 +1,172 @@ +-- Verify bugfarm (chapter 29: valgrind, sanitizers, and miri). +-- +-- "app " runs one seeded defect. This isn't +-- a normal program with an observable happy-path output -- the thing to +-- observe is a DETECTING TOOL catching the defect, so most of this script +-- builds an instrumented variant and runs the tool directly against the +-- binary (not through demo.sh, which only knows the plain build). +-- +-- Per language, what's actually expressible differs on purpose: +-- cpp -- all five: valgrind (leak, uninit), ASan (uaf, overflow), +-- TSan (race). +-- go -- leak (self-detected via runtime.NumGoroutine(), no external +-- tool watches Go goroutine leaks the way valgrind watches +-- malloc) and race (-race build, "DATA RACE", exit 66). uaf/ +-- uninit/overflow are not expressible in safe Go: bounds/nil +-- checks and the GC rule them out at runtime, so those exit 64 +-- with a fixed note. +-- rust -- leak only (Box::leak, a real unfreed allocation valgrind +-- reports as "definitely lost"; miri additionally confirms it +-- when available -- informational, since the pinned 1.97.1 +-- toolchain does not ship miri). uaf/uninit/overflow/race are +-- rejected by the borrow checker / bounds checks at compile +-- time or by construction, so those exit 64 with a fixed note. +-- +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR/REPO_ROOT set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local USAGE = "usage: app " + +-- Defensive: the CI harness always runs "./demo.sh LANG build" before this +-- script, but make verify.lua correct when run standalone too. +local r = checks.run("./demo.sh " .. lang .. " build") +checks.expect_exit(r, 0, lang .. ": plain build compiles") + +-- Bad usage, identical wording across all three languages. +r = checks.run("./demo.sh " .. lang .. " run") +checks.expect_exit(r, 2, lang .. ": no subcommand exits 2") +checks.expect_match(r.out, USAGE, lang .. ": no subcommand prints usage") + +r = checks.run("./demo.sh " .. lang .. " run bogus") +checks.expect_exit(r, 2, lang .. ": unknown bug exits 2") +checks.expect_match(r.out, "unknown bug: bogus", lang .. ": unknown bug diagnostic") + +-- A "prevented" subcommand: language runtime/compiler rules the bug out, +-- so it must print the fixed note and exit 64 (sysexits EX_USAGE) rather +-- than attempt anything. +local function expect_prevented(bug, phrase) + local pr = checks.run("./demo.sh " .. lang .. " run " .. bug) + checks.expect_exit(pr, 64, lang .. ": " .. bug .. " prevented exits 64") + checks.expect_match(pr.out, phrase, lang .. ": " .. bug .. " prints '" .. phrase .. "'") +end + +if lang == "cpp" then + -- C++ expresses all five defects; every one needs a tool that watches + -- memory or threads, not a plain run, to produce its finding. + + -- ASan+UBSan and TSan builds aren't part of the plain demo.sh contract + -- (build/run/TARGET deploy) -- build them directly via the CMake + -- presets that ship alongside the release preset. + r = checks.run("cd cpp && cmake --preset asan >/dev/null && cmake --build --preset asan >/dev/null") + checks.expect_exit(r, 0, "cpp: ASan+UBSan build configures and compiles") + + r = checks.run("cd cpp && cmake --preset tsan >/dev/null && cmake --build --preset tsan >/dev/null") + checks.expect_exit(r, 0, "cpp: TSan build configures and compiles") + + -- leak -> valgrind memcheck, "definitely lost". + r = checks.run("valgrind --leak-check=full --error-exitcode=1 ./cpp/build/release/app leak") + checks.expect_exit(r, 1, "cpp: valgrind flags the leak (nonzero exit)") + checks.expect_match(r.out, "definitely lost", "cpp: valgrind report says 'definitely lost'") + + -- uninit -> valgrind memcheck, "uninitialised value(s)". + r = checks.run("valgrind --error-exitcode=1 ./cpp/build/release/app uninit") + checks.expect_exit(r, 1, "cpp: valgrind flags the uninitialised read (nonzero exit)") + checks.expect_match(r.out, "ninitialised value", "cpp: valgrind report cites an uninitialised value") + + -- uaf -> ASan, "heap-use-after-free". + r = checks.run("./cpp/build/asan/app uaf") + checks.expect_exit(r, 1, "cpp: ASan build exits nonzero on uaf") + checks.expect_match(r.out, "heap%-use%-after%-free", "cpp: ASan report says 'heap-use-after-free'") + + -- overflow -> ASan, "heap-buffer-overflow". + r = checks.run("./cpp/build/asan/app overflow") + checks.expect_exit(r, 1, "cpp: ASan build exits nonzero on overflow") + checks.expect_match(r.out, "heap%-buffer%-overflow", "cpp: ASan report says 'heap-buffer-overflow'") + + -- race -> TSan, "data race", exit 66 (TSan's default halt_on_error exitcode). + r = checks.run("./cpp/build/tsan/app race") + checks.expect_exit(r, 66, "cpp: TSan build exits 66 on race") + checks.expect_match(r.out, "[Dd]ata race", "cpp: TSan report says 'data race'") + +elseif lang == "go" then + -- leak: self-detected. There is no valgrind-for-goroutines here; the + -- program itself compares runtime.NumGoroutine() before/after and + -- reports the delta, which is the idiomatic way this gets noticed. + r = checks.run("./demo.sh go run leak") + checks.expect_exit(r, 0, "go: leak run exits 0") + checks.expect_match(r.out, "leaked=%d+", "go: leak report has a leaked= field") + local leaked = tonumber(r.out:match("leaked=(%d+)")) + if leaked and leaked >= 1 then + checks.expect_match("ok", "ok", "go: at least one goroutine leaked (" .. leaked .. ")") + else + checks.expect_match(r.out, "leaked=[1-9]", "go: leaked count is >= 1") + end + + -- race: needs a -race build, which is not what the plain demo.sh build + -- produces for this chapter (unlike ch06, this chapter is not + -- exclusively about concurrency) -- build the instrumented binary + -- directly. + r = checks.run("cd go && go build -race -o bin/app-race .") + checks.expect_exit(r, 0, "go: race-instrumented build compiles") + + r = checks.run("./go/bin/app-race race") + checks.expect_exit(r, 66, "go: race build exits 66 (race detector default)") + checks.expect_match(r.out, "DATA RACE", "go: race detector report says 'DATA RACE'") + + -- uaf/uninit/overflow: not expressible in safe Go. + expect_prevented("uaf", "prevented by the runtime") + expect_prevented("uninit", "prevented by the runtime") + expect_prevented("overflow", "prevented by the runtime") + +else -- rust + -- leak: Box::leak is a safe API whose entire point is to opt an + -- allocation out of reclamation. --errors-for-leak-kinds=all makes + -- valgrind treat "still reachable" as an error too (the classification + -- a leaked-but-recently-live pointer gets depends on stray bit patterns + -- left on the stack, which is not something to pin a test on); the + -- byte count and "definitely lost" tag are what we actually assert on + -- this host. + r = checks.run( + "valgrind --leak-check=full --show-leak-kinds=all --errors-for-leak-kinds=all " .. + "--error-exitcode=1 ./rust/target/release/app leak") + checks.expect_exit(r, 1, "rust: valgrind flags the Box::leak allocation (nonzero exit)") + checks.expect_match(r.out, "definitely lost", "rust: valgrind report says 'definitely lost'") + checks.expect_match(r.out, "65,536 bytes", "rust: valgrind report names the leaked size") + + -- miri: informational only. The pinned toolchain (rust-toolchain.toml, + -- 1.97.1) does not ship a miri component; a nightly toolchain with miri + -- installed is a bonus double-check, never a requirement -- absence + -- must degrade gracefully, not fail the example. + local probe = checks.run("rustup which --toolchain nightly miri") + if probe.exit == 0 then + local mr = checks.run("cd rust && cargo +nightly miri run --quiet -- leak 2>&1") + if mr.out:find("memory leaked") then + print("info: rust: miri (nightly) also reports 'memory leaked' for the Box::leak allocation") + else + print("info: rust: miri ran on nightly but did not report the expected leak text (not asserted)") + end + else + print("info: rust: miri unavailable for the pinned 1.97.1 toolchain and no nightly+miri found " .. + "-- skipping the bonus check (valgrind above already covers the leak finding)") + end + + -- uaf/uninit/overflow/race: rejected by the compiler (or, for race, + -- have no data race to report because ownership rules prevent the + -- unsynchronized shared access in the first place). + expect_prevented("uaf", "prevented at compile time") + expect_prevented("uninit", "prevented at compile time") + expect_prevented("overflow", "prevented at compile time") + expect_prevented("race", "prevented at compile time") +end + +checks.finish() diff --git a/examples/30-ebpf-observation-toolkit/README.md b/examples/30-ebpf-observation-toolkit/README.md new file mode 100644 index 0000000..8236052 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/README.md @@ -0,0 +1,197 @@ +# 30 — eBPF observation toolkit + +**VM example.** This chapter is not about writing eBPF — it's about pointing +the standard eBPF *observation* tools (bcc-tools, bpftrace) at an ordinary +userspace program and reading what they say. `app work --seconds N` runs a +small, labeled workload; a driver script (`observe.sh`, running as root on the +lab VM) starts it in the background and then runs five separate tools against +it in turn, each proving it really saw an event our program produced: + +| Tool | What it watches | Bait in `app work` | +|---|---|---| +| `opensnoop` | `open(2)` calls | a fixed file, opened/written/closed every iteration | +| `execsnoop` | `exec(2)` calls | a short-lived `true` child, forked/exec'd every 4th iteration | +| `funccount` | per-symbol call counts via uprobe | the named hot function `busy_hash()`, called every iteration | +| `offcputime` | off-CPU time by stack | a ~230ms sleep between iterations | +| `bpftrace` | a one-line uprobe program | the same `busy_hash()`, counted with `@calls` | + +`busy_hash()` is deliberately kept unmangled and unstripped, and marked +noinline, in all three builds — cpp and rust emit the plain symbol +`busy_hash`, Go emits `main.busy_hash` — so `observe.sh` can target every +language with the *same* pattern: `:*busy_hash*`. + +All three languages produce byte-for-byte identical output, including the +final accumulated hash value (same seed, same algorithm, same iteration +count), so one `verify.lua` covers them. + +## The workload: `app work --seconds N` + +``` +work: start seconds=60 pid=13414 bait=/tmp/lsp-ebpf-work-bait.txt +work: exec i=0 status=0 +work: exec i=4 status=0 +... +work: done seconds=60 iters=260 opens=260 execs=65 busy_calls=260 busy_hash=12869313528293982360 +``` + +Each iteration (~230ms): open/write/close the bait file; every 4th iteration, +fork+exec `true` and wait for it; call `busy_hash()` once; sleep ~230ms. None +of this needs root — only the *observation* side does. + +| Language | How the loop runs | +|---|---| +| **C++23** | The loop body runs inside a `std::jthread` (takes a `std::stop_token`, ready for cooperative cancellation); `main()` joins it explicitly so the summary line always sees final counters. An RAII `Fd` class owns the bait file's descriptor. Every fallible syscall returns `std::expected`. Output goes through `std::println`. | +| **Go 1.26** | The loop runs in a goroutine; `main` blocks on a buffered result channel — the goroutine's "join". Every fallible call is wrapped with `fmt.Errorf("...: %w", err)`. | +| **Rust (edition 2024)** | Straight-line `Result`/`?` through the whole loop; the bait file's descriptor is a `rustix::fs` `OwnedFd`, closed by `Drop` — no explicit `close(2)`. | + +### A real optimizer gotcha found while building this + +The first version of this workload printed `busy_calls` but never the +accumulated hash value itself. At `-O2`, GCC's interprocedural analysis proved +`busy_hash()` had no observable side effects and that its final result was +never used anywhere — and deleted the *entire* call chain, `noinline` or not +(`noinline` only forbids inlining; it says nothing about dead-code +elimination of a provably-unused, provably-pure call). `funccount` and the +`bpftrace` uprobe both reported zero calls even though the binary clearly +still exported the `busy_hash` symbol and a plain `nm`/`objdump` looked +completely normal. The fix: print the final hash in the `done` summary line, +in all three languages. The moment the result is observable, the whole chain +is provably necessary again, and the uprobe fires. `busy_hash` is part of the +printed summary for that reason — it isn't an incidental extra field. + +## `observe.sh` — the actual demo + +`observe.sh [seconds]` runs **on the guest, as root** +(bcc-tools and bpftrace need `CAP_BPF`/`CAP_PERFMON`, which on this lab image +means root). It: + +1. starts ` work --seconds N` in the background, +2. runs `opensnoop -p PID`, `execsnoop -P PID`, `funccount -p PID -d N + ':*busy_hash*'`, `offcputime -p PID N`, and a `bpftrace` uprobe + one-liner against it in turn, each wrapped in its own `timeout`, +3. prints a `=== summary ===` block deriving a hit count per tool, +4. lets the workload finish (or stops it) before exiting. + +It deliberately does **not** use `set -e` — a bcc tool killed by its own +`timeout` wrapper (several of these tools don't reliably self-terminate on +their internal `-d`/duration flag; see below) is expected, not a driver +failure. Each step's output is checked on its own, at the end. + +### A real bcc-tools quirk this driver works around + +`opensnoop`'s own `-d SECONDS` flag did not reliably make it exit on its own +in testing — it consistently ran until an external `timeout` killed it, +regardless of the requested duration. Every invocation in `observe.sh` is +therefore wrapped in an outer `timeout`, and a `124` exit from that wrapper is +treated as expected, not a failure — only the captured *content* is asserted +on, never the tool's own exit code. + +## CLI + +``` +usage: app work --seconds N +``` + +Missing/invalid `--seconds`, or any subcommand other than `work`, prints the +usage line to stderr and exits `2`. + +## Demo contract + +Each language directory has the standard `demo.sh`: + +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary (`app`) locally +- With env `TARGET` set, `run` deploys to that lab VM via + `scripts/lab/deploy-to-vm.sh` and runs there instead (no root needed for + the workload itself) + +The top-level `./demo.sh [cpp|go|rust|all|build]` dispatches per language. +`observe.sh` is deliberately separate from this dispatcher — it's a custom +root-run driver script (starts the workload *and* orchestrates five different +tools against it), not a single binary+args invocation, so it doesn't fit the +`deploy-to-vm.sh` shape and is staged/run directly instead. + +## Try it (on the lab VM) + +```sh +export LIBVIRT_DEFAULT_URI=qemu:///system TARGET=systems-target +cd examples/30-ebpf-observation-toolkit + +# build and stage one language's binary, e.g. cpp +./cpp/demo.sh build +IP=$(../../scripts/lab/vm-ip.sh systems-target) +ssh "fedora@$IP" 'mkdir -p /tmp/lsp30-cpp' +scp cpp/build/release/app "fedora@$IP:/tmp/lsp30-cpp/app" +scp observe.sh "fedora@$IP:/tmp/lsp30-observe.sh" +ssh "fedora@$IP" 'chmod 755 /tmp/lsp30-cpp/app /tmp/lsp30-observe.sh' + +# run the observation, as root +ssh "fedora@$IP" 'sudo bash /tmp/lsp30-observe.sh /tmp/lsp30-cpp/app 60' +``` + +A real run's tail looks like this: + +``` +=== funccount busy_hash (uprobe/funccount bait) === +Tracing 1 functions for "b'/tmp/lsp30-cpp/app:*busy_hash*'"... Hit Ctrl-C to end. + +FUNC COUNT +busy_hash 21 +Detaching... + +=== offcputime (sleep off-CPU bait) === +(33 lines captured; frames naming our binary/function:) + - app (13416) + __clock_nanosleep_2 + do_nanosleep + hrtimer_nanosleep + __x64_sys_clock_nanosleep + +=== bpftrace uprobe:busy_hash === +Attached 2 probes +@calls: 22 + +=== summary === +opensnoop_opens=20 +execsnoop_execs=6 +funccount_calls=21 +offcputime_hits=2 +bpftrace_calls=22 +app_summary: work: done seconds=60 iters=260 opens=260 execs=65 busy_calls=260 busy_hash=12869313528293982360 +``` + +## Verification + +`verify.lua` (run per language under the runner's **vm mode**, +`TARGET=systems-target`) stages the built binary under `/tmp/lsp30-/app` +— so its `/proc` comm is exactly `app` regardless of which language built +it — plus `observe.sh`, then runs the driver as root and asserts, from its +captured output, that each tool really observed our program: + +1. **opensnoop** shows comm `app` opening the bait file's exact path. +2. **execsnoop** shows a `true` child whose **PPID equals the workload's own + pid** (parsed from `observe.sh`'s own `workload pid=` line) — not just + that *some* exec happened somewhere on the box. +3. **funccount** reports a **nonzero** call count for `busy_hash`. +4. **offcputime** shows a stack naming comm `app` (the ~230ms sleep between + iterations put it off-CPU during the trace window). +5. **bpftrace**'s uprobe one-liner reports `@calls` **nonzero**. + +Each is a positive count/match on a *specific* signal our program produced, +not merely "the tool ran and exited 0" — a tool that loads and exits cleanly +while counting nothing would fail every one of these. + +Run it: + +```sh +LIBVIRT_DEFAULT_URI=qemu:///system \ + python3 scripts/test-all-examples.py --only 30-ebpf-observation-toolkit --mode vm +``` + +**Why this is real observation, not an inference:** every assertion is tied +to something our own program did — the exact bait path, the workload's own +pid as the exec's parent, a call count on the exact function we wrote. This +is the standard eBPF-based observability toolkit doing what it's designed to +do: watch an ordinary, unmodified userspace binary from the outside. We write +no kernel-side eBPF program here — bcc-tools and bpftrace already ship +theirs; this chapter is about aiming them correctly and reading the answer. diff --git a/examples/30-ebpf-observation-toolkit/cpp/.clang-format b/examples/30-ebpf-observation-toolkit/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/30-ebpf-observation-toolkit/cpp/.clang-tidy b/examples/30-ebpf-observation-toolkit/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/30-ebpf-observation-toolkit/cpp/CMakeLists.txt b/examples/30-ebpf-observation-toolkit/cpp/CMakeLists.txt new file mode 100644 index 0000000..7a3ef02 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/CMakeLists.txt @@ -0,0 +1,11 @@ +cmake_minimum_required(VERSION 3.25) +project(ebpf_observation_toolkit CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +find_package(Threads REQUIRED) + +add_executable(app src/main.cpp) +target_link_libraries(app PRIVATE Threads::Threads) diff --git a/examples/30-ebpf-observation-toolkit/cpp/CMakePresets.json b/examples/30-ebpf-observation-toolkit/cpp/CMakePresets.json new file mode 100644 index 0000000..e31bd2f --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/CMakePresets.json @@ -0,0 +1,62 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/30-ebpf-observation-toolkit/cpp/demo.sh b/examples/30-ebpf-observation-toolkit/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp b/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp new file mode 100644 index 0000000..857ff34 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/cpp/src/main.cpp @@ -0,0 +1,210 @@ +// app work --seconds N — a small, observable workload for the eBPF +// observation toolkit (chapter 30). It does four things in a loop, each one +// deliberate bait for a specific bcc-tools/bpftrace probe: +// +// 1. opens (create+write+close) a fixed file every iteration -> opensnoop +// 2. fork/execs a short-lived child ("true") every 4th iter -> execsnoop +// 3. calls the named hot function busy_hash() every iteration -> funccount +// / uprobe +// 4. sleeps most of the iteration, so the process is off-CPU -> offcputime +// +// This file writes no kernel-side eBPF: it is the userspace *target* that +// examples/30-ebpf-observation-toolkit/observe.sh (running as root on the lab +// VM) points bcc-tools and bpftrace at. +// +// C++23 idioms: a std::jthread (with std::stop_token) runs the workload loop +// so main() gets automatic join-on-scope-exit; an RAII Fd wraps the bait +// file's descriptor; std::expected carries every fallible syscall; output +// goes through std::println. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +namespace { + +using namespace std::chrono_literals; + +constexpr auto kIterInterval = 230ms; +constexpr int kExecEveryNIters = 4; +constexpr std::uint64_t kBusyRounds = 300'000; +const char* kBaitPath = "/tmp/lsp-ebpf-work-bait.txt"; + +// The named hot function every uprobe/funccount/bpftrace command in this +// chapter targets by name. extern "C" so the symbol isn't mangled, matching +// the Rust build; noinline so -O2 can't fold it into the caller and make the +// uprobe's attach point disappear. +extern "C" [[gnu::noinline]] std::uint64_t busy_hash(std::uint64_t x) noexcept { + for (std::uint64_t i = 0; i < kBusyRounds; ++i) { + x ^= x << 7; + x ^= x >> 9; + } + return x; +} + +// RAII wrapper: closes the fd no matter how the scope is left. +class Fd { + public: + explicit Fd(int fd) noexcept : fd_(fd) {} + ~Fd() { + if (fd_ >= 0) { + ::close(fd_); + } + } + Fd(const Fd&) = delete; + Fd& operator=(const Fd&) = delete; + [[nodiscard]] int get() const noexcept { return fd_; } + [[nodiscard]] bool valid() const noexcept { return fd_ >= 0; } + + private: + int fd_; +}; + +// opensnoop bait: create/truncate, write a line, close. Every call is a fresh +// open(2) on the same path. +[[nodiscard]] std::expected open_bait(long iter) { + // 0666: the bait file may get created by a root-run observe.sh one time + // and an unprivileged demo.sh run the next — world-writable avoids a + // permission mismatch between those two ownership scenarios. + Fd fd{::open(kBaitPath, O_CREAT | O_WRONLY | O_TRUNC, 0666)}; + if (!fd.valid()) { + return std::unexpected(std::error_code{errno, std::system_category()}); + } + const std::string line = "iter " + std::to_string(iter) + "\n"; + if (::write(fd.get(), line.data(), line.size()) < 0) { + return std::unexpected(std::error_code{errno, std::system_category()}); + } + return {}; +} + +// execsnoop bait: fork + execvp("true"), wait for it. Returns the child's +// exit status, or a syscall error. +[[nodiscard]] std::expected spawn_true() { + const pid_t child = ::fork(); + if (child < 0) { + return std::unexpected(std::error_code{errno, std::system_category()}); + } + if (child == 0) { + char* argv[] = {const_cast("true"), nullptr}; + ::execvp("true", argv); + _exit(127); // execvp only returns on failure + } + int wstatus = 0; + if (::waitpid(child, &wstatus, 0) < 0) { + return std::unexpected(std::error_code{errno, std::system_category()}); + } + return WIFEXITED(wstatus) ? WEXITSTATUS(wstatus) : -1; +} + +struct Counters { + std::atomic iters{0}; + std::atomic opens{0}; + std::atomic execs{0}; + std::atomic busy_calls{0}; + // The accumulated busy_hash() result. Printed in the summary line below — + // that's what keeps the optimizer from proving the whole busy_hash call + // chain's result is unused and deleting it outright (noinline only stops + // inlining, not interprocedural dead-code elimination of a provably-pure, + // provably-unused call chain). + std::atomic busy_hash_val{0}; +}; + +void run_workload(std::stop_token stoken, int seconds, Counters& c) { + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(seconds); + long iter = 0; + std::uint64_t acc = 1; + while (!stoken.stop_requested() && std::chrono::steady_clock::now() < deadline) { + if (auto r = open_bait(iter); r) { + c.opens.fetch_add(1, std::memory_order_relaxed); + } else { + std::println(stderr, "work: open bait failed: {}", r.error().message()); + } + + if (iter % kExecEveryNIters == 0) { + if (auto r = spawn_true(); r) { + c.execs.fetch_add(1, std::memory_order_relaxed); + std::println("work: exec i={} status={}", iter, *r); + } else { + std::println(stderr, "work: spawn failed: {}", r.error().message()); + } + } + + acc = busy_hash(acc); + c.busy_calls.fetch_add(1, std::memory_order_relaxed); + + std::this_thread::sleep_for(kIterInterval); + ++iter; + } + c.iters.store(iter, std::memory_order_relaxed); + c.busy_hash_val.store(acc, std::memory_order_relaxed); +} + +int cmd_work(int seconds) { + std::println("work: start seconds={} pid={} bait={}", seconds, ::getpid(), kBaitPath); + Counters counters; + // jthread: takes a stop_token as its first parameter (so it's ready for + // cooperative cancellation, e.g. from a signal handler) and joins on + // request/destruction like a plain thread — join explicitly here so the + // summary line below always sees the final counters. + std::jthread worker(run_workload, seconds, std::ref(counters)); + worker.join(); + std::println("work: done seconds={} iters={} opens={} execs={} busy_calls={} busy_hash={}", + seconds, counters.iters.load(), counters.opens.load(), counters.execs.load(), + counters.busy_calls.load(), counters.busy_hash_val.load()); + return 0; +} + +void usage() { + std::println(stderr, "usage: app work --seconds N"); +} + +} // namespace + +int main(int argc, char** argv) { + // glibc fully-buffers stdout by default once it's not a terminal (e.g. + // redirected to observe.sh's log file), so progress lines would sit in a + // libc buffer instead of reaching a live "tail -f"/"head" the way Go's + // and Rust's stdout already do. Force line buffering so every std::println + // shows up the moment it's printed, redirected or not. + std::setvbuf(stdout, nullptr, _IOLBF, BUFSIZ); + + if (argc < 2 || std::string_view(argv[1]) != "work") { + usage(); + return 2; + } + int seconds = -1; + for (int i = 2; i < argc; ++i) { + const std::string_view arg = argv[i]; + if (arg == "--seconds" && i + 1 < argc) { + char* end = nullptr; + const long v = std::strtol(argv[++i], &end, 10); + if (end == argv[i] || *end != '\0' || v <= 0) { + std::println(stderr, "work: bad --seconds value: {}", argv[i]); + return 2; + } + seconds = static_cast(v); + } else { + usage(); + return 2; + } + } + if (seconds <= 0) { + usage(); + return 2; + } + return cmd_work(seconds); +} diff --git a/examples/30-ebpf-observation-toolkit/demo.sh b/examples/30-ebpf-observation-toolkit/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/30-ebpf-observation-toolkit/go/demo.sh b/examples/30-ebpf-observation-toolkit/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/30-ebpf-observation-toolkit/go/go.mod b/examples/30-ebpf-observation-toolkit/go/go.mod new file mode 100644 index 0000000..3ff116a --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/30-ebpf-observation-toolkit/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/30-ebpf-observation-toolkit/go/main.go b/examples/30-ebpf-observation-toolkit/go/main.go new file mode 100644 index 0000000..e9deb7e --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/go/main.go @@ -0,0 +1,160 @@ +// app work --seconds N — a small, observable workload for the eBPF +// observation toolkit (chapter 30). It does four things in a loop, each one +// deliberate bait for a specific bcc-tools/bpftrace probe: +// +// 1. opens (create+write+close) a fixed file every iteration -> opensnoop +// 2. fork/execs a short-lived child ("true") every 4th iter -> execsnoop +// 3. calls the named hot function busy_hash() every iteration -> funccount +// / uprobe +// 4. sleeps most of the iteration, so the process is off-CPU -> offcputime +// +// This file writes no kernel-side eBPF: it is the userspace *target* that +// examples/30-ebpf-observation-toolkit/observe.sh (running as root on the lab +// VM) points bcc-tools and bpftrace at. +// +// Go idioms: the workload loop runs in a goroutine; main blocks on a result +// channel (the goroutine's "join"). Every fallible call wraps its error with +// fmt.Errorf("...: %w", err) so the underlying syscall error survives. +package main + +import ( + "fmt" + "os" + "os/exec" + "strconv" + "time" +) + +const ( + iterInterval = 230 * time.Millisecond + execEveryNIters = 4 + busyRounds = 300_000 + baitPath = "/tmp/lsp-ebpf-work-bait.txt" +) + +// busy_hash is the named hot function every uprobe/funccount/bpftrace command +// in this chapter targets by name. Go 1.26 emits it as "main.busy_hash" in +// the symbol table; //go:noinline keeps the compiler from folding it into its +// caller and making the uprobe's attach point disappear. +// +//go:noinline +func busy_hash(x uint64) uint64 { + for i := 0; i < busyRounds; i++ { + x ^= x << 7 + x ^= x >> 9 + } + return x +} + +// openBait is the opensnoop bait: create/truncate, write a line, close. +// Every call is a fresh open(2) on the same path. +func openBait(iter int) error { + // 0666: the bait file may get created by a root-run observe.sh one time + // and an unprivileged demo.sh run the next — world-writable avoids a + // permission mismatch between those two ownership scenarios. + f, err := os.OpenFile(baitPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o666) + if err != nil { + return fmt.Errorf("open bait: %w", err) + } + defer f.Close() + if _, err := fmt.Fprintf(f, "iter %d\n", iter); err != nil { + return fmt.Errorf("write bait: %w", err) + } + return nil +} + +// spawnTrue is the execsnoop bait: exec "true" and wait for it. The Go +// runtime does the fork+exec; the child's PPID is this process either way. +func spawnTrue() (int, error) { + cmd := exec.Command("true") + if err := cmd.Run(); err != nil { + return -1, fmt.Errorf("spawn true: %w", err) + } + return cmd.ProcessState.ExitCode(), nil +} + +type result struct { + iters, opens, execs int + busyCalls int + // The accumulated busy_hash() result, printed in the summary line. Go's + // compiler doesn't dead-code-eliminate package-level calls the way GCC's + // interprocedural passes can, but keeping the final value observable is + // the reliable, optimizer-proof way to guarantee every call really ran. + busyHash uint64 +} + +func runWorkload(seconds int, done chan<- result) { + deadline := time.Now().Add(time.Duration(seconds) * time.Second) + var r result + acc := uint64(1) + for time.Now().Before(deadline) { + if err := openBait(r.iters); err == nil { + r.opens++ + } else { + fmt.Fprintln(os.Stderr, "work:", err) + } + + if r.iters%execEveryNIters == 0 { + if status, err := spawnTrue(); err == nil { + r.execs++ + fmt.Printf("work: exec i=%d status=%d\n", r.iters, status) + } else { + fmt.Fprintln(os.Stderr, "work:", err) + } + } + + acc = busy_hash(acc) + r.busyCalls++ + + time.Sleep(iterInterval) + r.iters++ + } + r.busyHash = acc + done <- r +} + +func cmdWork(seconds int) int { + fmt.Printf("work: start seconds=%d pid=%d bait=%s\n", seconds, os.Getpid(), baitPath) + + // The goroutine is the worker; the channel is its "join" — main blocks + // on the result rather than sharing counters behind a mutex. + done := make(chan result, 1) + go runWorkload(seconds, done) + r := <-done + + fmt.Printf("work: done seconds=%d iters=%d opens=%d execs=%d busy_calls=%d busy_hash=%d\n", + seconds, r.iters, r.opens, r.execs, r.busyCalls, r.busyHash) + return 0 +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: app work --seconds N") +} + +func main() { + if len(os.Args) < 2 || os.Args[1] != "work" { + usage() + os.Exit(2) + } + seconds := -1 + args := os.Args[2:] + for i := 0; i < len(args); i++ { + if args[i] == "--seconds" && i+1 < len(args) { + v, err := strconv.Atoi(args[i+1]) + if err != nil || v <= 0 { + fmt.Fprintf(os.Stderr, "work: bad --seconds value: %s\n", args[i+1]) + os.Exit(2) + } + seconds = v + i++ + } else { + usage() + os.Exit(2) + } + } + if seconds <= 0 { + usage() + os.Exit(2) + } + os.Exit(cmdWork(seconds)) +} diff --git a/examples/30-ebpf-observation-toolkit/observe.sh b/examples/30-ebpf-observation-toolkit/observe.sh new file mode 100755 index 0000000..93f837a --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/observe.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# observe.sh — the actual demo of this chapter. Runs ON THE LAB GUEST AS ROOT +# (bcc-tools and bpftrace both need CAP_BPF/CAP_PERFMON, which on this lab +# image means root). It starts `app work --seconds N` in the background and +# then points five separate observation tools at it in turn, proving each one +# saw a real event our program produced: +# +# opensnoop -> the bait file open(2) +# execsnoop -> the fork/exec of "true" +# funccount -> a per-call count of busy_hash() +# offcputime -> the sleep between iterations (off-CPU stacks) +# bpftrace -> a uprobe on busy_hash(), one-liner style +# +# This is tooling observing our userspace binary via uprobes/tracepoints; it +# writes no kernel-side eBPF program of its own. +# +# Usage: observe.sh [seconds] +# +# Not `set -e`: a bcc tool killed by its own `timeout` wrapper is expected, +# not a driver failure — each step is checked/reported on its own. +set -uo pipefail + +BIN="${1:?usage: observe.sh [seconds]}" +WORK_SECONDS="${2:-60}" +BCC=/usr/share/bcc/tools +BAIT=/tmp/lsp-ebpf-work-bait.txt +OUT=/tmp/lsp-ebpf-observe +COMM="$(basename "$BIN")" # /proc comm the kernel records for this binary (<=15 chars: "app") +PATTERN="${BIN}:*busy_hash*" # matches busy_hash (cpp/rust) and main.busy_hash (go) + +if [[ $EUID -ne 0 ]]; then + echo "observe.sh must run as root (bcc-tools/bpftrace need it)" >&2 + exit 1 +fi + +mkdir -p "$OUT" +rm -f "$OUT"/*.out "$BAIT" +pkill -x "$COMM" 2>/dev/null || true +sleep 0.5 + +echo "=== workload ===" +"$BIN" work --seconds "$WORK_SECONDS" >"$OUT/app.log" 2>&1 & +APP_PID=$! +sleep 0.7 +if ! kill -0 "$APP_PID" 2>/dev/null; then + echo "workload failed to start:" >&2 + cat "$OUT/app.log" >&2 + exit 1 +fi +echo "workload pid=$APP_PID" +head -1 "$OUT/app.log" + +echo +echo "=== opensnoop (open() bait) ===" +timeout 6 "$BCC/opensnoop" -p "$APP_PID" >"$OUT/opensnoop.out" 2>&1 || true +tail -n +1 "$OUT/opensnoop.out" + +echo +echo "=== execsnoop (fork/exec bait) ===" +timeout 6 "$BCC/execsnoop" -P "$APP_PID" >"$OUT/execsnoop.out" 2>&1 || true +tail -n +1 "$OUT/execsnoop.out" + +echo +echo "=== funccount busy_hash (uprobe/funccount bait) ===" +timeout 7 "$BCC/funccount" -p "$APP_PID" -d 5 "$PATTERN" >"$OUT/funccount.out" 2>&1 || true +cat "$OUT/funccount.out" + +echo +echo "=== offcputime (sleep off-CPU bait) ===" +timeout 8 "$BCC/offcputime" -p "$APP_PID" 6 >"$OUT/offcputime.out" 2>&1 || true +echo "($(wc -l <"$OUT/offcputime.out") lines captured; frames naming our binary/function:)" +grep -E "busy_hash|nanosleep|${COMM} \(" "$OUT/offcputime.out" 2>/dev/null | sort -u + +echo +echo "=== bpftrace uprobe:busy_hash ===" +timeout 7 bpftrace -e "uprobe:${BIN}:*busy_hash* { @calls = count(); } interval:s:5 { print(@calls); exit(); }" \ + >"$OUT/bpftrace.out" 2>&1 || true +cat "$OUT/bpftrace.out" + +# Let the workload finish on its own — it's bounded by its own --seconds +# deadline, budgeted comfortably longer than the tool sequence above. Poll +# rather than a bare `wait`, so a shell job-control quirk can't cause a +# spurious early kill of a workload that's still legitimately running; only +# force it if it somehow overruns its own budget by a wide margin. +deadline_ts=$(($(date +%s) + WORK_SECONDS + 20)) +while kill -0 "$APP_PID" 2>/dev/null; do + if [[ $(date +%s) -ge $deadline_ts ]]; then + echo "workload overran its budget; stopping it" >&2 + kill "$APP_PID" 2>/dev/null || true + break + fi + sleep 0.5 +done +wait "$APP_PID" 2>/dev/null || true + +echo +echo "=== summary ===" +opens=$(grep -c "$BAIT" "$OUT/opensnoop.out" 2>/dev/null || true) +execs=$(grep -c '^true' "$OUT/execsnoop.out" 2>/dev/null || true) +funccalls=$(grep -oE 'busy_hash[[:space:]]+[0-9]+' "$OUT/funccount.out" 2>/dev/null | awk '{print $2}') +offcpu_hits=$(grep -c "${COMM} (" "$OUT/offcputime.out" 2>/dev/null || true) +bt_calls=$(grep -oE '@calls: [0-9]+' "$OUT/bpftrace.out" 2>/dev/null | tail -1 | awk '{print $2}') +echo "opensnoop_opens=${opens:-0}" +echo "execsnoop_execs=${execs:-0}" +echo "funccount_calls=${funccalls:-0}" +echo "offcputime_hits=${offcpu_hits:-0}" +echo "bpftrace_calls=${bt_calls:-0}" +echo "app_summary: $(tail -1 "$OUT/app.log")" diff --git a/examples/30-ebpf-observation-toolkit/rust/Cargo.lock b/examples/30-ebpf-observation-toolkit/rust/Cargo.lock new file mode 100644 index 0000000..0ce08a1 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/rust/Cargo.lock @@ -0,0 +1,73 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "ebpf-observation-toolkit" +version = "0.1.0" +dependencies = [ + "anyhow", + "rustix", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] diff --git a/examples/30-ebpf-observation-toolkit/rust/Cargo.toml b/examples/30-ebpf-observation-toolkit/rust/Cargo.toml new file mode 100644 index 0000000..85864cb --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/rust/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "ebpf-observation-toolkit" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +anyhow = "1" +rustix = { version = "1", features = ["fs"] } diff --git a/examples/30-ebpf-observation-toolkit/rust/demo.sh b/examples/30-ebpf-observation-toolkit/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/30-ebpf-observation-toolkit/rust/rust-toolchain.toml b/examples/30-ebpf-observation-toolkit/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/30-ebpf-observation-toolkit/rust/src/main.rs b/examples/30-ebpf-observation-toolkit/rust/src/main.rs new file mode 100644 index 0000000..ec64b04 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/rust/src/main.rs @@ -0,0 +1,161 @@ +//! app work --seconds N — a small, observable workload for the eBPF +//! observation toolkit (chapter 30). It does four things in a loop, each one +//! deliberate bait for a specific bcc-tools/bpftrace probe: +//! +//! 1. opens (create+write+close) a fixed file every iteration -> opensnoop +//! 2. fork/execs a short-lived child ("true") every 4th iter -> execsnoop +//! 3. calls the named hot function busy_hash() every iteration -> funccount / uprobe +//! 4. sleeps most of the iteration, so the process is off-CPU -> offcputime +//! +//! This file writes no kernel-side eBPF: it is the userspace *target* that +//! examples/30-ebpf-observation-toolkit/observe.sh (running as root on the +//! lab VM) points bcc-tools and bpftrace at. +//! +//! Rust idioms: every fallible call flows through Result and `?`; the bait +//! file's descriptor is a rustix OwnedFd, closed by its Drop impl the moment +//! it goes out of scope — no explicit close(2). + +use std::time::{Duration, Instant}; + +use anyhow::{Context, Result}; +use rustix::fs::{Mode, OFlags, open}; +use rustix::io::write; + +const ITER_INTERVAL: Duration = Duration::from_millis(230); +const EXEC_EVERY_N_ITERS: u64 = 4; +const BUSY_ROUNDS: u64 = 300_000; +const BAIT_PATH: &str = "/tmp/lsp-ebpf-work-bait.txt"; + +/// The named hot function every uprobe/funccount/bpftrace command in this +/// chapter targets by name. `#[unsafe(no_mangle)]` keeps the symbol as the +/// plain string "busy_hash" (matching the C++ build); `#[inline(never)]` +/// keeps the optimizer from folding it into its caller and making the +/// uprobe's attach point disappear. +#[unsafe(no_mangle)] +#[inline(never)] +pub extern "C" fn busy_hash(mut x: u64) -> u64 { + for _ in 0..BUSY_ROUNDS { + x ^= x << 7; + x ^= x >> 9; + } + x +} + +/// opensnoop bait: create/truncate, write a line, close. Every call is a +/// fresh open(2) on the same path; the OwnedFd's Drop does the close(2). +fn open_bait(iter: u64) -> Result<()> { + // 0666: the bait file may get created by a root-run observe.sh one time + // and an unprivileged demo.sh run the next — world-writable avoids a + // permission mismatch between those two ownership scenarios. + let fd = open( + BAIT_PATH, + OFlags::CREATE | OFlags::WRONLY | OFlags::TRUNC, + Mode::from_raw_mode(0o666), + ) + .context("open bait")?; + write(&fd, format!("iter {iter}\n").as_bytes()).context("write bait")?; + Ok(()) + // `fd` (an OwnedFd) drops here, closing the descriptor. +} + +/// execsnoop bait: spawn "true" and wait for it. std::process::Command does +/// the fork+exec; the child's PPID is this process either way. +fn spawn_true() -> Result { + let status = std::process::Command::new("true") + .status() + .context("spawn true")?; + Ok(status.code().unwrap_or(-1)) +} + +#[derive(Default)] +struct Counters { + iters: u64, + opens: u64, + execs: u64, + busy_calls: u64, + // The accumulated busy_hash() result, printed in the summary line. LLVM + // can prove a #[no_mangle]/#[inline(never)] function is still pure and + // dead-code-eliminate a whole call chain whose final result is never + // observed — keeping this value alive here is what guarantees every call + // really executed, not just that the symbol exists in the binary. + busy_hash: u64, +} + +fn run_workload(seconds: u64) -> Counters { + let deadline = Instant::now() + Duration::from_secs(seconds); + let mut c = Counters::default(); + let mut acc: u64 = 1; + while Instant::now() < deadline { + match open_bait(c.iters) { + Ok(()) => c.opens += 1, + Err(e) => eprintln!("work: {e:#}"), + } + + if c.iters % EXEC_EVERY_N_ITERS == 0 { + match spawn_true() { + Ok(status) => { + c.execs += 1; + println!("work: exec i={} status={}", c.iters, status); + } + Err(e) => eprintln!("work: {e:#}"), + } + } + + acc = busy_hash(acc); + c.busy_calls += 1; + + std::thread::sleep(ITER_INTERVAL); + c.iters += 1; + } + c.busy_hash = acc; + c +} + +fn cmd_work(seconds: u64) -> i32 { + println!( + "work: start seconds={} pid={} bait={}", + seconds, + std::process::id(), + BAIT_PATH + ); + let c = run_workload(seconds); + println!( + "work: done seconds={} iters={} opens={} execs={} busy_calls={} busy_hash={}", + seconds, c.iters, c.opens, c.execs, c.busy_calls, c.busy_hash + ); + 0 +} + +fn usage() { + eprintln!("usage: app work --seconds N"); +} + +fn main() -> std::process::ExitCode { + let args: Vec = std::env::args().collect(); + if args.len() < 2 || args[1] != "work" { + usage(); + return std::process::ExitCode::from(2); + } + let mut seconds: i64 = -1; + let mut i = 2; + while i < args.len() { + if args[i] == "--seconds" && i + 1 < args.len() { + match args[i + 1].parse::() { + Ok(v) if v > 0 => seconds = v, + _ => { + eprintln!("work: bad --seconds value: {}", args[i + 1]); + return std::process::ExitCode::from(2); + } + } + i += 2; + } else { + usage(); + return std::process::ExitCode::from(2); + } + } + if seconds <= 0 { + usage(); + return std::process::ExitCode::from(2); + } + std::process::ExitCode::from(cmd_work(seconds as u64) as u8) +} diff --git a/examples/30-ebpf-observation-toolkit/verify.lua b/examples/30-ebpf-observation-toolkit/verify.lua new file mode 100644 index 0000000..b291898 --- /dev/null +++ b/examples/30-ebpf-observation-toolkit/verify.lua @@ -0,0 +1,126 @@ +-- Verify the eBPF observation toolkit (chapter 30) — a VM example. This is +-- TOOLING (bcc-tools + bpftrace) observing our userspace binary via uprobes +-- and tracepoints; we write no kernel-side eBPF program of our own. +-- +-- The workload (`app work --seconds N`) is deployed to the lab guest, then +-- observe.sh (staged alongside it, run as root via SUDO) starts the workload +-- in the background and points five separate tools at it in turn. Each +-- assertion below is the OBSERVABLE proof that a specific tool really saw a +-- real event our program produced, not just that the tool ran without error: +-- +-- 1. opensnoop — shows our bait file's path opened by comm "app". +-- 2. execsnoop — shows a "true" child with PPID == our workload's pid. +-- 3. funccount — a nonzero per-call count for busy_hash() via uprobe. +-- 4. offcputime — a stack naming comm "app" (the sleep between iterations +-- put it off-CPU during the trace window). +-- 5. bpftrace — a one-liner uprobe on busy_hash() reports @calls > 0. +-- +-- funccount/bpftrace both target the pattern ":*busy_hash*", which +-- matches the plain symbol "busy_hash" (cpp/rust) and the Go runtime's +-- "main.busy_hash" without needing a different pattern per language. +-- +-- Runs under the runner's vm mode (TARGET=systems-target). Staging goes +-- straight through ssh/scp (not deploy-to-vm.sh, which only shells out a +-- single binary+args — this needs a custom root-run driver script too). + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local target = os.getenv("TARGET") +if not target or target == "" then + checks.skip("no TARGET set — this is a vm example (run with --mode vm)") +end + +local example_dir = os.getenv("EXAMPLE_DIR") or "." +local home = os.getenv("HOME") or "/root" + +-- Local path of the freshly built binary for this language. +local bin_rel = ({ + cpp = "cpp/build/release/app", + go = "go/bin/app", + rust = "rust/target/release/app", +})[lang] +local local_bin = example_dir .. "/" .. bin_rel +local local_observe = example_dir .. "/observe.sh" + +-- Resolve the guest IP (same helper deploy-to-vm.sh uses). +local ipf = io.popen(repo_root .. "/scripts/lab/vm-ip.sh '" .. target .. "' 2>/dev/null") +local ip = ipf and ipf:read("*l") or nil +if ipf then ipf:close() end +if not ip or ip == "" then + checks.skip("cannot resolve IP for VM '" .. target .. "' (lab down?)") +end + +local ssh_opts = + "-o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=" .. home .. "/.ssh/known_hosts" + +-- Stage the binary under a per-language directory so the deployed file's +-- basename — and therefore its /proc comm — is exactly "app" regardless of +-- which language built it, matching the book-wide "binary stays app" rule. +local remote_dir = "/tmp/lsp30-" .. lang +local remote_bin = remote_dir .. "/app" +local remote_observe = "/tmp/lsp30-observe.sh" + +local stage_cmd = string.format( + "ssh %s 'fedora@%s' 'mkdir -p %s' && " .. + "scp %s '%s' 'fedora@%s:%s' && " .. + "scp %s '%s' 'fedora@%s:%s' && " .. + "ssh %s 'fedora@%s' 'chmod 755 %s %s'", + ssh_opts, ip, remote_dir, + ssh_opts, local_bin, ip, remote_bin, + ssh_opts, local_observe, ip, remote_observe, + ssh_opts, ip, remote_bin, remote_observe) +local st = checks.run(stage_cmd) +checks.expect_exit(st, 0, lang .. ": stage app binary + observe.sh on guest") + +-- Run the observation driver as root; 60s of workload comfortably covers the +-- ~34s of sequential tool budgets (opensnoop 6s, execsnoop 6s, funccount 7s, +-- offcputime 8s, bpftrace 7s) plus ssh/bcc-compile overhead. +local observe = checks.run(string.format( + "ssh %s 'fedora@%s' 'sudo bash %s %s 60'", + ssh_opts, ip, remote_observe, remote_bin)) +checks.expect_exit(observe, 0, lang .. ": observe.sh exits 0") +if observe.exit ~= 0 then + print(observe.out) +end +local out = observe.out + +checks.expect_match(out, "work: start seconds=60 pid=%d+ bait=/tmp/lsp%-ebpf%-work%-bait%.txt", + lang .. ": workload prints its start line") +checks.expect_match(out, + "work: done seconds=60 iters=%d+ opens=%d+ execs=%d+ busy_calls=%d+ busy_hash=%d+", + lang .. ": workload prints its done/summary line") + +-- 1. opensnoop saw our bait file opened by comm "app". +checks.expect_match(out, "app%s+%d+%s+%d+%s+/tmp/lsp%-ebpf%-work%-bait%.txt", + lang .. ": opensnoop shows app opening the bait file") + +-- 2. execsnoop saw the "true" child, parented by our workload's pid. +local app_pid = out:match("workload pid=(%d+)") +if app_pid then + checks.expect_match(out, "true%s+%d+%s+" .. app_pid .. "%s+0%s+/usr/[a-z]*/?true", + lang .. ": execsnoop shows a true child with PPID=" .. app_pid) +else + checks.expect_match(out, "\1IMPOSSIBLE\1", + lang .. ": could not read workload pid from observe.sh output") +end + +-- 3. funccount counted real calls to busy_hash() via uprobe (nonzero count). +checks.expect_match(out, "busy_hash%s+[1-9]%d*", + lang .. ": funccount reports a nonzero busy_hash call count") + +-- 4. offcputime attributed off-CPU time to a stack naming comm "app". +checks.expect_match(out, "app %(%d+%)", lang .. ": offcputime shows a stack for comm \"app\"") + +-- 5. bpftrace's uprobe on busy_hash fired (nonzero @calls). +checks.expect_match(out, "@calls:%s*[1-9]%d*", + lang .. ": bpftrace uprobe busy_hash calls is nonzero") + +checks.finish() diff --git a/examples/31-per-language-toolbelts/README.md b/examples/31-per-language-toolbelts/README.md new file mode 100644 index 0000000..b22f3aa --- /dev/null +++ b/examples/31-per-language-toolbelts/README.md @@ -0,0 +1,313 @@ +# 31-per-language-toolbelts — toolbelt + +Chapter 31 closes the debugging arc by putting each language's *native* +profiling toolbelt to work on the same program. There is nothing wrong with +this code — the point is not to find a defect, but to point a real profiler +at a real binary and watch it name the function that is actually costing the +cycles or the bytes. + +``` +app [--n N] +``` + +- `hot` — counts primes in `[2, n)` by unoptimized trial division (every + candidate divisor up to `sqrt(i)`). One function, `spin_hot` / + `hotSpin`, does all the work — it is the whole point of a CPU profile. +- `alloc` — builds a 1000-key string index from `n` iterations of + round-robin overwrite, each iteration allocating a fresh key and value + string. One function, `alloc_churn` / `allocChurn`, is the only + allocation site — it is the whole point of an allocation profile. +- `--n N` overrides the default size (`hot`: 3,000,000; `alloc`: 200,000); + must be a positive integer. + +## Behavior (identical across C++, Go, and Rust) + +- On success, exactly one stdout line: + `app: mode= n= result= ms=` + where `result` is the prime count (`hot`) or the summed byte length of the + final 1000-entry index (`alloc`), and `ms` is the wall time of the + workload. The algorithm is identical byte-for-byte across languages, so + `result` is too: `app hot` always answers `216816` at the default size, + `app alloc` always answers `5000`. +- Bad usage (missing/unknown mode, non-positive `--n`, unknown flag) prints + a diagnostic plus `usage: app [--n N]` on stderr and exits 2. +- Go's binary additionally accepts `--cpuprofile FILE`, `--memprofile FILE`, + and `--trace FILE` — see "Why Go's CLI has extra flags" below. Its usage + banner lists them; C++ and Rust do not accept them (unknown-argument + errors bad-usage the same as any other unrecognized flag). + +### Why Go's CLI has extra flags + +`perf` attaches to *any* ELF binary from the outside — nothing in the C++ or +Rust source needs to know it is being profiled. Go's CPU and heap profilers +are different: `runtime/pprof` and `runtime/trace` are libraries the program +calls into, so writing a profile is part of the binary's own behavior. That +asymmetry is real and is the reason the usage banners differ even though the +` [--n N]` contract is identical. + +## Try it + +```console +$ ./demo.sh cpp run hot +app: mode=hot n=3000000 result=216816 ms=745 +$ ./demo.sh go run hot +app: mode=hot n=3000000 result=216816 ms=799 +$ ./demo.sh rust run hot +app: mode=hot n=3000000 result=216816 ms=458 +$ ./demo.sh cpp run alloc +app: mode=alloc n=200000 result=5000 ms=21 +$ ./demo.sh cpp run bogus +app: unknown mode: bogus +usage: app [--n N] +``` + +(Timings vary by host and load; `result` does not — same algorithm, same +input, same answer, in all three languages.) + +## The toolbelts, run for real + +Every command below was actually run against the binaries in this directory +(`cpp/build/release/app`, `go/bin/app`, `rust/target/release/app`) on this +host (Fedora 44, GCC 16.1.1, clang 22.1.8, go1.26.5, rustc 1.97.1, perf +7.1.3, gdb 17.2). Numbers are host- and load-dependent; the shapes and the +named functions are the point. + +### C++: perf record/report, clang-tidy, -fanalyzer + +`spin_hot` has C linkage on purpose (`extern "C"`), so perf and gdb print it +as the plain symbol `spin_hot` instead of an Itanium-mangled name. + +```console +$ perf stat -e cycles,instructions -- ./cpp/build/release/app hot +app: mode=hot n=3000000 result=216816 ms=745 + + Performance counter stats for './cpp/build/release/app hot': + + 3,212,149,799 cycles:u + 3,185,629,622 instructions:u + + 0.747341237 seconds time elapsed +``` + +```console +$ perf record -o cpp.perf.data --call-graph fp -- ./cpp/build/release/app hot +[ perf record: Captured and wrote 0.191 MB cpp.perf.data (3093 samples) ] +$ perf report -i cpp.perf.data --stdio +# Samples: 3K of event 'cpu/cycles/Pu' +# Event count (approx.): 3220157668 +# +# Children Self Command Shared Object Symbol +# ........ ........ ....... .................... ................ +# + 99.93% 99.93% app app [.] spin_hot + | + ---spin_hot +``` + +**Flamegraph note**: the classic Brendan Gregg `stackcollapse-perf.pl` / +`flamegraph.pl` scripts are not installed on this host (no `FlameGraph/` tree +on `PATH` or under any locatable directory), so per the fallback this chapter +documents, the C++ step stops at `perf report --stdio` above rather than +producing an SVG. (Compare the Rust section, where `cargo flamegraph` — a +Rust-native reimplementation of the same collapse+render pipeline — *is* +installable and produces a real SVG.) + +`clang-tidy` (bugprone-*, modernize-*, performance-*, readability-*): + +```console +$ clang-tidy -p cpp/build/release cpp/src/main.cpp +main.cpp:143:5: warning: an exception may be thrown in function 'main' which + should not throw exceptions [bugprone-exception-escape] + (traced to std::format's format_error, reachable only if the format + string itself were malformed — it is a compile-time string literal) +``` + +One warning, and it is a known shape with `std::format`/`std::println`: +`bugprone-exception-escape` traces a theoretical `std::format_error` path +that only fires on a malformed format string — ours are literals, so the +path is unreachable in practice, but the checker cannot prove that +statically. Nothing else fires. + +GCC's `-fanalyzer` (its own preset: `cmake --build --preset analyze`) is +clean — zero diagnostics on this source. + +### Go: pprof (CPU + heap), go tool trace, delve + +```console +$ ./go/bin/app hot --cpuprofile cpu.prof +app: mode=hot n=3000000 result=216816 ms=848 +$ go tool pprof --top --nodecount=5 ./go/bin/app cpu.prof + flat flat% sum% cum cum% + 840ms 98.82% 98.82% 840ms 98.82% main.hotSpin + 10ms 1.18% 100% 10ms 1.18% runtime.futex + 0 0% 100% 840ms 98.82% main.main +``` + +```console +$ ./go/bin/app alloc --memprofile mem.prof +app: mode=alloc n=200000 result=5000 ms=30 +$ go tool pprof --top --cum --alloc_space --nodecount=6 ./go/bin/app mem.prof + flat flat% sum% cum cum% + 2580.29kB 55.74% 55.74% 4116.40kB 88.92% main.allocChurn + 0 0% 55.74% 4116.40kB 88.92% main.main + 512.01kB 11.06% 66.80% 1536.11kB 33.18% fmt.Sprintf +``` + +`--alloc_space` is required for the heap check: by the time +`pprof.WriteHeapProfile` runs, the churned map from the just-finished +`allocChurn` call is already garbage (it went out of scope when the +function returned), so the default *in-use* view would show almost nothing. +`alloc_space` is pprof's cumulative-since-start counter — it remembers +everything `mallocgc` ever handed out, freed or not, which is the view that +actually answers "where did the allocations happen". + +`go tool trace` opens an interactive, browser-based UI by default, which +does not script well — but `-d=parsed` dumps the same event stream as text, +which is enough to confirm the tracer captured `hotSpin`'s execution: + +```console +$ ./go/bin/app hot --n 800000 --trace trace.out +$ go tool trace -d=parsed trace.out | grep -B2 hotSpin | head -3 + runtime.asyncPreempt @ 0x484d8a + /usr/lib/golang/src/runtime/preempt_amd64.s:124 + main.hotSpin @ 0x4d272c +``` + +Delve, scripted non-interactively (no TUI) with a piped command script: + +```console +$ echo -e "break main.hotSpin\ncontinue\nprint n\nbt\ncontinue\nquit" \ + | dlv --allow-non-terminal-interactive=true exec ./go/bin/app -- hot --n 500000 +(dlv) Breakpoint 1 set at 0x4d26e1 for main.hotSpin() ./go/main.go:100 +(dlv) > [Breakpoint 1] main.hotSpin() ./go/main.go:100 (hits goroutine(1):1 total:1) +(dlv) 500000 +(dlv) 0 0x00000000004d26e1 in main.hotSpin + at ./go/main.go:100 +1 0x00000000004d2f95 in main.run + at ./go/main.go:175 +``` + +(`dlv` is not preinstalled on this host; `go install +github.com/go-delve/delve/cmd/dlv@latest` pulls it from the Go module proxy +into `$(go env GOPATH)/bin`. Where that install is unavailable — no module +proxy reachable — the pprof and trace steps above still stand on their own; +delve is demonstrated here, not gated on by `verify.lua`.) + +### Rust: perf record/report, cargo flamegraph, clippy + +```console +$ perf record -o rust.perf.data --call-graph fp -- ./rust/target/release/app hot +[ perf record: Captured and wrote 0.124 MB rust.perf.data (1783 samples) ] +$ perf report -i rust.perf.data --stdio +# Samples: 1K of event 'cpu/cycles/Pu' + 99.99% 0.00% app [unknown] [.] 0xcccccccccccccccc + | + ---0xcccccccccccccccc + spin_hot (inlined) +``` + +`cargo-flamegraph` is not preinstalled, but *is* installable — +`cargo install flamegraph --locked` pulls it (and the `inferno` crate it +wraps, a Rust reimplementation of the classic Perl collapse+render scripts) +from crates.io in well under a minute: + +```console +$ cargo flamegraph --output rust-hot.svg -- hot +[ perf record: Captured and wrote 28.245 MB perf.data (461 samples) ] +writing flamegraph to "rust-hot.svg" +$ grep -c spin_hot rust-hot.svg +1 +``` + +The resulting SVG contains a `spin_hot` frame, confirming the same story +`perf report` tells above, rendered instead of tabulated. Where network +access to crates.io is unavailable, the `perf record`/`perf report --stdio` +pair above is the fallback path — same data, no plot. + +`tokio-console`: not applicable here — this program has no async runtime +(no `tokio`, no `.await`), so there are no tasks for `tokio-console` to +show. It belongs to the async chapter (27), not this one. + +`cargo clippy --release --all-targets` and `cargo fmt --check` are both +clean — no findings. + +## Layout + +``` +31-per-language-toolbelts/ +├── demo.sh # dispatcher: ./demo.sh [cpp|go|rust|all|build] [args...] +├── verify.lua # automated check driven by CI +├── cpp/ # CMake preset build (release/debug/asan/analyze/*-clang) +├── go/ # go build, demo.sh +└── rust/ # cargo build, demo.sh +``` + +The C++ `analyze` preset (`cmake --build --preset analyze`) builds with +`-fanalyzer` instead of running it as a separate step — it is just another +`CMakeLists.txt`-driven build, same as `asan`. + +## Demo contract + +Each language directory has a `demo.sh` with the book-wide interface: + +- `./demo.sh build` — build only +- `./demo.sh run [args]` — run the built binary (builds first if needed) +- With env `TARGET` set, `run` deploys the binary to that lab VM via + `scripts/lab/deploy-to-vm.sh` instead of running locally. + +The top-level `demo.sh` dispatches: `./demo.sh go run hot --n 500` execs +`go/demo.sh run hot --n 500`. The binary is always named `app`. The +profiler invocations above run directly against the built binary paths +(`cpp/build/release/app`, `go/bin/app`, `rust/target/release/app`) rather +than through `demo.sh`, since wrapping `perf`/`pprof`/`dlv` around the demo +dispatcher is not part of the book-wide demo contract. + +## Verification + +`verify.lua` (run per language with `LSP_LANG` set) skips (exit 77) if the +binary is not yet built or if `perf` is not on `PATH`; otherwise it asserts, +against the real built binary: + +- Bad usage (missing mode, unknown mode, non-numeric `--n`, unknown flag) + exits 2 with the expected diagnostic and usage banner. +- `hot` and `alloc` at default size and at small sizes (`--n 10` / + `--n 7`) exit 0 with the exact report line, and `result` matches the + value the same algorithm produces in every language (`216816` / `5000` + at the defaults; `4` / `7` at the small sizes). +- `perf stat -e cycles,instructions` on `app hot` exits 0 and its output + parses two numeric counters — this step is identical for all three + languages, since perf attaches to any ELF binary. +- **C++ and Rust**: `perf record` then `perf report --stdio` exits 0 and + the report names `spin_hot`. +- **Go**: a CPU profile (`--cpuprofile`) is written and `go tool pprof + --top` names `main.hotSpin`; a heap profile (`--memprofile`) is written + and `go tool pprof --top --cum --alloc_space` names `main.allocChurn`. + +`go tool trace`, `delve`, `cargo flamegraph`, `clang-tidy`, and +`-fanalyzer` are demonstrated above with real captured output but are not +asserted by `verify.lua` — they either open an interactive UI +(`go tool trace`'s default mode), depend on a piped TTY session +(`delve`), or depend on a tool this session had to install over the +network (`dlv`, `cargo-flamegraph`) that may not be present in every +environment. Gating CI on them would trade a reproducible pass/fail for a +flaky one; the chapter text, not `verify.lua`, carries that evidence. + +## Runner + +| Step | C++ | Go | Rust | +|---|---|---|---| +| Build | `cmake --preset release` + ninja | `go build` | `cargo build --release` | +| CPU profile | `perf record` / `perf report --stdio` | `runtime/pprof` + `go tool pprof --top` | `perf record` / `perf report --stdio`, or `cargo flamegraph` | +| Allocation profile | *(not asserted; `alloc` mode exits 0)* | `runtime/pprof` heap + `go tool pprof --top --cum --alloc_space` | *(not asserted; `alloc` mode exits 0)* | +| Execution trace | — | `go tool trace -d=parsed` | — | +| Interactive debugger | `gdb` (chapter 28) | `dlv` | `gdb`/`rust-gdb` (chapter 28) | +| Static analysis | `clang-tidy`, `-fanalyzer` | `go vet` | `cargo clippy` | +| Result asserted by `verify.lua` | `spin_hot` named in `perf report --stdio` | `main.hotSpin`/`main.allocChurn` named in `go tool pprof --top` | `spin_hot` named in `perf report --stdio` | + +## Mapping to a chapter + +This example backs chapter 31 (per-language toolbelts), the closing chapter +of the debugging arc that opened with chapter 28 (gdb and remote debugging) +and chapter 29 (valgrind, sanitizers, miri). Where those chapters aim a +generic tool at a specific defect, this one aims each language's own +profiler at a workload built to have exactly one obvious answer. diff --git a/examples/31-per-language-toolbelts/cpp/.clang-format b/examples/31-per-language-toolbelts/cpp/.clang-format new file mode 100644 index 0000000..2c8640f --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/.clang-format @@ -0,0 +1,8 @@ +--- +BasedOnStyle: LLVM +Language: Cpp +Standard: c++20 +IndentWidth: 4 +ColumnLimit: 100 +PointerAlignment: Left +AllowShortFunctionsOnASingleLine: Inline diff --git a/examples/31-per-language-toolbelts/cpp/.clang-tidy b/examples/31-per-language-toolbelts/cpp/.clang-tidy new file mode 100644 index 0000000..2f275ba --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/.clang-tidy @@ -0,0 +1,11 @@ +--- +Checks: > + bugprone-*, + modernize-*, + performance-*, + readability-*, + -modernize-use-trailing-return-type, + -readability-identifier-length +WarningsAsErrors: '' +HeaderFilterRegex: '.*' +FormatStyle: file diff --git a/examples/31-per-language-toolbelts/cpp/CMakeLists.txt b/examples/31-per-language-toolbelts/cpp/CMakeLists.txt new file mode 100644 index 0000000..b7a9375 --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/CMakeLists.txt @@ -0,0 +1,8 @@ +cmake_minimum_required(VERSION 3.25) +project(toolbelt CXX) + +set(CMAKE_CXX_STANDARD 23) +set(CMAKE_CXX_STANDARD_REQUIRED ON) +set(CMAKE_CXX_EXTENSIONS OFF) + +add_executable(app src/main.cpp) diff --git a/examples/31-per-language-toolbelts/cpp/CMakePresets.json b/examples/31-per-language-toolbelts/cpp/CMakePresets.json new file mode 100644 index 0000000..1f4c65d --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/CMakePresets.json @@ -0,0 +1,72 @@ +{ + "version": 6, + "cmakeMinimumRequired": { "major": 3, "minor": 25, "patch": 0 }, + "configurePresets": [ + { + "name": "_base", + "hidden": true, + "generator": "Ninja", + "binaryDir": "${sourceDir}/build/${presetName}", + "cacheVariables": { + "CMAKE_EXPORT_COMPILE_COMMANDS": "ON" + } + }, + { + "name": "release", + "inherits": "_base", + "displayName": "Release (-O2 with debug info)", + "cacheVariables": { "CMAKE_BUILD_TYPE": "RelWithDebInfo" } + }, + { + "name": "debug", + "inherits": "_base", + "displayName": "Debug", + "cacheVariables": { "CMAKE_BUILD_TYPE": "Debug" } + }, + { + "name": "asan", + "inherits": "_base", + "displayName": "ASan + UBSan instrumented build", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined", + "CMAKE_EXE_LINKER_FLAGS": "-fsanitize=address,undefined" + } + }, + { + "name": "analyze", + "inherits": "_base", + "displayName": "GCC static analyzer (-fanalyzer)", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Debug", + "CMAKE_CXX_FLAGS": "-fanalyzer -O1 -g" + } + }, + { + "name": "release-clang", + "inherits": "release", + "displayName": "Release (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + }, + { + "name": "debug-clang", + "inherits": "debug", + "displayName": "Debug (clang)", + "cacheVariables": { + "CMAKE_C_COMPILER": "clang", + "CMAKE_CXX_COMPILER": "clang++" + } + } + ], + "buildPresets": [ + { "name": "release", "configurePreset": "release" }, + { "name": "debug", "configurePreset": "debug" }, + { "name": "asan", "configurePreset": "asan" }, + { "name": "analyze", "configurePreset": "analyze" }, + { "name": "release-clang", "configurePreset": "release-clang" }, + { "name": "debug-clang", "configurePreset": "debug-clang" } + ] +} diff --git a/examples/31-per-language-toolbelts/cpp/demo.sh b/examples/31-per-language-toolbelts/cpp/demo.sh new file mode 100755 index 0000000..6af6509 --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/demo.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +set -euo pipefail + +cd "$(dirname "${BASH_SOURCE[0]}")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=build/release/app + +build() { + cmake --preset release + cmake --build --preset release +} + +run() { + if [[ ! -x "$BIN" ]]; then + build + fi + if [[ -n "${TARGET:-}" ]]; then + "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) + build + ;; + run) + shift + run "$@" + ;; + "") + build + run + ;; + *) + echo "usage: $0 [build|run [args]]" >&2 + exit 2 + ;; +esac diff --git a/examples/31-per-language-toolbelts/cpp/src/main.cpp b/examples/31-per-language-toolbelts/cpp/src/main.cpp new file mode 100644 index 0000000..8a14d79 --- /dev/null +++ b/examples/31-per-language-toolbelts/cpp/src/main.cpp @@ -0,0 +1,165 @@ +// toolbelt — three focused profiling targets, one binary per language, each +// exercised by that language's native profiler (chapter 31: per-language +// toolbelts). +// +// app [--n N] +// +// hot: counts primes in [2, n) by trial division. spin_hot is the only +// function that does real work — it dominates a CPU profile. +// alloc: builds a 1000-key string index by round-robin overwrite from n +// iterations of freshly allocated strings. alloc_churn is the only +// allocation site — it dominates an allocation profile. +// +// Output: "app: mode= n= result= ms=" on success. +// spin_hot has C linkage on purpose: perf/gdb show it as the plain symbol +// "spin_hot" instead of an Itanium-mangled name, which keeps the profiler +// output (and the pattern that greps it) simple. + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +constexpr long kDefaultHotN = 3'000'000; +constexpr long kDefaultAllocN = 200'000; +constexpr long kKeyspace = 1'000; // distinct keys; the rest is churn + +struct Config { + std::string mode; + long n = 0; +}; + +// RAII scoped timer: measures its own lifetime and writes the elapsed +// milliseconds into `out_ms` on destruction, however the scope is left. +class ScopedTimer { + public: + explicit ScopedTimer(long& out_ms) noexcept + : out_ms_(out_ms), start_(std::chrono::steady_clock::now()) {} + ~ScopedTimer() { + const auto end = std::chrono::steady_clock::now(); + out_ms_ = std::chrono::duration_cast(end - start_).count(); + } + ScopedTimer(const ScopedTimer&) = delete; + ScopedTimer& operator=(const ScopedTimer&) = delete; + + private: + long& out_ms_; + std::chrono::steady_clock::time_point start_; +}; + +[[nodiscard]] std::expected parse_positive(std::string_view s) { + long v = 0; + const auto [ptr, ec] = std::from_chars(s.data(), s.data() + s.size(), v); + if (ec != std::errc{} || ptr != s.data() + s.size() || v <= 0) { + return std::unexpected("not a positive integer: " + std::string{s}); + } + return v; +} + +[[nodiscard]] std::expected parse_args(int argc, char** argv) { + if (argc < 2) { + return std::unexpected("missing mode"); + } + Config cfg; + cfg.mode = argv[1]; + if (cfg.mode == "hot") { + cfg.n = kDefaultHotN; + } else if (cfg.mode == "alloc") { + cfg.n = kDefaultAllocN; + } else { + return std::unexpected("unknown mode: " + cfg.mode); + } + + for (int i = 2; i < argc; ++i) { + const std::string_view arg = argv[i]; + if (arg == "--n" && i + 1 < argc) { + const auto v = parse_positive(argv[++i]); + if (!v) { + return std::unexpected(v.error()); + } + cfg.n = *v; + } else { + return std::unexpected("unknown argument: " + std::string{arg}); + } + } + return cfg; +} + +// The CPU-bound target: trial division is deliberately unoptimized (checks +// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +// buys a profile with a single, unmistakable hot frame. +extern "C" [[gnu::noinline]] std::uint64_t spin_hot(long n) { + std::uint64_t count = 0; + for (long i = 2; i < n; ++i) { + bool prime = true; + for (long d = 2; d * d <= i; ++d) { + if (i % d == 0) { + prime = false; + break; + } + } + if (prime) { + ++count; + } + } + return count; +} + +// The allocation-heavy target: n iterations of churn, each a fresh key and +// value string, round-robin overwriting a 1000-entry index. Every +// allocation in the program happens inside this one function. +[[nodiscard]] std::expected alloc_churn(long n) { + std::unordered_map index; + index.reserve(kKeyspace); + for (long i = 0; i < n; ++i) { + const long idx = i % kKeyspace; + index.insert_or_assign(std::format("k{}", idx), std::format("{:x}", i)); + } + + const long want = std::min(n, kKeyspace); + if (index.size() != static_cast(want)) { + return std::unexpected(std::format("index has {} entries, want {}", index.size(), want)); + } + std::uint64_t total = 0; + for (const auto& [k, v] : index) { + total += v.size(); + } + if (total == 0) { + return std::unexpected("summed zero bytes"); + } + return total; +} + +} // namespace + +int main(int argc, char** argv) { + const auto cfg = parse_args(argc, argv); + if (!cfg) { + std::println(stderr, "app: {}", cfg.error()); + std::println(stderr, "usage: app [--n N]"); + return 2; + } + + long ms = 0; + std::expected result; + { + ScopedTimer timer(ms); + result = (cfg->mode == "hot") ? std::expected(spin_hot(cfg->n)) + : alloc_churn(cfg->n); + } + if (!result) { + std::println(stderr, "app: {}", result.error()); + return 1; + } + + std::println("app: mode={} n={} result={} ms={}", cfg->mode, cfg->n, *result, ms); + return 0; +} diff --git a/examples/31-per-language-toolbelts/demo.sh b/examples/31-per-language-toolbelts/demo.sh new file mode 100755 index 0000000..afbbe4c --- /dev/null +++ b/examples/31-per-language-toolbelts/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" + +langs=(cpp go rust) + +case "${1:-all}" in + cpp|go|rust) + lang="$1" + shift + exec "./$lang/demo.sh" "$@" + ;; + all) + for lang in "${langs[@]}"; do "./$lang/demo.sh"; done + ;; + build) + for lang in "${langs[@]}"; do "./$lang/demo.sh" build; done + ;; + *) + echo "usage: $0 [cpp|go|rust|all|build] [args...]" >&2 + exit 2 + ;; +esac diff --git a/examples/31-per-language-toolbelts/go/demo.sh b/examples/31-per-language-toolbelts/go/demo.sh new file mode 100755 index 0000000..28606df --- /dev/null +++ b/examples/31-per-language-toolbelts/go/demo.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "$0")" +REPO_ROOT="$(cd ../../.. && pwd)" +BIN=bin/app + +build() { + go build -o "$BIN" . +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + exec "$REPO_ROOT/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + fi + exec "./$BIN" "$@" +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/31-per-language-toolbelts/go/go.mod b/examples/31-per-language-toolbelts/go/go.mod new file mode 100644 index 0000000..38f5223 --- /dev/null +++ b/examples/31-per-language-toolbelts/go/go.mod @@ -0,0 +1,5 @@ +module github.com/patterncatalyst/linux-systems-programming/examples/31-per-language-toolbelts/go + +go 1.26 + +toolchain go1.26.5 diff --git a/examples/31-per-language-toolbelts/go/go.sum b/examples/31-per-language-toolbelts/go/go.sum new file mode 100644 index 0000000..e69de29 diff --git a/examples/31-per-language-toolbelts/go/main.go b/examples/31-per-language-toolbelts/go/main.go new file mode 100644 index 0000000..7c97a48 --- /dev/null +++ b/examples/31-per-language-toolbelts/go/main.go @@ -0,0 +1,214 @@ +// toolbelt — three focused profiling targets, one binary per language, each +// exercised by that language's native profiler (chapter 31: per-language +// toolbelts). +// +// app [--n N] [--cpuprofile FILE] [--memprofile FILE] [--trace FILE] +// +// hot: counts primes in [2, n) by trial division. hotSpin is the only +// +// function that does real work — it dominates a CPU profile. +// +// alloc: builds a 1000-key string index by round-robin overwrite from n +// +// iterations of freshly allocated strings. allocChurn is the only +// allocation site — it dominates an allocation profile. +// +// Output: "app: mode= n= result= ms=" on success. +// +// --cpuprofile/--memprofile/--trace are Go-only: unlike perf (which attaches +// to any binary from the outside), Go's runtime/pprof and runtime/trace are +// libraries the program calls into, so writing a profile or a trace is part +// of this binary's own behavior. C++ and Rust have no equivalent flags — +// their profiling is done externally with perf — which is why the usage +// banner differs per language even though the core [--n N] +// contract is identical. +package main + +import ( + "fmt" + "os" + "runtime" + "runtime/pprof" + "runtime/trace" + "strconv" + "time" +) + +const ( + defaultHotN = 3_000_000 + defaultAllocN = 200_000 + keyspace = 1_000 // distinct keys; the rest is churn +) + +type config struct { + mode string + n int + cpuProfile string + memProfile string + trace string +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: app [--n N] [--cpuprofile FILE] [--memprofile FILE] [--trace FILE]") +} + +func parseArgs(args []string) (config, error) { + if len(args) == 0 { + return config{}, fmt.Errorf("missing mode") + } + cfg := config{mode: args[0]} + switch cfg.mode { + case "hot": + cfg.n = defaultHotN + case "alloc": + cfg.n = defaultAllocN + default: + return cfg, fmt.Errorf("unknown mode: %s", cfg.mode) + } + + for i := 1; i < len(args); i++ { + switch { + case args[i] == "--n" && i+1 < len(args): + i++ + v, err := strconv.Atoi(args[i]) + if err != nil || v <= 0 { + return cfg, fmt.Errorf("not a positive integer: %s", args[i]) + } + cfg.n = v + case args[i] == "--cpuprofile" && i+1 < len(args): + i++ + cfg.cpuProfile = args[i] + case args[i] == "--memprofile" && i+1 < len(args): + i++ + cfg.memProfile = args[i] + case args[i] == "--trace" && i+1 < len(args): + i++ + cfg.trace = args[i] + default: + return cfg, fmt.Errorf("unknown argument: %s", args[i]) + } + } + return cfg, nil +} + +// The CPU-bound target: trial division is deliberately unoptimized (checks +// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +// buys a profile with a single, unmistakable hot frame. //go:noinline keeps +// the compiler from folding it into run and hiding it from the sampler. +// +//go:noinline +func hotSpin(n int) uint64 { + var count uint64 + for i := 2; i < n; i++ { + prime := true + for d := 2; d*d <= i; d++ { + if i%d == 0 { + prime = false + break + } + } + if prime { + count++ + } + } + return count +} + +// The allocation-heavy target: n iterations of churn, each a fresh key and +// value string, round-robin overwriting a 1000-entry index. Every +// allocation in the program happens inside this one function. +// +//go:noinline +func allocChurn(n int) (uint64, error) { + index := make(map[string]string, keyspace) + for i := 0; i < n; i++ { + key := fmt.Sprintf("k%d", i%keyspace) + value := fmt.Sprintf("%x", i) + index[key] = value + } + + want := n + if want > keyspace { + want = keyspace + } + if len(index) != want { + return 0, fmt.Errorf("index has %d entries, want %d", len(index), want) + } + var total uint64 + for _, v := range index { + total += uint64(len(v)) + } + if total == 0 { + return 0, fmt.Errorf("summed zero bytes") + } + return total, nil +} + +func run(cfg config) (uint64, error) { + if cfg.cpuProfile != "" { + f, err := os.Create(cfg.cpuProfile) + if err != nil { + return 0, fmt.Errorf("create cpuprofile: %w", err) + } + defer f.Close() + if err := pprof.StartCPUProfile(f); err != nil { + return 0, fmt.Errorf("start cpuprofile: %w", err) + } + defer pprof.StopCPUProfile() + } + if cfg.trace != "" { + f, err := os.Create(cfg.trace) + if err != nil { + return 0, fmt.Errorf("create trace: %w", err) + } + defer f.Close() + if err := trace.Start(f); err != nil { + return 0, fmt.Errorf("start trace: %w", err) + } + defer trace.Stop() + } + + var result uint64 + var err error + switch cfg.mode { + case "hot": + result = hotSpin(cfg.n) + case "alloc": + result, err = allocChurn(cfg.n) + } + if err != nil { + return 0, err + } + + if cfg.memProfile != "" { + f, err := os.Create(cfg.memProfile) + if err != nil { + return 0, fmt.Errorf("create memprofile: %w", err) + } + defer f.Close() + runtime.GC() // materialize a consistent snapshot before writing + if err := pprof.WriteHeapProfile(f); err != nil { + return 0, fmt.Errorf("write memprofile: %w", err) + } + } + return result, nil +} + +func main() { + cfg, err := parseArgs(os.Args[1:]) + if err != nil { + fmt.Fprintf(os.Stderr, "app: %v\n", err) + usage() + os.Exit(2) + } + + t0 := time.Now() + result, err := run(cfg) + ms := time.Since(t0).Milliseconds() + if err != nil { + fmt.Fprintf(os.Stderr, "app: %v\n", err) + os.Exit(1) + } + + fmt.Printf("app: mode=%s n=%d result=%d ms=%d\n", cfg.mode, cfg.n, result, ms) +} diff --git a/examples/31-per-language-toolbelts/rust/Cargo.lock b/examples/31-per-language-toolbelts/rust/Cargo.lock new file mode 100644 index 0000000..f268968 --- /dev/null +++ b/examples/31-per-language-toolbelts/rust/Cargo.lock @@ -0,0 +1,16 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "toolbelt" +version = "0.1.0" +dependencies = [ + "anyhow", +] diff --git a/examples/31-per-language-toolbelts/rust/Cargo.toml b/examples/31-per-language-toolbelts/rust/Cargo.toml new file mode 100644 index 0000000..70c71fc --- /dev/null +++ b/examples/31-per-language-toolbelts/rust/Cargo.toml @@ -0,0 +1,14 @@ +[package] +name = "toolbelt" +version = "0.1.0" +edition = "2024" + +[[bin]] +name = "app" +path = "src/main.rs" + +[dependencies] +anyhow = "1.0.104" + +[profile.release] +debug = true diff --git a/examples/31-per-language-toolbelts/rust/demo.sh b/examples/31-per-language-toolbelts/rust/demo.sh new file mode 100755 index 0000000..220feea --- /dev/null +++ b/examples/31-per-language-toolbelts/rust/demo.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail +cd "$(dirname "${BASH_SOURCE[0]}")" + +BIN=target/release/app + +build() { + cargo build --release +} + +run() { + if [[ -n "${TARGET:-}" ]]; then + local repo_root + repo_root=$(cd ../../.. && pwd) + "$repo_root/scripts/lab/deploy-to-vm.sh" "$TARGET" "$BIN" -- "$@" + else + "./$BIN" "$@" + fi +} + +case "${1:-}" in + build) build ;; + run) shift; run "$@" ;; + "") build; run ;; + *) echo "usage: $0 [build|run [args...]]" >&2; exit 2 ;; +esac diff --git a/examples/31-per-language-toolbelts/rust/rust-toolchain.toml b/examples/31-per-language-toolbelts/rust/rust-toolchain.toml new file mode 100644 index 0000000..4099f1a --- /dev/null +++ b/examples/31-per-language-toolbelts/rust/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "1.97.1" +components = ["rustfmt", "clippy", "rust-src"] diff --git a/examples/31-per-language-toolbelts/rust/src/main.rs b/examples/31-per-language-toolbelts/rust/src/main.rs new file mode 100644 index 0000000..4ed0f80 --- /dev/null +++ b/examples/31-per-language-toolbelts/rust/src/main.rs @@ -0,0 +1,152 @@ +//! toolbelt — three focused profiling targets, one binary per language, each +//! exercised by that language's native profiler (chapter 31: per-language +//! toolbelts). +//! +//! app [--n N] +//! +//! hot: counts primes in [2, n) by trial division. `spin_hot` is the only +//! function that does real work — it dominates a CPU profile (perf +//! record/report, or `cargo flamegraph` where installable). +//! alloc: builds a 1000-key string index by round-robin overwrite from n +//! iterations of freshly allocated strings. `alloc_churn` is the +//! only allocation site — it dominates an allocation profile. +//! +//! Output: "app: mode= n= result= ms=" on success. + +use std::collections::HashMap; +use std::process::ExitCode; +use std::time::Instant; + +use anyhow::{Result, bail}; + +const DEFAULT_HOT_N: u64 = 3_000_000; +const DEFAULT_ALLOC_N: u64 = 200_000; +const KEYSPACE: u64 = 1_000; // distinct keys; the rest is churn + +struct Config { + mode: &'static str, + n: u64, +} + +fn usage() { + eprintln!("usage: app [--n N]"); +} + +fn parse_args() -> Result { + let args: Vec = std::env::args().skip(1).collect(); + if args.is_empty() { + return Err("missing mode".to_string()); + } + let mode: &'static str = match args[0].as_str() { + "hot" => "hot", + "alloc" => "alloc", + other => return Err(format!("unknown mode: {other}")), + }; + let mut cfg = Config { + mode, + n: if mode == "hot" { + DEFAULT_HOT_N + } else { + DEFAULT_ALLOC_N + }, + }; + + let mut i = 1; + while i < args.len() { + match args[i].as_str() { + "--n" if i + 1 < args.len() => { + i += 1; + cfg.n = match args[i].parse::() { + Ok(v) if v > 0 => v, + _ => return Err(format!("not a positive integer: {}", args[i])), + }; + } + other => return Err(format!("unknown argument: {other}")), + } + i += 1; + } + Ok(cfg) +} + +/// The CPU-bound target: trial division is deliberately unoptimized (checks +/// every candidate divisor up to sqrt(i)) so a couple of seconds of runtime +/// buys a profile with a single, unmistakable hot frame. `#[inline(never)]` +/// keeps the compiler from folding it into `run` and hiding it from the +/// sampler. +#[inline(never)] +fn spin_hot(n: u64) -> u64 { + let mut count = 0u64; + for i in 2..n { + let mut prime = true; + let mut d = 2u64; + while d * d <= i { + if i % d == 0 { + prime = false; + break; + } + d += 1; + } + if prime { + count += 1; + } + } + count +} + +/// The allocation-heavy target: n iterations of churn, each a fresh key and +/// value string, round-robin overwriting a 1000-entry index. Every +/// allocation in the program happens inside this one function. +#[inline(never)] +fn alloc_churn(n: u64) -> Result { + let mut index: HashMap = HashMap::with_capacity(KEYSPACE as usize); + for i in 0..n { + let idx = i % KEYSPACE; + let key = format!("k{idx}"); + let value = format!("{i:x}"); + index.insert(key, value); + } + + let want = n.min(KEYSPACE) as usize; + if index.len() != want { + bail!("index has {} entries, want {want}", index.len()); + } + let total: u64 = index.values().map(|v| v.len() as u64).sum(); + if total == 0 { + bail!("summed zero bytes"); + } + Ok(total) +} + +fn run(cfg: &Config) -> Result { + match cfg.mode { + "hot" => Ok(spin_hot(cfg.n)), + "alloc" => alloc_churn(cfg.n), + _ => unreachable!("parse_args only produces hot|alloc"), + } +} + +fn main() -> ExitCode { + let cfg = match parse_args() { + Ok(c) => c, + Err(e) => { + eprintln!("app: {e}"); + usage(); + return ExitCode::from(2); + } + }; + + let t0 = Instant::now(); + let result = run(&cfg); + let ms = t0.elapsed().as_millis(); + + match result { + Ok(r) => { + println!("app: mode={} n={} result={r} ms={ms}", cfg.mode, cfg.n); + ExitCode::SUCCESS + } + Err(e) => { + eprintln!("app: {e:#}"); + ExitCode::from(1) + } + } +} diff --git a/examples/31-per-language-toolbelts/verify.lua b/examples/31-per-language-toolbelts/verify.lua new file mode 100644 index 0000000..f2a788c --- /dev/null +++ b/examples/31-per-language-toolbelts/verify.lua @@ -0,0 +1,146 @@ +-- Verify toolbelt: both targets (hot/alloc) produce the exact single-line +-- report with numbers that parse to sane values, bad usage exits 2, and the +-- default sizes (3,000,000 / 200,000) are unchanged. Then each language's +-- native profiler is actually run against the binary and asserted to name +-- the hot function: +-- cpp/rust: perf record -> perf report --stdio names spin_hot +-- go: runtime/pprof CPU profile -> go tool pprof --top names +-- main.hotSpin; heap profile -> go tool pprof --top --cum +-- --alloc_space names main.allocChurn +-- plus a plain "perf stat" run on `app hot` parsing cycles/instructions, +-- which works identically for all three (perf attaches to any ELF binary). +-- Invoked from the example directory with LSP_LANG (cpp|go|rust) and +-- EXAMPLE_DIR set. + +local script_dir = arg[0]:match("(.*/)") or "./" +local repo_root = os.getenv("REPO_ROOT") or (script_dir .. "../..") +package.path = package.path .. ";" .. repo_root .. "/scripts/lib/?.lua" +local checks = require("checks") + +local lang = os.getenv("LSP_LANG") +if lang ~= "cpp" and lang ~= "go" and lang ~= "rust" then + checks.skip("LSP_LANG must be cpp|go|rust (got " .. tostring(lang) .. ")") +end + +local bin = ({ cpp = "cpp/build/release/app", go = "go/bin/app", rust = "rust/target/release/app" })[lang] + +if checks.run("test -x " .. bin).exit ~= 0 then + checks.skip(bin .. " missing — run './demo.sh " .. lang .. " build' first") +end + +if checks.run("command -v perf >/dev/null 2>&1").exit ~= 0 then + checks.skip("perf not found on PATH") +end + +local function demo(args) + return checks.run("./demo.sh " .. lang .. " run " .. args) +end + +-- Bad usage: missing/unknown mode, non-numeric --n, unknown flag -> usage on +-- stderr, exit 2. The diagnostic text and the shared " [--n N]" +-- prefix of the usage banner are identical across all three languages (Go's +-- banner has extra profiling flags appended, which the plain substring +-- match below tolerates). +local usage_pat = "usage: app %[%-%-n N%]" + +local r = demo("") +checks.expect_exit(r, 2, lang .. ": missing mode exits 2") +checks.expect_match(r.out, "missing mode", lang .. ": missing mode diagnostic") +checks.expect_match(r.out, usage_pat, lang .. ": missing mode prints usage") + +r = demo("bogus") +checks.expect_exit(r, 2, lang .. ": unknown mode exits 2") +checks.expect_match(r.out, "unknown mode: bogus", lang .. ": unknown mode diagnostic") + +r = demo("hot --n zero") +checks.expect_exit(r, 2, lang .. ": non-numeric --n exits 2") +checks.expect_match(r.out, "not a positive integer: zero", lang .. ": non-numeric --n diagnostic") + +r = demo("hot --bogus") +checks.expect_exit(r, 2, lang .. ": unknown flag exits 2") +checks.expect_match(r.out, "unknown argument: %-%-bogus", lang .. ": unknown flag diagnostic") + +-- The report line, parsed strictly. +local line_pat = "^app: mode=(%a+) n=(%d+) result=(%d+) ms=(%d+)\n$" + +local function check_run(args, want_mode, want_n, want_result) + r = demo(args) + checks.expect_exit(r, 0, lang .. ": '" .. args .. "' exits 0") + checks.expect_match(r.out, line_pat, lang .. ": '" .. args .. "' report is the exact single line") + local mode, n, result, ms = r.out:match(line_pat) + if not mode then + return + end + checks.expect_match(mode, "^" .. want_mode .. "$", lang .. ": '" .. args .. "' mode=" .. want_mode) + checks.expect_match(n, "^" .. want_n .. "$", lang .. ": '" .. args .. "' echoes n=" .. want_n) + checks.expect_match(result, "^" .. want_result .. "$", lang .. ": '" .. args .. "' result=" .. want_result) + local ms_n = tonumber(ms) + if ms_n and ms_n >= 0 and ms_n < 300000 then + checks.expect_match("ok", "ok", lang .. ": '" .. args .. "' ms sane (" .. ms .. ")") + else + checks.expect_match(ms or "", "impossible-ms", lang .. ": '" .. args .. "' ms sane") + end +end + +-- Defaults: identical across all three languages, including the numeric +-- result — same algorithm, same input, same answer. +check_run("hot", "hot", "3000000", "216816") +check_run("alloc", "alloc", "200000", "5000") + +-- Small sizes still work (n below the 1000-entry keyspace, and a tiny hot +-- count) — the arithmetic isn't hardcoded to the defaults. +check_run("hot --n 10", "hot", "10", "4") +check_run("alloc --n 7", "alloc", "7", "7") + +-- perf stat on `app hot`: works identically for any ELF binary. Parses two +-- counters; values are printed for the log but not bounded (host load +-- varies), only their presence and shape are asserted. +r = checks.run("perf stat -e cycles,instructions -- ./" .. bin .. " hot --n 500000 2>&1") +checks.expect_exit(r, 0, lang .. ": perf stat run exits 0") +local cycles = r.out:match("([%d,]+)%s+cycles") +local instrs = r.out:match("([%d,]+)%s+instructions") +if cycles and tonumber((cycles:gsub(",", ""))) then + checks.expect_match("ok", "ok", lang .. ": perf stat parses cycles (" .. cycles .. ")") +else + checks.expect_match(r.out, "cycles:?u?%s", lang .. ": perf stat reports cycles") +end +if instrs and tonumber((instrs:gsub(",", ""))) then + checks.expect_match("ok", "ok", lang .. ": perf stat parses instructions (" .. instrs .. ")") +else + checks.expect_match(r.out, "instructions:?u?%s", lang .. ": perf stat reports instructions") +end + +-- Per-language native profiler: run it for real, on the real binary, and +-- assert it names the hot/alloc-heavy function. +if lang == "cpp" or lang == "rust" then + local perf_data = os.tmpname() .. "-" .. lang .. ".perf.data" + r = checks.run(string.format( + "perf record -o %s --call-graph fp -- ./%s hot --n 1500000", perf_data, bin)) + checks.expect_exit(r, 0, lang .. ": perf record exits 0") + + r = checks.run(string.format("perf report -i %s --stdio", perf_data)) + checks.expect_exit(r, 0, lang .. ": perf report --stdio exits 0") + checks.expect_match(r.out, "spin_hot", lang .. ": perf report --stdio names spin_hot") + os.remove(perf_data) +elseif lang == "go" then + local cpu_prof = os.tmpname() .. "-go.cpu.prof" + r = checks.run(string.format("./%s hot --n 1500000 --cpuprofile %s", bin, cpu_prof)) + checks.expect_exit(r, 0, "go: --cpuprofile run exits 0") + + r = checks.run(string.format("go tool pprof --top --nodecount=5 %s %s", bin, cpu_prof)) + checks.expect_exit(r, 0, "go: go tool pprof --top (cpu) exits 0") + checks.expect_match(r.out, "main%.hotSpin", "go: cpu profile names main.hotSpin") + os.remove(cpu_prof) + + local mem_prof = os.tmpname() .. "-go.mem.prof" + r = checks.run(string.format("./%s alloc --n 200000 --memprofile %s", bin, mem_prof)) + checks.expect_exit(r, 0, "go: --memprofile run exits 0") + + r = checks.run(string.format( + "go tool pprof --top --cum --alloc_space --nodecount=5 %s %s", bin, mem_prof)) + checks.expect_exit(r, 0, "go: go tool pprof --top --cum --alloc_space (heap) exits 0") + checks.expect_match(r.out, "main%.allocChurn", "go: heap profile names main.allocChurn") + os.remove(mem_prof) +end + +checks.finish() diff --git a/examples/manifest.yaml b/examples/manifest.yaml index 94366b5..654c931 100644 --- a/examples/manifest.yaml +++ b/examples/manifest.yaml @@ -113,3 +113,19 @@ examples: langs: [cpp, go, rust] mode: local timeout: 360 + - id: 28-gdb-and-remote-debugging + langs: [cpp, go, rust] + mode: local + timeout: 300 + - id: 29-valgrind-sanitizers-miri + langs: [cpp, go, rust] + mode: local + timeout: 360 + - id: 30-ebpf-observation-toolkit + langs: [cpp, go, rust] + mode: vm + timeout: 360 + - id: 31-per-language-toolbelts + langs: [cpp, go, rust] + mode: local + timeout: 360