Repository navigation
docs(r09): complete Part 10-11 observability + low-latency (ch 36-40) - #9
Merged
Merged
Conversation
Examples 36-41 scaffolded and registered in manifest.yaml. Complete: 36 (tri-language), 39 (tri-language + chapter). Partial: 37, 38, 40, 41 — stub languages and verify.lua remain. Diagrams present for 36 and 39 only.
Behavioral contracts derived from the C++ reference in each example, 9 sequenced steps (VM steps serial, no concurrent cargo/cmake builds), 13 acceptance criteria, 8 risks. Checkpointed before execution so an interrupted session does not lose the plan.
Bans lie/lies/lying when the subject is the work, the method, or the reader; keeps it when the subject is a machine reporting a wrong value (ch07 mtimes, ch34 hardware_concurrency), which is accurate idiom. Adds the identifier guidance: name a deliberately-bad variant after its defect, never --lie, since flag names propagate into all three language sources, verify.lua, recorded chapter output, and the example slug.
S1/S2 of batch 1. Captures exact stdout/stderr/exit for every invocation the verify.lua files will assert against, on systems-target (2 vCPU, kernel 6.19.10-300.fc44). Calibration: ex40 fastpath beats naive by 1.42-1.49x across three independent bench runs (stable, never inverted); ex37 --resource mem exits 0 without OOM; analyze samplers confirmed concurrent (10.37s wall for --seconds 10, not ~50s). Records a suspected saturator bug in the ex37 C++ reference, pending verification.
…ntly std::jthread's destructor calls request_stop() *before* join(). Each sat_* function spawned its workers into a vector whose scope closed immediately after the spawn loop, so every worker saw stop_requested() on its first iteration and bailed in milliseconds -- the deadline check in each loop was dead code and --seconds N had no effect. Adds sleep_until(deadline) inside each pool scope so the destructor's stop request becomes the intended post-deadline signal. Affects all four saturators (cpu, mem, io, net). Verified on host: --resource cpu|io|net --seconds 3 now takes 3.00s wall (was milliseconds), with ops rising from near-zero to 2.9e10 iterations / 86k writes / 7.6M packets respectively. Also drops a banned word from the file header comment.
Supersedes the pre-f2b2dba ex37 captures, which recorded the broken instant-exit behavior. ex40 section untouched. Timings now honor --seconds: cpu 5.002s, io 5.012s, net 5.207s for --seconds 5 (was 0.007s/0.058s/0.220s). Load attribution is stable across 3 attempts -- busy_pct 1.34 idle to 100.00 under load, run_queue 3.00 to 7.00-8.00, Utilization fired=true, verdict=cpu with ratio 1.75-2.00, well clear of the proposed gate. mem re-tested under real sustained pressure: no OOM, guest stayed responsive, swap recovered within ~1s. New nuance -- mem can overrun its requested duration (8s -> 12.6s) because the inner touch loop only checks stop between passes, so verify.lua needs a generous upper bound for mem specifically.
Sets published: false on the two r09 batch-1 working files (execution plan and raw golden-capture dumps) so they stay in git for resumability without rendering as public pages. Does NOT exclude the _plans collection wholesale: iteration-plan.md is reader-facing and linked as "Roadmap" from _includes/footer.html:14 and _layouts/part_index.html:40, so dropping the collection would 404 both. Records that constraint in CLAUDE.md along with the published: false convention for future working plans.
Replaces the 17-line scaffold with 379 lines / 28 assertion call sites (50 runtime checks): error shapes, saturate mode for cpu/io/net/mem with worker-formula and ops and elapsed-duration bounds, the full 25-line analyze skeleton in order, 3-attempt load attribution, and sampler-concurrency wall-clock sanity. Gated on the C++ reference: PASS 50 / FAIL 0 on systems-target. Negative control -- staging /bin/true as app -- correctly fails PASS 9 / FAIL 41, so the suite is not satisfiable by exit 0 alone. The duration assertion is the one that would have caught the saturator bug fixed in f2b2dba.
Replaces the hello-syscall template check with behavioral assertions against systems-target: error shapes (exit 2 + usage), out-of-range --pin and measure connect/IPv4 errors (exit 1), the real run-latency-bench.sh, and SIGINT shutdown per server. Pinning is proven at the kernel level by reading Cpus_allowed_list from /proc/<pid>/status in the driver rather than trusting the program's own printed line. The perf gate compares median-of-three p50 per variant with a conservative 0.90 factor. Verified on systems-target: cpp PASS 46 / FAIL 0, with median(naive p50)=11287 ns vs median(fastpath p50)=7580 ns (1.489x), inside the 1.425-1.488x golden-capture band the threshold was calibrated against. Go and Rust remain the new-example.sh scaffold and fail this verify until ported.
Ports the 577-line C++ reference to Go with byte-identical observable
behavior: same usage text, same "app: ..." lines, same percentiles_ns
format. Verified on systems-target, PASS 28 / FAIL 0, median(naive
p50)=14818 ns vs median(fastpath p50)=9748 ns (1.52x).
Two deliberate divergences from the C++ port, both documented in the
file header:
- Shutdown: the C++ port uses a non-SA_RESTART SIGINT handler to break
a blocking read with EINTR. Go's netpoller parks the goroutine off
the syscall, so EINTR never arrives; instead a signal.Notify
goroutine sets the stop flag and closes the listener/conn to unblock
the parked read (the shutdown(2)-driven wakeup, Go-idiomatic). The
accept loop carries a 200ms deadline mirroring the C++ poll timeout.
- Range check: runtime.NumCPU() returns the affinity-MASK size, so
under the bench driver's `taskset -c 1` it reports 1 and rejected a
pin to CPU 1 as out of range. Replaced with onlineCPUs(), which
reads /sys/devices/system/cpu/online to match the C++ port's
sysconf(_SC_NPROCESSORS_ONLN) system-wide count. A count-lies-about-
-its-container bug hit via taskset rather than a cgroup.
Rust still the scaffold; ported next.
Ports the C++ reference to Rust with byte-identical observable behavior: same usage text, same "app: ..." lines, same percentiles_ns format. Verified on systems-target, PASS with median(naive p50)=12063 ns vs median(fastpath p50)=8578 ns (1.41x). This is the port that follows the C++ original most literally, because it can: Rust's std makes real blocking syscalls (no netpoller parking work off the syscall), so the C++ technique -- a non-SA_RESTART signal handler that lets a blocking read/poll return EINTR so the loop notices the stop flag -- works verbatim, where Go needed a close-the-fd wakeup. The servers are built on raw fds via nix (OwnedFd as the RAII Socket) rather than std::net, because std's accept and read_exact swallow EINTR internally, which would defeat the signal this example is about. Pinning is a single sched_setaffinity(0, ...) -- the process is single-threaded (tid == pid), so it both pins the hot loop and shows up in /proc/<pid>/status. Uses sysconf(_SC_NPROCESSORS_ONLN) for the range check to match C++ and avoid the taskset-masked count the Go port had to work around. Deps: dropped the hello-syscall scaffold's rustix/anyhow/core_affinity; nix now carries socket/net/poll/sched/signal/fs. All three languages of example 40 now pass the vm-mode verify gate.
Adds a Port findings section to the stub-completion plan capturing the
three Go divergences and the Rust verbatim-port note from S7/S8:
1. Go SA_RESTART/netpoller shutdown (predicted, Risk 3) -> close-fd
wakeup instead of EINTR.
2. Go /proc thread-group-leader affinity proof (predicted, Risk 2) ->
dual SchedSetaffinity(0) + SchedSetaffinity(pid).
3. Go runtime.NumCPU() masked by taskset (NOT predicted) -> the range
check rejected a pin to CPU 1 under `taskset -c 1`; fixed with
onlineCPUs() reading /sys to match C++ sysconf semantics. This is
the hardware_concurrency-lies-about-its-container theme reached via
an affinity mask rather than a cgroup, and is the finding worth
carrying into chapter 40's prose.
4. Rust follows C++ verbatim (real blocking syscalls); raw-fd nix
because std's accept/read_exact swallow EINTR.
Progress table: S4 verify.lua 40, S7 Go port 40, S8 Rust port 40 done;
S9 recorded as 40-complete, 37 pending. S1/S2 marked done per the
golden-capture companion plan.
Writes _docs/40-low-latency-fastpath.md to the chapter spine (hook →
Figure 40.1 → concept → how-the-code-works → errors → concurrency lens →
build/run/observe → cross-check → what-you-learned → verified footer),
with the three cross-language port findings woven into the prose:
- The taskset/runtime.NumCPU() range-check divergence anchors a
"count that lies about its container" section beside the pin_to_cpu
codetab — the affinity-mask door to the same bug hardware_concurrency
hits via a cgroup.
- The SA_RESTART/netpoller shutdown split drives the Concurrency lens:
C++/Rust unblock with a non-SA_RESTART EINTR handler, Go closes the
fd because no syscall is in flight to interrupt (codetab shows the
two handlers beside Go's trigger).
- Rust's verbatim-C++ raw-fd rationale (std's accept/read_exact
swallow EINTR) in the same lens.
The busy-poll thesis is proven from the kernel, not the program:
voluntary_ctxt_switches from /proc/<pid>/status moved 2->20500 (naive,
one sleep/wake per message) versus 2->3 (busy-poll) over a 20k-message
run this session — the chapter's centerpiece cross-check, captured on
systems-target. Adds Figure 40.1 (40-fastpath-datapath, paired
SVG+Excalidraw) and its catalogue entry. Status footer verified against
this session's runner PASS x3 and the real captures. validate.py OK.
Ports the 659-line C++ loadmix reference to Go with byte-identical observable behavior: same usage text, same loadmix:/analyze: line shapes, same worker-count formulas, same percentile/verdict output. Verified on systems-target, PASS 50 / FAIL 0 against the unchanged verify.lua. Saturators use goroutines + sync.WaitGroup + sync/atomic counters in place of C++ jthread/stop_token (each worker observes the deadline itself, so wg.Wait holds for the full --seconds with no sleep-before-join workaround). satIO uses x/sys/unix Open+Pwrite with O_DSYNC; satNet a net.UDPConn with SetReadDeadline. analyze shells out via `sh -c "( env LC_ALL=C LANG=C <cmd> ) 2>&1"` (the popen subshell trick), runs the five interval samplers concurrently, and parses every table by column name from its own header row. go.mod/go.sum unchanged.
Ports the C++ loadmix reference to Rust with byte-identical observable behavior. Verified on systems-target, PASS 50 / FAIL 0 against the unchanged verify.lua. Saturators use std::thread::scope as the jthread analogue (workers loop on their own deadline check, so no sleep-before-join hack is needed). sat_io uses OpenOptionsExt::custom_flags(O_DSYNC) + FileExt::write_at; sat_net nests two scopes so the UdpSocket receiver outlives the sender pool, mirroring the C++ join order. analyze shells out via `sh -c "( env LC_ALL=C LANG=C <cmd> ) 2>&1"`, runs the five samplers concurrently, and parses every table by column name. Cargo.toml: package renamed template-hello -> loadmix, rustix replaced with libc = "0.2" (cached; builds --offline), edition 2024 and [[bin]] name = "app" kept; Cargo.lock regenerated.
Replaces the hello-syscall template check with behavioral assertions for
lsp-otel (local mode, requires the lsp-lgtm stack). Builds, runs serve in
the background, drives N traced requests, SIGINTs serve to flush the SDK,
then proves the telemetry LANDED in the stack -- not the program's own
claim -- by querying through Grafana's datasource proxy:
- requests_total{service_name=lsp-otel-<lang>} >= N in Prometheus,
- request_duration_milliseconds_count{service_name=...} >= N,
- at least one trace with that service.name in Tempo.
Each language tags a distinct service.name so the three never collide in
the shared stack. Metrics/traces are polled with a bounded retry (SDK
batches ~2s + backend ingest lag). SKIPs when the stack is unreachable.
Verified against the Go reference: PASS 15 / FAIL 0.
Writes _docs/37-use-method-60s-analysis.md to the chapter spine (hook → Figure 37.1 → two-modes → parse-by-column-name → how-the-code-works → errors → concurrency lens → build/run/observe → cross-check → what-you-learned → verified footer). Part "Observability". Throughline: the USE method as a runnable checklist; parsing every tool's table by column name from its own header row (robust to sysstat/procps column reordering); running the five interval samplers concurrently so an N-second checklist costs ~N seconds not 5N; and the payoff — a concurrent analyze attributes a background --resource cpu saturator (busy_pct 1.34→100.00, run_queue 3→7-8, verdict cpu). Two codetabs (Row/make_row and parse_mpstat) verbatim across cpp/go/rust; the Concurrency lens explains the jthread request_stop-before-join fix and why Go/Rust (each worker checks its own deadline) never needed it. All numbers from the golden capture and this session's gate (PASS 50 / FAIL 0 per language on systems-target). Adds Figure 37.1 (37-use-method-checklist, paired SVG+Excalidraw) and its catalogue entry. validate.py OK.
Writes _docs/36-proc-sys-and-the-agent.md to the chapter spine. Part "Observability". sysagent v0 is a USE-method collector that reads /proc + /sys/fs/cgroup directly (no root, no shelled-out tools) — the deliberate contrast with chapter 37, which shells out to the real checklist tools. Throughline: the /proc/stat DELTA (two cumulative reads --interval-ms apart -> one utilization %), the six sources gathered concurrently with the sampling sleep, graceful degradation when PSI is absent (psi_available:false, not an error), and a byte-for-byte identical JSON schema across cpp/go/rust exposed by both `sample --json` and `serve /metrics`. Two verbatim codetabs (the /proc/stat delta and the concurrent-gather orchestration). Reuses the existing 36-proc-datasource-map figure. All numbers real (PASS 32 / FAIL 0 per language, local mode; the captured sample --json/serve output). validate.py OK.
Ports the Go OTLP reference to Rust with opentelemetry-otlp 0.32 (http-proto + reqwest-blocking-client, no tokio). Same serve/drive CLI, same output lines, same trace structure (request → work + respond), the requests_total counter and request_duration histogram, service.name lsp-otel-rust. Verified on the live LGTM stack: PASS 15 / FAIL 0 — metrics reach Prometheus and traces reach Tempo. Two fixes over the initial draft: the per-signal OTLP HTTP exporters need the full /v1/traces and /v1/metrics URLs (opentelemetry-otlp 0.32 does NOT append the signal path to an explicit with_endpoint, so posting to the bare base 404'd and the export silently failed); and dropped a TRACE-level tracing_subscriber that flooded stderr and diverged from the quiet Go/C++ ports (with the now-unused tracing/tracing-subscriber deps). Builds offline from the cached crates.
Ports the Go OTLP reference to C++23 with opentelemetry-cpp 1.24.0 (OTLP
HTTP exporter), completing a crashed first attempt. Same serve/drive CLI,
output lines, trace structure (request → work + respond), requests_total
counter and request_duration histogram, service.name lsp-otel-cpp.
Verified on the live LGTM stack: PASS 15 / FAIL 0 — metrics in Prometheus,
traces in Tempo.
Signal handling mirrors Go/Rust: a sigwait() thread feeds an EventChannel
the main loop selects over alongside accepted connections. Fixes worth
recording:
- full /v1/traces and /v1/metrics exporter URLs (the base 404s);
- a NoopLogHandler + LogLevel::None so the SDK's default stdout/stderr
logging doesn't break the two-line output contract;
- SIGINT was inherited as SIG_IGN from bash's background-job disposition
(cmd &), which pthread_sigmask+sigwait can't override — install a real
(no-op) sigaction before blocking, before telemetry::init so the SDK
export threads inherit the blocked mask too;
- Histogram::Record needs the (value, Context{}) overload under this
build's ABI v1;
- the otlp_http static archives have a genuine link cycle — bracket the
resolved library list with -Wl,--start-group/--end-group via a
CMAKE_CXX_LINK_EXECUTABLE override.
Builds offline from the conan cache.
Writes _docs/38-opentelemetry-to-lgtm.md to the chapter spine. Part
"Observability". lsp-otel is a traced request/response server exporting
OTLP traces + metrics to the LGTM stack: a parent `request` span wrapping
`work` + `respond` children, a requests_total counter, a request_duration
histogram, driven by an untraced `drive` client.
Throughline: the three OTel signal types from one request; parent/child
span nesting carried by context (Go/Rust explicit context value, C++ Scope
guard); batched export with a flush-on-shutdown; and proving the export
LANDED by querying Tempo (traces) and Prometheus (metrics) through
Grafana's proxy rather than trusting the program. Two verbatim codetabs
(the handle_request span pipeline and the signal+accept event-channel
select). The Concurrency lens records the three real portability lessons
from the ports: full /v1/{traces,metrics} URLs, silencing each SDK's
default logging, and the C++ SIG_IGN-inheritance signal fix.
All numbers real (PASS 15 / FAIL 0 per language; requests_total=10 and
request_duration_milliseconds_count=10 for all three; Tempo trace tree
request→{work,respond}). Adds Figure 38.1 (38-otlp-pipeline) and its
catalogue entry. validate.py OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes every chapter and example before the capstone (41).
Chapters authored
sysagent) — Observabilityloadmix) + Figure 37.1lsp-otel) + Figure 38.1Examples completed (Go + Rust + C++ ports, verify.lua, gated on real environments)
Verification
Each example gated this session with the runner (lab up / LGTM up): 36 PASS×3, 37 PASS×3, 38 PASS×3, 40 PASS×3.
scripts/validate.py: OK.Notable findings recorded in-tree
runtime.NumCPU()returns the affinity-mask size — ataskset-driven "count lies about its container" bug (ex 40); fixed with a/sysread./v1/{traces,metrics}URLs; each SDK's default logging must be silenced to hold the output contract (ex 38).