diff --git a/_docs/02-lab-setup.md b/_docs/02-lab-setup.md index e3ceba1..6e9f202 100644 --- a/_docs/02-lab-setup.md +++ b/_docs/02-lab-setup.md @@ -185,8 +185,77 @@ it as `OTEL_EXPORTER_OTLP_ENDPOINT` through `env` explicitly, because `sudo` strips the environment — without that, a privileged guest binary would fall back to its own localhost and its telemetry would never reach the host's observability stack. This works because host builds and the guest are both -Fedora 44 (glibc parity); the in-VM toolchains are the fallback when that -assumption breaks. +Fedora 44. That parity holds when the guests are provisioned and lapses as the +host keeps updating, which "When the guest falls behind the host" describes. + +### When the guest falls behind the host + +Deploying a host build assumes the guest can still run it. Both machines start +as Fedora 44, so they can. Then the host takes updates, as workstations do, +while the guest stays where its snapshot left it. From then on the host +compiler emits binaries against runtime symbols the guest does not have. + +Here is that failure, exactly as it appeared on this lab: + +``` +[host]$ TARGET=systems-target ./demo.sh cpp run +→ copying app to fedora@192.168.124.7:/home/fedora/app +/home/fedora/app: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.36' not found (required by /home/fedora/app) +``` + +**How it presents.** Run through a demo's own harness, the binary never +starts, and every assertion downstream reports a symptom of the demo: a port +that never opened, a Landlock line that never printed, an OOM kill that never +happened. The loader message above names the cause, and it appears only when +you run the binary directly. + +**The tell is which languages fail.** On the run that found this, six examples +failed on the guest and every one of them failed in C++ alone, with Go and Rust +passing on the same guest in the same run. The asymmetry has a cause: Go and +Rust link their runtimes statically, so the guest's libraries do not affect +them, while the C++ binaries load the guest's `libstdc++`. When one language +fails on the guest and the others pass, check the runtime before the code. + +**Compare symbol versions, which the loader checks.** Compiler versions can +agree while the runtimes differ, because the compiler and the runtime are +separate packages that update independently. On this host, `g++ --version` +said 16.1.1 and the guest's `rpm -q libstdc++` said 16.1.1, while the host's +own libstdc++ had moved to 16.2.1. The highest `GLIBCXX_` version on each side +shows the gap: + +```bash +[host]$ strings /lib64/libstdc++.so.6 | grep -oE 'GLIBCXX_3\.4\.[0-9]+' | sort -V | tail -1 +GLIBCXX_3.4.36 +[vm]$ strings /lib64/libstdc++.so.6 | grep -oE 'GLIBCXX_3\.4\.[0-9]+' | sort -V | tail -1 +GLIBCXX_3.4.35 +``` + +Guest lower than host means host builds cannot run there. The same check works +for glibc with `GLIBC_` in place of `GLIBCXX_`. + +The fix is to bring the guest forward and then **re-snapshot it**, so that the +next revert keeps the update: + +```bash +[vm]$ sudo dnf -y update libstdc++ libgcc +[host]$ ./scripts/lab/snapshot-vm.sh systems-target lab-ready +``` + +Apply it to **every** guest. On this lab `systems-peer` carried the same drift +with no symptom, because no C++ binary is deployed to it; re-snapshotting it +unchanged would have kept the drift for the first example that needs it. + +Then confirm the fix survives a revert by reverting and re-running: + +```bash +[host]$ ./scripts/lab/revert-vm.sh systems-target lab-ready +[host]$ python3 scripts/test-all-examples.py --mode vm --lang cpp +8 passed, 0 failed, 0 skipped +``` + +An update applied only to the running guest lasts until the next revert. The +in-VM toolchains remove the question altogether: a binary built on the guest +links against the guest's own runtime. ## Build, run, observe @@ -258,10 +327,16 @@ guest's own `uname` in your terminal. - Snapshot `lab-ready` after provisioning, revert after destructive demos — and trust `/var/log/lab-ready` plus a guest kernel string in your output, not assumptions. +- **Host/guest parity holds at provisioning and drifts afterwards.** A snapshot + fixes the guest's packages while the workstation keeps updating, and host-built + binaries then fail on the guest for a missing `GLIBCXX_` or `GLIBC_` symbol, + which the demo harness reports as a failing demo. The tell is C++ failing where + Go and Rust pass, because those two link statically. Compare symbol versions, + update every guest, and **re-snapshot** so the next revert keeps the fix. Next, the third and final piece of setup: the Podman **LGTM observability stack**, so every later demo has somewhere to send its telemetry. --- -

verified — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); /var/log/lab-ready read back bpftrace v0.24.2 and kernel 6.19.10-300.fc44 on both; lab-ready snapshots exist on both; and deploy-to-vm.sh ran the host-built C++ and Go binaries on systems-target, printing the guest kernel string 6.19.10-300.fc44.x86_64 versus 7.1.3-200.fc44.x86_64 locally.

+

verified — both guests were provisioned from Fedora-Cloud-Base-Generic-44-1.7 on this host (target 4 GB at 192.168.124.7, peer 2 GB at 192.168.124.95); /var/log/lab-ready read back bpftrace v0.24.2 and kernel 6.19.10-300.fc44 on both; lab-ready snapshots exist on both; and deploy-to-vm.sh ran the host-built C++ and Go binaries on systems-target, printing the guest kernel string 6.19.10-300.fc44.x86_64 versus 7.1.3-200.fc44.x86_64 locally. The “When the guest falls behind the host” section was added later and is verified separately, on the same lab: the host’s libstdc++ reached 16.2.1-2.fc44 (libstdc++.so.6.0.36, max GLIBCXX_3.4.36) while both guests sat at 16.1.1-2.fc44 (libstdc++.so.6.0.35, max GLIBCXX_3.4.35) behind snapshots taken before the update, and every host-built C++ binary then failed on the guest with the quoted version `GLIBCXX_3.4.36' not found. A full-matrix run reproduced the asymmetry the section describes — six VM examples failing in C++ while Go and Rust passed on the same guest in the same run — and the quoted diagnostic commands, the dnf fix, and the re-snapshot were all run here: after revert-vm.sh systems-target lab-ready the guest reported GLIBCXX_3.4.36 and test-all-examples.py --mode vm --lang cpp reported the quoted 8 passed, 0 failed, 0 skipped, including the two-guest 41-capstone-fleet. systems-peer was confirmed to carry the same drift while showing no symptom. Not exercised: unverified — the GLIBC_ form of the symbol check is stated by analogy and was not the failure encountered here, which was GLIBCXX_ only; and building inside the guest with the in-VM toolchains is named as the standing fallback but is not exercised by any example in this book.