test(bootstrap): integrate cache hygiene diagnostics

[dirvine]

Linear issue

V2-884 — Age-bound and refresh the saorsa-core close-group bootstrap cache

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

This is a draft integration/testnet PR. It pins the exact saorsa-core #149 head so an ant-node binary can exercise that change, and adds logging-only ant-node bootstrap-state diagnostics.

Compatibility

• Wire: none.
• Storage: none; saorsa-core feat: commitment-bound quote pricing (ADR-0004) #149 retains the existing close-group cache format.
• API: none in ant-node. The patched saorsa-core API/semver impact remains documented on feat: commitment-bound quote pricing (ADR-0004) #149.

Semver impact

• breaking
• feature
• fix

Test evidence

• Cargo resolves exactly one saorsa-core, shared by ant-node and ant-protocol, at saorsa-core feat: commitment-bound quote pricing (ADR-0004) #149 head e74a1bf4085cbd175136b331a0ecfb9c7e423752.
• cargo check --all-targets --all-features — passed.
• cargo check --no-default-features — passed.
• cargo check --no-default-features --features logging — passed.
• cargo test --lib replication:: — 565 passed, 0 failed.
• Strict clippy (panic, unwrap_used, expect_used denied) — passed.
• cargo fmt --all -- --check and git diff --check — passed.
• cargo build --release --no-default-features --features logging --bin ant-node — passed on macOS arm64. Local binary SHA-256: f09c794988f3abf0485703e0460761c79cdb7778c63282f03a18e1edbf6566ff.

A Linux GNU release build and dev-testnet run remain required before rollout. Detailed matrix: V2-894.

Testnet evidence map

The patched saorsa-core emits:

• fresh cache age and saved timestamp;
• explicit stale-cache rejection;
• periodic cache-save results during normal running;
• INFO dial outcomes tagged bootstrap_source=cache|configured and outcome;
• reason-labelled cache saves (periodic|post_bootstrap|shutdown);
• an INFO Bootstrap reachability summary with cache/configured candidate and success counts.

This PR additionally emits Replication bootstrap state snapshot every 60 seconds in logging builds with:

• drained;
• pending_peer_requests;
• pending_keys;
• capacity_rejected_sources;
• is_bootstrapping;
• snapshot_state=pending|drained_waiting_for_bootstrap_flag|healthy;
• uptime_secs.

Snapshots stop after the first healthy sample to avoid permanent fleet log volume.

The saorsa-core fields directly verify #149. The ant-node fields do not prove cache refresh themselves; they separate a remaining replication-drain wedge from stale-cache or general connectivity failures.

New dependency

none. A temporary [patch.crates-io] source override pins the existing saorsa-core dependency to #149’s exact Git revision.

ADR

saorsa-core ADR-016 — Age-Bounded, Periodically Refreshed Close-Group Cache — Proposed; human acceptance pending.

Mitigation / rollback

Revert this integration commit or remove the crates.io patch and logging loop; no stored or wire data migration is required.

[jacderida]

Testnet validation — every marker fires; one diagnostic-semantics finding

Fault-injection run on a dev testnet, 2026-08-10 (V2-894), using this branch @ 3c3e12a with its [patch.crates-io] pin on saorsa-core#149 head a4faf6f7 (verified equal at run start). Topology: backbone of genesis + 5 bootstraps + 45 WAN nodes, plus 18 single-service probe VMs (12 probes, 3 untouched controls, 3 spares) — one ant-node service per probe VM so each iptables fault is scoped to exactly one node.

The cache-hygiene results are on WithAutonomi/saorsa-core#149. This comment covers the diagnostics integrated here.

Markers: all present and correctly fielded

Every marker in the evidence dictionary fired with its documented fields, and the forwarding path (JSON logs → Elasticsearch) carried all of them — they are INFO/WARN, so nothing was dropped. Replication bootstrap state snapshot emitted drained, pending_peer_requests, pending_keys, capacity_rejected_sources, is_bootstrapping, snapshot_state, uptime_secs as specified, on the 60 s cadence, with the stopping line after the first healthy sample.

The commit field on every log line was genuinely useful here: this branch builds as 0.17.0, byte-identical in version string to the shipped v0.17.0 release, so --version cannot distinguish the branch build from the release. commit=3c3e12a on every document was the only in-band proof the right binary was running.

---

Finding 1 — a totally unreachable node reports snapshot_state=healthy, then stops reporting

Under a full non-loopback UDP partition in both directions, held 18 minutes, all three probes produced one snapshot line each:

01:07:35  Bootstrap reachability summary  cache_dial_successes=0,
          configured_dial_successes=0, outbound_bootstrap_successes=0,
          outbound_reachable=false
01:07:40  Failed to connect to any bootstrap peers
01:08:40  Replication bootstrap state snapshot  drained=true,
          is_bootstrapping=false, pending_keys=0, snapshot_state=healthy
01:08:40  Replication bootstrap diagnostics reached healthy state;
          stopping periodic snapshots

Identical on P07 (01:08:40.335Z), P08 (01:08:41.291Z), P09 (01:08:42.661Z). Confirmed against the on-disk JSON logs as well as Elasticsearch, so this is not a forwarding gap — the node really did emit one snapshot and stop.

Mechanism. Bootstrap failed outright, so replication bootstrap never began. With nothing pending, drained=true and is_bootstrapping=false — which is exactly how snapshot_state derives healthy:

let snapshot_state = if drained && !bootstrapping { "healthy" }
                     else if drained { "drained_waiting_for_bootstrap_flag" }
                     else { "pending" };

Why it's worth addressing:

1. A node that cannot reach the network at all self-reports healthy. That is the most severe state a node can be in, reported as the least severe.
2. Having reported healthy, it stops emitting replication diagnostics permanently for that process — so anything watching this signal goes quiet precisely when the node is most broken. Silence is indistinguishable from "still healthy".
3. The distinction the diagnostic exists to make — drain debt on a connected node vs. something else — is inverted in this case: the state most in need of a signal produces the "all clear".

The saorsa-core Bootstrap reachability summary reports this state correctly and unambiguously, so the information is available; it just isn't reflected in snapshot_state. Two options that would preserve the current semantics while closing the gap: gate the healthy transition on having actually bootstrapped (rather than on the absence of pending work), or keep the loop alive when outbound_reachable=false so the condition remains visible.

Practical note for anyone writing test procedures against this: a step of the form "hold the fault ≥3 minutes (≥3 snapshot ticks)" is unsatisfiable under total unreachability — there will only ever be one tick.

Finding 2 — snapshot_state=healthy with non-zero pending_keys

Observed on 5 of 12 probes, in cases unrelated to the partition: pending_keys = 176 (P02), 278 (P10), 395 (P11), 508 (P12), and 1070 on a probe in an earlier attempt — all while drained=true, is_bootstrapping=false, snapshot_state=healthy.

Consistent with the derivation above and not a failure against any stated criterion. Raised only because the field pairing invites the opposite reading: "healthy" does not imply the replication queue is empty, and a reader glancing at these two fields together would reasonably conclude otherwise.

Where the diagnostics performed well

Case 2 is the good case for this instrumentation. With all cached endpoints blackholed but configured ones reachable, the source labelling made the situation readable at a glance: 38–39 Bootstrap dial failed with bootstrap_source=cache, outcome=connect_error, then configured successes, in one summary line. The whole dial phase took ~48 seconds — a blocked-cache start is diagnosable from the reachability summary in under a minute, and any longer wait is drain, not bootstrap. (In an earlier run one probe took 32 minutes to reach healthy under the identical fault; the snapshot sequence showed pending_keys falling 723 → 645 → … → 196 → 0 across 31 ticks, which correctly attributed the delay to replication drain rather than bootstrap.)

Coverage

All four cases pass on all 3 probes each. Controls healthy at six boundaries with zero restarts; iptables -S archived clean on all 12 probes; backbone transfers 101 uploads ok / 1 failed, 120 downloads ok / 0 failed.

Full per-case evidence: V2-895, V2-896, V2-897, V2-899.