Wire tracing diagnostics through libazureinit-kvp by peytonr18 · Pull Request #319 · Azure/azure-init

Peyton Robertson (peytonr18) · 2026-09-24T19:19:22Z

Tracing, Diagnostics, and KVP Integration

Azure Init now uses libazureinit-kvp for diagnostics and provisioning reports instead of maintaining its own KVP encoder and writer. The reusable, synchronous DiagnosticsKvp bridge handles tracing; the application owns logging policy and report orchestration.

Architecture

Component	Responsibility
`libazureinit-kvp`	Pool I/O and locking, `DIAG` encoding/chunking, the report model and writer, CLI, and optional tracing bridge
`libazureinit`	Provisioning operations, Azure error-to-report mapping, and the HTTP-only `wireserver` module
`azure-init`	Subscriber setup, configuration and filters, pool cleanup, and final report delivery

Two KVP paths share the same store: diagnostics append operation history; the provisioning report replaces the final status. HTTP reporting is a separate transport.

flowchart TD
    APP["Azure Init"] --> TRACE["Tracing spans and events"]
    TRACE --> LAYER["DiagnosticsKvp Layer<br/>capture and emit"]
    LAYER --> WRITER["DiagnosticWriter<br/>validate, encode, chunk"]
    WRITER -->|append| STORE["KvpPoolStore<br/>flock + OFD fcntl"]
    APP --> RESULT["Provisioning result<br/>one typed report"]
    RESULT --> PUBLISH["publish_provisioning_report<br/>tokio::join!"]
    PUBLISH --> WRITE["write_report<br/>spawn_blocking"]
    PUBLISH --> HTTP["wireserver<br/>report_ready<br/>report_failure"]
    WRITE -->|upsert| STORE
    STORE --> POOL["Guest pool 1<br/>.kvp_pool_1"]
    POOL -. host reads later .-> HOST["hv_kvp_daemon<br/>kernel and Hyper-V host"]
    HTTP --> WS["Azure wireserver"]

Runtime Behavior

Diagnostics: on_new_span and on_close produce start/finish records with a shared operation UUID; on_record retains field updates. Events receive independent UUIDs, including events outside spans. Payloads contain the tracing target, level, and recorded fields as JSON text.

Span outcomes use an explicit diagnostic.result, otherwise infer failure from observed ERROR events and success when none were observed. Detected unwinding reports failure. This is tracing-based inference: filtering and missing instrumentation can hide errors.

Synchronous delivery: callbacks write through DiagnosticsKvp::emit() and the existing DiagnosticWriter validation and framing. No runtime, worker, queue, or drain lifecycle is required by the bridge. Callback failures go to stderr to avoid recursive tracing. Lock contention can delay callers; successful local writes do not guarantee durability or host receipt.

Final status: the binary constructs one ProvisioningReport from the actual provisioning result and attempts KVP and HTTP delivery with tokio::join!. write_report(&store, &report) runs through spawn_blocking and upserts PROVISIONING_REPORT. Failure HTTP requests reuse the report's encoding and timestamp; successful HTTP requests remain state-only.

Final reports bypass tracing filters and remain available if diagnostic-writer initialization fails after pool preparation. Both delivery results are observed without changing the provisioning exit code. Existing report-size and store-capacity limits remain in effect.

Startup and Configuration

main uses a scoped stderr-only bootstrap subscriber while obtaining the VM ID and loading configuration.
setup_layers runs through spawn_blocking, prepares the log file, and performs explicit stale-pool cleanup when KVP is enabled.
main installs the configured subscriber globally. Defaults are stderr at ERROR, the private 0600 log file at DEBUG, and KVP at INFO (INFO/WARN/ERROR).

AZURE_INIT_LOG controls file/console verbosity independently. KVP filter precedence is AZURE_INIT_KVP_FILTER, then telemetry.kvp_filter, then info; empty or invalid values fall through to the next source.

telemetry.kvp_diagnostics = false disables KVP setup, cleanup, diagnostics, and final-report writes without disabling local logs or wireserver HTTP. The reusable bridge itself installs no subscriber, chooses no filter, and performs no implicit cleanup; it is available behind the crate's tracing feature.

Cleanup and CI

Remove the legacy libazureinit KVP, health, and logging modules and their unused OpenTelemetry, fs2, and sysinfo dependencies. The binary retains sysinfo for OS/kernel information.
Update instrumentation and callers for the new diagnostics and wireserver APIs. CLI cleanup tests disable KVP to avoid touching the host pool.
Run KVP CLI conformance checks in a separate testinit CI step using scratch pools, without launching extra provisioning runs.
Use a private container tmpfs for the agent pool and systemd results for service completion. Collect parsed telemetry through the shipped CLI after the run, replacing the mock wireserver's POST-time snapshot. Container checks cover local pool behavior, not host retrieval.

Codecov Comments Bot (codecov-commenter) · 2026-09-24T19:22:37Z

Codecov Report

❌ Patch coverage is 97.87956% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.34%. Comparing base (c6d1301) to head (2757b2a).

Files with missing lines	Patch %	Lines
src/main.rs	88.78%	25 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #319      +/-   ##
==========================================
+ Coverage   96.66%   99.34%   +2.68%     
==========================================
  Files          29       29              
  Lines        9960     9673     -287     
==========================================
- Hits         9628     9610      -18     
+ Misses        332       63     -269

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Peyton Robertson (peytonr18) · 2026-09-24T22:41:53Z

Codecov Report

❌ Patch coverage is 95.50459% with 49 lines in your changes missing coverage. Please review. ✅ Project coverage is 98.96%. Comparing base (c6d1301) to head (47ed958).

Files with missing lines Patch % Lines
src/main.rs 83.83% 27 Missing ⚠️
src/logging.rs 86.50% 22 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #319      +/-   ##
==========================================
+ Coverage   96.66%   98.96%   +2.30%     
==========================================
  Files          29       29              
  Lines        9960     9596     -364     
==========================================
- Hits         9628     9497     -131     
+ Misses        332       99     -233     
☔ View full report in Codecov by Harness. 📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

I can circle back to this and give it another shot, but there are pieces of main.rs specifically that I don't think we can "cover" without writing strange production code. Most of the functionality in this file is covered by integration tests, but those aren't counted towards "line coverage".

I'll give updating coverage in these two files another shot, but I don't want to sacrifice better and clearer production code for the sake of hitting 100% coverage. I don't think there's value in that, though I'm open to suggestions if others disagree.

In the meantime, I'll try my best to get us closer to 100% here.

…ble helpers to improve coverage

Copilot

Copilot review overview

🟡 Changes recommended

Final KVP reports can be lost, HTTP status handling does not follow the intended policy, and the new proxy-based test cannot reach its listener.

Review effort: Balanced
Findings: 6

Open (6)

Invalid diagnostic results suppress failure recording during unwinding · New
HTTP status classification is bypassed by premature error handling · New
Stale cleanup and skip path can leave no final provisioning report · New
Diagnostic history can exhaust capacity and block final report publication · New
Malformed TOML is misclassified as an sshd configuration failure · New
Missing system-proxy support breaks the configuration failure report test · New

What changed in this PR

This PR moves Azure Init’s KVP diagnostics and final provisioning reports to libazureinit-kvp, while keeping HTTP health reporting in libazureinit and subscriber setup in the binary.

Changes:

Add a synchronous tracing-to-KVP bridge and typed final reports.
Move logging setup and report delivery into azure-init; separate wireserver HTTP reporting.
Update container checks, CLI verification, CI artifacts, and documentation.

File	Description
`tests/functional_tests.rs`	Uses the wireserver reporting API.
`tests/cli.rs`	Adds failure-report tests and disables KVP for cleanup tests.
`testinit/verify_kvp.py`	Adds scratch-pool CLI checks.
`testinit/testing-server/wireserver_handler.py`	Removes POST-time pool snapshots.
`testinit/testing-server/test_healthcheck.py`	Tests mock-server readiness probes.
`testinit/testing-server/healthcheck.py`	Probes both mock listeners without HTTP requests.
`testinit/testing-server/docker-compose.yml`	Uses the new readiness probe.
`testinit/start-all.sh`	Waits for server readiness and the agent’s systemd result.
`testinit/README.md`	Documents the updated container checks.
`testinit/Dockerfile`	Builds and installs the agent and KVP CLI.
`testinit/docker-compose.yml`	Gives the agent a private KVP tmpfs.
`src/main.rs`	Orchestrates logging, provisioning, and final reports.
`src/logging.rs`	Configures stderr, file, and KVP tracing layers.
`libazureinit/src/wireserver.rs`	Provides HTTP-only health reporting.
`libazureinit/src/status.rs`	Updates VM-ID and status tracing.
`libazureinit/src/provision/user.rs`	Updates user-provisioning tracing.
`libazureinit/src/provision/ssh.rs`	Updates SSH-provisioning tracing.
`libazureinit/src/provision/password.rs`	Updates password-provisioning tracing.
`libazureinit/src/provision/mod.rs`	Updates provisioning span instrumentation.
`libazureinit/src/provision/hostname.rs`	Records hostname errors in tracing.
`libazureinit/src/media.rs`	Updates media-operation tracing.
`libazureinit/src/logging.rs`	Removes legacy library logging.
`libazureinit/src/lib.rs`	Exports wireserver in place of legacy modules.
`libazureinit/src/kvp.rs`	Removes the legacy KVP writer.
`libazureinit/src/http.rs`	Updates HTTP request tracing.
`libazureinit/src/health.rs`	Removes combined HTTP and KVP reporting.
`libazureinit/src/error.rs`	Maps errors to typed provisioning reports.
`libazureinit/src/config.rs`	Clarifies telemetry settings and tracing.
`libazureinit/Cargo.toml`	Adds the KVP library and removes unused dependencies.
`libazureinit-kvp/tests/diagnostics.rs`	Tests tracing records through the public API.
`libazureinit-kvp/src/vm_id.rs`	Raises empty VM-ID logging to a warning.
`libazureinit-kvp/src/report.rs`	Exposes report encoding.
`libazureinit-kvp/src/lib.rs`	Exports the optional tracing bridge.
`libazureinit-kvp/src/diagnostics/writer.rs`	Supports captured diagnostic timestamps.
`libazureinit-kvp/src/diagnostics/tracing_layer.rs`	Implements synchronous tracing delivery.
`libazureinit-kvp/src/diagnostics/mod.rs`	Gates and exports the tracing layer.
`libazureinit-kvp/Cargo.toml`	Adds the optional tracing feature.
`doc/kvp.md`	Updates pool-write documentation.
`doc/diagnostics.md`	Documents tracing-produced diagnostics.
`doc/configuration.md`	Documents KVP filters and logging defaults.
`Cargo.toml`	Enables KVP tracing for the binary.
`.github/workflows/e2e-testing.yml`	Adds CLI verification and telemetry artifacts.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        let explicit = match state.fields.outcome() {
+            Ok(result) => result,
+            Err(error) => {
+                eprintln!("Dropped KVP diagnostic finish: {error}");
+                return;
+            }
+        };
+        // A span closing during a panic reports failure even without an ERROR.
+        let result = if std::thread::panicking() {
+            Outcome::Failure
+        } else {
+            explicit.unwrap_or(if state.saw_error {
+                Outcome::Failure
+            } else {
+                Outcome::Success
+            })
+        };


+
+    let mut remaining = retry_for;
+    while !remaining.is_zero() {
+        let (response, new_remaining) = http::post(


+    let kvp_layer = if config.telemetry.kvp_diagnostics {
+        match KvpPoolStore::new_in(KvpPool::Guest, kvp_dir, PoolMode::Safe)
+            .and_then(|store| {
+                store.clear_if_stale()?;


+        if let Some(store) = store {
+            let store = store.clone();
+            let report = report.clone();
+            tokio::task::spawn_blocking(move || write_report(&store, &report))


            let err = LibError::LoadSshdConfig {
                details: format!("{error:?}"),
            };


+        .env("HTTP_PROXY", &proxy)
+        .env("http_proxy", &proxy)
+        .env("NO_PROXY", "")
+        .env("no_proxy", "")


Cade Jacobson (cadejacobson)

Adding an initial review to provide feedback, but this is looking awesome so far! Testinit is showing very solid output!

Cade Jacobson (cadejacobson) · 2026-09-30T20:47:11Z

                self.disable_password_authentication,
            ) {
-                tracing::error!(
+                tracing::debug!(


Since this section is only reached in the event that ssh_config_update_required is true, this tracing message showing a failure in expected configuration is likely relevant to a user when looking at the KVP file. Especially since the default filter level is Info which debug falls below, this failure may not be collected in that case. I feel the original error level may be best here, but my interpretation could be incorrect here!

Agreed, addressed in latest!

Cade Jacobson (cadejacobson) · 2026-09-30T21:06:10Z

+            "No authorizedkeysfile setting found in sshd configuration"
+        );
+    } else {
+        tracing::Span::current().record("diagnostic.result", "success");


Could we use Outcome::Success or Outcome::Success.to_str() (whatever helps fit the Rust compile checker) here so that we can avoid passing plain strings? It looks like we check if it matches the string in that spot.

Absolutely. I added an as_str() method to the Outcome enum and updated all related callers to use it instead of hard-coded outcome strings.

Cade Jacobson (cadejacobson) · 2026-09-30T21:07:56Z

+    fn outcome(&self) -> Result<Option<Outcome>, &'static str> {
+        match self.0.get(OUTCOME_FIELD) {
+            None => Ok(None),
+            Some(Value::String(value)) if value == "success" => {


Similar here, would it make sense to avoid the hard coded string since we do already provide the enum?

Agreed that it would make sense -- addressed in latest!

Cade Jacobson (cadejacobson) · 2026-09-30T21:10:23Z

+                Ok(Some(Outcome::Success))
+            }
+            Some(Value::String(value)) if value == "fail" => {
+                Ok(Some(Outcome::Failure))


I believe I missed the discussion on the previous PR: what was the reason for the Outcome::Success -> success but Outcome::Failure -> fail instead of failure?

We can update the in line comment to include a one line justification potentially as well 😁

Good question!!!! I went back and checked this too.

Failure is just the name of our Rust enum variant, while the DIAG format from the earlier PR uses fail. Cloud-init reports SUCCESS/FAIL, which our reader maps to success/fail, so using fail here keeps the parsed outcome consistent with that vocabulary.

There isn't anything in Rust that requires us to use fail instead of failure, I just originally chose it to match the cloud-init contract. For now, I've kept fail and added a short comment next to the enum to make that mapping clearer, but I'm definitely open to changing it if you think failure would be better!

Cade Jacobson (cadejacobson) · 2026-09-30T21:17:56Z

    let mount_result = mount_fn();
    if let Err(ref e) = mount_result {
-        tracing::error!(error = ?e, "Failed to mount media.");
+        tracing::debug!(error = ?e, "Failed to mount media.");


This likely falls under the same discussion as the earlier SSHD config error message so this comment may be a duplicate, but this message may also be important enough to link as a warning to avoid filtering from the Info level default.

Agreed, addressed in latest!

Cade Jacobson (cadejacobson) · 2026-09-30T21:18:11Z

    let unmount_result = unmount_fn(mounted);
    if let Err(ref e) = unmount_result {
-        tracing::error!(error = ?e, "Failed to remove media.");
+        tracing::debug!(error = ?e, "Failed to remove media.");


ditto, copying the same here:
This likely falls under the same discussion as the earlier SSHD config error message so this comment may be a duplicate, but this message may also be important enough to link as a warning to avoid filtering from the Info level default.

Agreed, addressed in latest!

Cade Jacobson (cadejacobson) · 2026-09-30T21:18:53Z

    let unmount_result = unmount_fn(mounted);
    if let Err(ref e) = unmount_result {
-        tracing::error!(error = ?e, "Failed to remove media.");
+        tracing::debug!(error = ?e, "Failed to remove media.");


Would it make sense to update remove to unmount?

tracing::error!(error = ?e, "Failed to unmount media.");

Yes! That got changed along the way, but it's back to unmount!

Cade Jacobson (cadejacobson) · 2026-09-30T21:22:55Z

-                    "VM ID (Gen1, swapped): {}",
-                    final_id
-                );
+                tracing::Span::current().record("diagnostic.result", "success");


Could we consume the const OUTCOME_FIELD: &str = "diagnostic.result"; from the tracing layer to avoid needing to copy/paste the same diagnostic.result string?

Yep, good call! I moved OUTCOME_FIELD out so we can reuse it for these calls instead of hardcoding "diagnostic.result" in multiple places."

I also kept the constant independent of the optional tracing feature, so producers can use it without needing a dependency on the tracing layer.

Cade Jacobson (cadejacobson) · 2026-09-30T21:25:16Z

        let stderr = String::from_utf8_lossy(&output.stderr);
        let stdout = String::from_utf8_lossy(&output.stdout);
-        tracing::error!(
+        tracing::debug!(


ditto, copying the same here:
This likely falls under the same discussion as the earlier SSHD config error message so this comment may be a duplicate, but this message may also be important enough to link as a warning to avoid filtering from the Info level default.

Feel free to resolve all of these debug/error comments when we come to a conclusion, I just figured I would point to the ones I see.😁

These are helpful! I agreed with the findings for all of them, so I restored them to error. It's a good call out!

…t so the final report always fits

refactor(diagnostics): route tracing to KVP via libazureinit-kvp.

ff77b60

Peyton Robertson (peytonr18) added 2 commits September 24, 2026 13:00

Fix(testinit): fix azure-provisioning-agent not found

cc634c3

test: improve coverage for tracing, logging, and provisioning reports

47ed958

test: extract KVP setup and provisioning-report logic into unit-testa…

2757b2a

…ble helpers to improve coverage

Peyton Robertson (peytonr18) force-pushed the probertson-wire-tracing branch from dfcca71 to 2757b2a Compare September 24, 2026 23:32

Peyton Robertson (peytonr18) requested a balanced review from Copilot September 30, 2026 20:18

Copilot started reviewing on behalf of Peyton Robertson (peytonr18) September 30, 2026 20:19 View session

This was referenced Sep 30, 2026

build(deps): update opentelemetry_sdk requirement from 0.32 to 0.33 #323

Closed

build(deps): update opentelemetry requirement from 0.32 to 0.33 #322

Closed

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Cade Jacobson (cadejacobson) reviewed Sep 30, 2026

View reviewed changes

Peyton Robertson (peytonr18) added 2 commits October 7, 2026 14:20

test: expand agent coverage and address tracing review feedback

e27565c

feat(kvp): reserve PROVISIONING_REPORT slot with an in_progress repor…

ae42777

…t so the final report always fits

Conversation

Peyton Robertson (peytonr18) commented Sep 24, 2026

Tracing, Diagnostics, and KVP Integration

Architecture

Runtime Behavior

Startup and Configuration

Cleanup and CI

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Peyton Robertson (peytonr18) commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Cade Jacobson (cadejacobson) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Codecov Comments Bot (codecov-commenter) commented Sep 24, 2026 •

edited

Loading

Peyton Robertson (peytonr18) commented Sep 24, 2026 •

edited

Loading