Skip to content

feat: report CLI usage analytics to Argvio instead of OpenTelemetry - #329

Open
emmanuelgautier wants to merge 4 commits into
mainfrom
feat/argvio
Open

emmanuelgautier wants to merge 4 commits into
mainfrom
feat/argvio

Conversation

@emmanuelgautier

@emmanuelgautier emmanuelgautier commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Reports anonymous CLI usage analytics (command, flag names, exit code, latency) to vulnapi's own Argvio tenant through telemetryx.Instrument, and embeds its public_ingest ingest key. It only allows sending usage data, so it is safe to ship in the binary.
  • Removes the OpenTelemetry pipeline (telemetryx.New, every scan, discover and OpenAPI counter). Argvio's command invocations and exit codes replace the per-command success and error counts.
  • Failures carry the same fixed reasons the old error_reason did (for example invalid-client, error-executing-scan), through telemetryx.Reason.
  • Counters for the OpenAPI operations scanned (by HTTP method) and the security schemes seen (by kind, and the unsupported ones). Labels come from fixed lists in code via telemetryx.Vocab, never from the scanned spec.
  • Records a mistyped command as an unknown-command attempt (nearest command as suggestion, never the typed text) through telemetryx.Execute. Shell, terminal and runtime on session start are sent by argvio itself at the Basic tier.
  • Times building the operations from the OpenAPI spec (openapi.operations).
  • Uses telemetryx.Exit / telemetryx.Fatal instead of os.Exit / log.Fatal inside commands, so failed runs are recorded and flushed.
  • Opt out with --sqa-opt-out, DO_NOT_TRACK or ARGVIO_DISABLED. Interactive runs report at the Basic tier, CI runs at Anonymous. Counters are accepted at the anonymous tier, so they are also sent from CI; they only ever carry fixed identifiers written in code.
  • Updates the README and docs Telemetry sections.

Dependency

Uses github.com/cerberauth/x at fc43052 on main (it includes #61 and #66, both merged) and argvio-go v0.4.0, both released. Nothing else needs to merge first.

Test plan

  • go build ./..., go vet ./..., go test ./..., golangci-lint run --new-from-rev=origin/main (0 issues)
  • Ran the built binary against the live endpoint, including failing commands; exit codes and output are unchanged.
  • Error reasons and counters are accepted by Argvio v0.5.0, but the dashboard panels for them (getargvio/ui#26, CSP not configured #27) don't exist yet, so they can't be checked there for now.

Update: Argvio v0.7.0 product metrics

Merge order: cerberauth/x#69 (argvio-go v0.5.0 + dimensioned metrics) first. This branch points at that PR's head (x@2cfb133eb4f2) and will be re-pointed at the merged x. Then cerberauth/ci#42 (catalog workflow) and cerberauth/infrastructure#5 (Argvio v0.7.0 server + UI; it creates this CLI's metrics_admin key).

  • argvio-go v0.5.0 through x: command panics are now recorded (category internal, reason panic, exit 2, never the panic value), so --sqa-opt-out's "crash reports" is true.
  • Metrics use named dimensions (telemetryx.NewCounter/NewDuration/NewValue) instead of one packed label.
  • argvio.metrics.yaml: the catalog the Argvio Product view renders from (titles, units, dimension order, ratios). Validated offline with argvio metric validate v0.7.0. .github/workflows/argvio-metrics.yml validates it on PRs and syncs it on release / manual dispatch with the ARGVIO_METRICS_ADMIN_KEY secret (sync is skipped with a warning while the secret isn't set).
  • Kept, now dimensioned: openapi.operations (method), openapi.security-scheme (scheme × supported; merges the old .unsupported counter). The OpenAPI build timing is renamed openapi.load (a catalog name is unique across kinds).
  • New: scan.operations (value: operations per scan); shared scan outcome via x/telemetryx/scanmetrics: scan.checks (status), scan.vulnerable (check × severity), scan.findings (value per scan by severity), plus check vulnerable/error-rate ratios. vulnapi's own *finding.Finding results are counted through the reporter's Findings hook.
  • Telemetry docs updated (README, EN, FR).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 34.28571% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.20%. Comparing base (f9991d1) to head (5526be5).

Files with missing lines Patch % Lines
cmd/scan/openapi.go 0.00% 9 Missing ⚠️
cmd/discover/api.go 0.00% 7 Missing ⚠️
cmd/scan/curl.go 0.00% 7 Missing ⚠️
cmd/scan/graphql.go 0.00% 6 Missing ⚠️
cmd/discover/domain.go 0.00% 5 Missing ⚠️
cmd/root.go 0.00% 4 Missing ⚠️
internal/finding/attempt.go 0.00% 4 Missing ⚠️
internal/cmd/report.go 0.00% 2 Missing ⚠️
openapi/operation.go 60.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #329      +/-   ##
==========================================
+ Coverage   65.45%   67.20%   +1.74%     
==========================================
  Files          91       91              
  Lines        2565     2464     -101     
==========================================
- Hits         1679     1656      -23     
+ Misses        886      808      -78     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@emmanuelgautier emmanuelgautier changed the title feat: report CLI usage analytics to Argvio feat: report CLI usage analytics to Argvio instead of OpenTelemetry Oct 18, 2026
@emmanuelgautier
emmanuelgautier force-pushed the feat/argvio branch 4 times, most recently from bf7811c to d83d11e Compare November 1, 2026 10:19
emmanuelgautier and others added 3 commits November 3, 2026 00:22
…s.yaml catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….5.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant