Skip to content

feat: report CLI usage analytics to Argvio instead of OpenTelemetry - #147

Open
emmanuelgautier wants to merge 2 commits into
mainfrom
feat/argvio
Open

emmanuelgautier wants to merge 2 commits into
mainfrom
feat/argvio

Conversation

@emmanuelgautier

@emmanuelgautier emmanuelgautier commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Reports anonymous CLI usage analytics (command, flag names, exit code, latency) to openapi-oathkeeper's own Argvio tenant through telemetryx.Instrument, and embeds its public_ingest ingest key. It only allows sending usage data, so it is safe to ship in the binary.
  • Removes the OpenTelemetry pipeline (telemetryx.New and the generator and authenticator counters). Argvio's command invocations and exit codes replace the per-command success and error counts.
  • Failures carry the same fixed reasons the old error_reason did (for example failed-to-load-openapi-file), through telemetryx.Reason.
  • Counters for rules generated or failed (by HTTP verb), authenticators created (by scheme kind) and the output format. Labels come from fixed lists in code via telemetryx.Vocab.
  • Also fixes the encode-failure branch of generate, which reported a nil err instead of encodeErr (the same bug is on main).
  • Records a mistyped command as an unknown-command attempt (nearest command as suggestion, never the typed text) through telemetryx.Execute. Shell, terminal and runtime on session start are sent by argvio itself at the Basic tier.
  • Times rule generation (generator.generate).
  • Uses telemetryx.Exit / telemetryx.Fatal instead of os.Exit / log.Fatal inside commands, so failed runs are recorded and flushed.
  • Opt out with --sqa-opt-out, DO_NOT_TRACK or ARGVIO_DISABLED. Interactive runs report at the Basic tier, CI runs at Anonymous. Counters are accepted at the anonymous tier, so they are also sent from CI; they only ever carry fixed identifiers written in code.
  • Fixes the README Telemetry section, which was copied from vulnapi and described scan and vulnerability counts.

Dependency

Uses github.com/cerberauth/x at fc43052 on main (it includes #61 and #66, both merged) and argvio-go v0.4.0, both released. Nothing else needs to merge first.

Test plan

  • go build ./..., go vet ./..., go test ./..., golangci-lint run --new-from-rev=origin/main (0 issues)
  • Ran the built binary against the live endpoint, including failing commands; exit codes and output are unchanged.
  • Error reasons and counters are accepted by Argvio v0.5.0, but the dashboard panels for them (getargvio/ui#26, fix: match more params chars #27) don't exist yet, so they can't be checked there for now.

Update: Argvio v0.7.0 product metrics

Merge order: cerberauth/x#69 (argvio-go v0.5.0 + dimensioned metrics) first. This branch points at that PR's head (x@2cfb133eb4f2) and will be re-pointed at the merged x. Then cerberauth/ci#42 (catalog workflow) and cerberauth/infrastructure#5 (Argvio v0.7.0 server + UI; it creates this CLI's metrics_admin key).

  • argvio-go v0.5.0 through x: command panics are now recorded (category internal, reason panic, exit 2, never the panic value), so --sqa-opt-out's "crash reports" is true.
  • Metrics use named dimensions (telemetryx.NewCounter/NewDuration/NewValue) instead of one packed label.
  • argvio.metrics.yaml: the catalog the Argvio Product view renders from (titles, units, dimension order, ratios). Validated offline with argvio metric validate v0.7.0. .github/workflows/argvio-metrics.yml validates it on PRs and syncs it on release / manual dispatch with the ARGVIO_METRICS_ADMIN_KEY secret (sync is skipped with a warning while the secret isn't set).
  • Kept, now dimensioned: generator.rules (method × result; was verb.generated in one label), authenticator.created (scheme), generate.output (format × destination file/stdout, new), generator.generate duration.
  • New: generate.rules (value: rules per run), generator.rule-failure-rate ratio.
  • README telemetry text updated.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.22%. Comparing base (1d38329) to head (310adba).

Files with missing lines Patch % Lines
cmd/generate/root.go 0.00% 16 Missing ⚠️
cmd/root.go 0.00% 4 Missing ⚠️
authenticator/authenticator.go 87.50% 1 Missing ⚠️
generator/generator.go 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #147      +/-   ##
==========================================
+ Coverage   55.58%   57.22%   +1.64%     
==========================================
  Files          11       11              
  Lines         358      332      -26     
==========================================
- Hits          199      190       -9     
+ Misses        159      142      -17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@emmanuelgautier emmanuelgautier changed the title feat: report CLI usage analytics to Argvio feat: report CLI usage analytics to Argvio instead of OpenTelemetry Oct 18, 2026
@emmanuelgautier
emmanuelgautier force-pushed the feat/argvio branch 4 times, most recently from 2151cfc to 021db76 Compare November 1, 2026 10:19
…cs.yaml catalog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant