Repository navigation
Document telemetry resource attributes and runtime metrics - #592
Merged
Merged
Conversation
kagent-dev/kagent#3107 moved the runtime identity to resource attributes (gen_ai.main_agent.*, kagent.runtime), added the gen_ai.invoke_agent.duration metric behind otel.metrics.enabled, and added a reference Collector values file. Update the tracing, metrics, and OTel stack pages to match. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Kristin Brown <kristin.brown@solo.io>
Docs preview
Both are uploaded Worker versions and serve no production traffic. |
|
|
||
| The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack. | ||
|
|
||
| ## Agent runtime metrics |
Contributor
There was a problem hiding this comment.
Checked whether this section should also carry alpha12's kagent.execute_tool.duration and kagent.client.inference.*. It should not: those exist only in telemetry/registry/ and the contract doc, with no Go emitting them yet. Scope here is right.
|
|
||
| ## Agent runtime metrics | ||
|
|
||
| Agent runtimes export metrics over OTLP when the kagent Helm release sets `otel.metrics.enabled` to `true`. The stack guides convert these OpenTelemetry metrics to Prometheus names in the collector. |
Contributor
There was a problem hiding this comment.
claude and codex are the only runtimes that report the histogram in this section, so it is worth saying before the table.
Suggested change
| Agent runtimes export metrics over OTLP when the kagent Helm release sets `otel.metrics.enabled` to `true`. The stack guides convert these OpenTelemetry metrics to Prometheus names in the collector. | |
| Agent runtimes export metrics over OTLP when the kagent Helm release sets `otel.metrics.enabled` to `true`. The `claude` and `codex` runtimes are the only ones that report the histogram in this section. The stack guides convert these OpenTelemetry metrics to Prometheus names in the collector. |
|
|
||
| | Metric | Type | Measures | | ||
| | ------ | ---- | -------- | | ||
| | `gen_ai_invoke_agent_duration_seconds` | Histogram | Time that one `invoke_agent` execution segment takes. kagent records this metric with every `invoke_agent` span that it opens. A collector can derive the same metric from runtime-owned `invoke_agent` spans before sampling. | |
Contributor
There was a problem hiding this comment.
InvocationDuration is constructed in one place, go/adk/pkg/a2a/server/tracing.go:59, inside if telemetry.Runtime.NativeHarness() — true only for claude and codex, covered by TestInvocationDurationOnlyForNativeHarnesses. On the default kagent runtime this metric and the p95 query never produce data. Your closing sentence already points at the workaround; this makes the opening agree with it.
Suggested change
| | `gen_ai_invoke_agent_duration_seconds` | Histogram | Time that one `invoke_agent` execution segment takes. kagent records this metric with every `invoke_agent` span that it opens. A collector can derive the same metric from runtime-owned `invoke_agent` spans before sampling. | | |
| | `gen_ai_invoke_agent_duration_seconds` | Histogram | Time that one `invoke_agent` execution segment takes. Only the `claude` and `codex` runtimes record it. The `kagent` runtime emits its own `invoke_agent` spans, so its wrapper stays a transport span and records no duration. For that runtime, derive the metric from those spans in the collector before sampling. | |
Rachael-Graham
approved these changes
Oct 13, 2026
Signed-off-by: Kristin Brown <kristin.brown@solo.io>
Contributor
Author
|
Addressed feedback: 302953e |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the user-facing telemetry changes from kagent-dev/kagent#3107 (first shipped in v1.0.0-alpha10).
What changed in kagent
gen_ai.main_agent.id,gen_ai.main_agent.name, andkagent.runtime. The resource-levelgen_ai.agent.*names are gone.gen_ai.invoke_agent.durationhistogram over OTLP whenotel.metrics.enabledistrue.examples/observability/collector-values.yaml.What this PR changes
gen_ai.main_agent.id. The search step shows both kinds of filter.gen_ai_invoke_agent_duration_seconds, its labels, and an example p95 query.otel.metricsin the kagent Helm values, add it to the field table and the cleanup step, and point production readers to the reference Collector values.Not verified
Nothing here was run on a cluster. To check it:
otel.metrics.enabled: true.histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))). It should return data.gen_ai.conversation.id, and the resource carriesgen_ai.main_agent.idandkagent.runtime.kagent-dev/kagent#3103 (same author, merged the day before, same source files) may be part of the same change and was not reviewed for this PR.
🤖 Generated with Claude Code