Document telemetry resource attributes and runtime metrics

docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md

@@ -124,7 +124,7 @@ Install the [Prometheus](https://github.com/prometheus-community/helm-charts/tre
 
 ## Install the OTel collector
 
-Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for HTTP. The collector sends traces to Jaeger, serves metrics on port `8889`, and prints logs to its own output. The configuration uses the `contrib` collector image, because the Prometheus exporter is not part of the core image.
+Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for HTTP. The collector sends traces to Jaeger, serves metrics on port `8889`, and prints logs to its own output. The configuration uses the `contrib` collector image, because the Prometheus exporter is not part of the core image. For production, start from the reference Collector values in the kagent source at `examples/observability/collector-values.yaml`. The smaller collector in this guide omits the persistent queue and content-scrubbing processors.
 
 ```bash
 helm upgrade --install otel-collector opentelemetry-collector \
@@ -182,7 +182,7 @@ EOF
 
 ## Send kagent telemetry to the collector
 
-Turn on the kagent trace and log exporters, and point both at the collector. Also turn on the controller's metrics endpoint, and allow Prometheus to scrape it.
+Turn on the kagent trace, log, and metrics exporters, and point them at the collector. Also turn on the controller's metrics endpoint, and allow Prometheus to scrape it.
 
 1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision.
    ```bash
@@ -213,6 +213,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
        enabled: true
      logs:
        enabled: true
+     metrics:
+       enabled: true
    controller:
      metrics:
        enabled: true
@@ -319,6 +321,7 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in
      --namespace kagent --reuse-values \
      --set otel.traces.enabled=false \
      --set otel.logs.enabled=false \
+     --set otel.metrics.enabled=false \
      --set controller.metrics.enabled=false
    helm upgrade substrate \
      {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \

docs-site/content/kagent/1.x/observability/metrics.md

@@ -5,7 +5,7 @@ weight: 40
 author: kagent.dev
 ---
 
-{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent Substrate pushes its metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them.
+{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent runtimes and Agent Substrate push metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both collection paths are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn on both collection paths and set up Prometheus to collect them.
 
 ## kagent controller metrics
 
@@ -45,6 +45,16 @@ Besides the standard Go runtime and process metrics, the controller reports the
 
 The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack.
 
+## Agent runtime metrics
+
+Agent runtimes export metrics over OTLP when the kagent Helm release sets `otel.metrics.enabled` to `true`. The `claude` and `codex` runtimes are the only ones that report the histogram in this section. The stack guides convert these OpenTelemetry metrics to Prometheus names in the collector.
+
+| Metric | Type | Measures |
+| ------ | ---- | -------- |
+| `gen_ai_invoke_agent_duration_seconds` | Histogram | Time that one `invoke_agent` execution segment takes. Only the `claude` and `codex` runtimes record it. The `kagent` runtime emits its own `invoke_agent` spans, so its wrapper stays a transport span and records no duration. For that runtime, derive the metric from those spans in the collector before sampling. |
+
+The metric carries labels for the Agent name, model, and error type when those values exist. When the collector copies resource attributes to labels, the metric also carries labels such as `service_name`, `service_namespace`, `service_version`, `kagent_runtime`, and `gen_ai_main_agent_id`.
+
 ## Agent Substrate metrics
 
 ### Export the metrics
@@ -109,6 +119,11 @@ Run the following queries in Prometheus or Grafana.
   sum by (controller) (rate(controller_runtime_reconcile_errors_total{job="kagent-controller-metrics"}[5m]))
   ```
 
+- The 95th percentile agent invocation duration, per Agent. The query returns data only for Agents on the `claude` and `codex` runtimes.
+  ```text
+  histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m])))
+  ```
+
 ## Next steps
 
 {{< cards >}}

docs-site/content/kagent/1.x/observability/otel-stack.md

@@ -33,7 +33,7 @@ flowchart LR
         prom["Prometheus"]
     end
     grafana["Grafana"]
-    kagent -->|traces, logs| collector
+    kagent -->|traces, logs, runtime metrics| collector
     substrate -->|traces, logs, metrics| collector
     kagent -->|metrics scrape| prom
     collector --> tempo
@@ -52,7 +52,7 @@ The following table lists what each source sends. For the meaning of each signal
 
 | Source | Traces | Logs | Metrics |
 | ------ | ------ | ---- | ------- |
-| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. |
+| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Agent runtimes export runtime metrics over OTLP. Prometheus scrapes reconciliation, work queue, and gRPC API metrics from the controller's `/metrics` endpoint. |
 | Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | A record of every Actor state change, and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. |
 
 Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window.
@@ -317,7 +317,7 @@ Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for H
 
 ## Send kagent telemetry to the collector
 
-Turn on the kagent trace and log exporters, and point both at the collector. Also turn on the controller's metrics endpoint, together with the ServiceMonitor that the kagent chart creates for it.
+Turn on the kagent trace, log, and metrics exporters, and point them at the collector. Also turn on the controller's metrics endpoint, together with the ServiceMonitor that the kagent chart creates for it.
 
 1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision.
    ```bash
@@ -348,6 +348,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
        enabled: true
      logs:
        enabled: true
+     metrics:
+       enabled: true
    controller:
      metrics:
        enabled: true
@@ -366,6 +368,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
    | `otel.exporter.otlp` | The OTLP endpoint and protocol that every signal exports to. An `http://` endpoint sends plaintext. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). |
    | `otel.traces` | Exports traces from the controller and from the agent runtimes that it starts. |
    | `otel.logs` | Exports log records from the agent runtimes. The `kagent` runtime never puts prompts or replies in a log record. For where that content goes, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). |
+   | `otel.metrics` | Exports runtime metrics, such as `gen_ai.invoke_agent.duration`, over OTLP to the collector. |
    | `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. |
    | `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. |
 
@@ -497,6 +500,7 @@ Log in to Grafana, and query each backend from the **Explore** view.
      --namespace kagent --reuse-values \
      --set otel.traces.enabled=false \
      --set otel.logs.enabled=false \
+     --set otel.metrics.enabled=false \
      --set controller.metrics.enabled=false \
      --set controller.metrics.serviceMonitor.enabled=false
    helm upgrade substrate \

docs-site/content/kagent/1.x/observability/tracing.md

@@ -70,15 +70,16 @@ The `kagent` runtime creates the same spans for every agent, and most span names
 
 ### Correlation attributes
 
-A trace tells you which request you are looking at through attributes on its spans, not through the span names. The runtime stamps the following four attributes onto its root span and copies them onto every descendant span. A search on any one of these attributes returns the whole subtree rather than a single span.
+Use attributes, not span names, to identify a request in a trace. The runtime copies the request attributes to every descendant span. The runtime identity and main Agent identity are resource attributes, so use them to filter all spans from the same runtime process.
 
-| Attribute | Value |
-| --------- | ----- |
-| `a2a.task.id` | The A2A task ID, which identifies one turn of a conversation. |
-| `gen_ai.conversation.id` | The A2A context ID, which identifies the conversation and is stable across its turns. |
-| `gen_ai.agent.id` | The Agent, as `<namespace>/<name>`. The shorter `gen_ai.agent.name` carries the name alone. |
-| `enduser.id` | The authenticated caller, such as `admin@kagent.dev`. |
-| `kagent.runtime` | The runtime that served the request, such as `adk-go`. |
+| Attribute | Where | Value |
+| --------- | ----- | ----- |
+| `a2a.task.id` | Span | The A2A task ID, which identifies one turn of a conversation. |
+| `gen_ai.conversation.id` | Span | The A2A context ID, which identifies the conversation and is stable across its turns. |
+| `gen_ai.agent.id` | Span | The Agent, as `<namespace>/<name>`. The shorter `gen_ai.agent.name` carries the name alone. |
+| `enduser.id` | Span | The authenticated caller, such as `admin@kagent.dev`. |
+| `gen_ai.main_agent.id` | Resource | The main Agent that the runtime process was compiled for, as `<namespace>/<name>`. This resource attribute replaces the resource-level `gen_ai.agent.id` name. |
+| `kagent.runtime` | Resource | The runtime that served the request, such as `adk-go`. |
 
 The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.<key>` attribute, so a client can tag a request and search for it later. Unlike the correlation attributes, these tags stay on the `a2a.request` span alone, so a search on one returns that span instead of the whole subtree.
 
@@ -227,7 +228,7 @@ Send a request to a new Session, then find its trace in the backend that you set
                    HTTP POST                                my-first-agent
    ```
 
-4. To narrow a search to one conversation, search by a correlation attribute, such as `gen_ai.conversation.id=<context-id>`.
+4. Use a span attribute, such as `gen_ai.conversation.id=<context-id>`, to narrow a search to one conversation. Use a resource attribute, such as `gen_ai.main_agent.id=<namespace>/<agent>`, to narrow a search to one runtime process.
 
 ## Agent Substrate traces