diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md index 10aa2b9c..584400d8 100644 --- a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -152,6 +152,31 @@ config: endpoint: 0.0.0.0:4318 processors: batch: {} + connectors: + signaltometrics: + spans: + - name: gen_ai.invoke_agent.duration + description: The duration of one agent invocation, derived from its span. + unit: s + conditions: + - span.attributes["gen_ai.operation.name"] == "invoke_agent" and instrumentation_scope.name != "github.com/kagent-dev/kagent/go/adk/pkg/a2a/server" and instrumentation_scope.name != "kagent.core.a2a" + attributes: + - key: gen_ai.agent.name + optional: true + - key: gen_ai.request.model + optional: true + - key: error.type + optional: true + include_resource_attributes: + - key: service.name + - key: service.namespace + optional: true + - key: service.version + optional: true + - key: kagent.runtime + optional: true + - key: gen_ai.main_agent.id + optional: true exporters: debug: verbosity: basic @@ -168,13 +193,13 @@ config: traces: receivers: [otlp] processors: [batch] - exporters: [otlp_grpc/jaeger] + exporters: [signaltometrics, otlp_grpc/jaeger] logs: receivers: [otlp] processors: [batch] exporters: [debug] metrics: - receivers: [otlp] + receivers: [otlp, signaltometrics] processors: [batch] exporters: [prometheus] EOF @@ -213,6 +238,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als enabled: true logs: enabled: true + metrics: + enabled: true controller: metrics: enabled: true @@ -303,6 +330,10 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in ``` 2. In your browser, open Prometheus at [http://localhost:9090/targets](http://localhost:9090/targets), and confirm that the `otel-collector` and `kagent-controller` targets are `UP`. 3. Open the **Query** page, and run a query such as `ate_workerpool_ready_workers`. For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). + To check agent invocation latency, run the following query. + ```text + histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))) + ``` 3. Check that the collector receives logs. Each line reports a batch of log records from one source. ```bash @@ -319,6 +350,7 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in --namespace kagent --reuse-values \ --set otel.traces.enabled=false \ --set otel.logs.enabled=false \ + --set otel.metrics.enabled=false \ --set controller.metrics.enabled=false helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/1.x/observability/metrics.md index 083cd61b..419fb179 100644 --- a/docs-site/content/kagent/1.x/observability/metrics.md +++ b/docs-site/content/kagent/1.x/observability/metrics.md @@ -5,7 +5,7 @@ weight: 40 author: kagent.dev --- -{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent Substrate pushes its metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them. +{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent runtimes and Agent Substrate push their metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them. ## kagent controller metrics @@ -45,6 +45,30 @@ Besides the standard Go runtime and process metrics, the controller reports the The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack. +## kagent runtime metrics {#runtime-metrics} + +### Export the metrics + +kagent runtimes export metrics over OTLP when metric export is enabled and an OTLP endpoint is configured. The setting affects new runtime revisions, so create a new Session after the Helm upgrade. + +```yaml +otel: + exporter: + otlp: + endpoint: http://otel-collector.telemetry.svc.cluster.local:4317 + protocol: grpc + metrics: + enabled: true +``` + +### Metric reference + +The collector's Prometheus exporter converts OpenTelemetry metric names to Prometheus names. For example, `gen_ai.invoke_agent.duration` becomes the `gen_ai_invoke_agent_duration_seconds` histogram. + +| Metric | Type | Measures | +| ------ | ---- | -------- | +| `gen_ai.invoke_agent.duration` | Histogram | Time for one execution segment of a kagent Agent. The metric carries `gen_ai.agent.name`, `gen_ai.request.model` when the runtime is compiled against one model, and `error.type` on failed invocations. The stack guides also derive the metric from `invoke_agent` spans that the runtime emits before sampling. | + ## Agent Substrate metrics ### Export the metrics @@ -109,6 +133,11 @@ Run the following queries in Prometheus or Grafana. sum by (controller) (rate(controller_runtime_reconcile_errors_total{job="kagent-controller-metrics"}[5m])) ``` +- The 95th percentile time for agent invocations, by Agent. + ```text + histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))) + ``` + ## Next steps {{< cards >}} diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md index a979f06c..0ddc17f2 100644 --- a/docs-site/content/kagent/1.x/observability/otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -52,7 +52,7 @@ The following table lists what each source sends. For the meaning of each signal | Source | Traces | Logs | Metrics | | ------ | ------ | ---- | ------- | -| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. | +| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. Agent invocation duration metrics, exported by the runtimes over OTLP and derived from `invoke_agent` spans before sampling. | | Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | A record of every Actor state change, and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. | Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window. @@ -257,6 +257,31 @@ Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for H endpoint: 0.0.0.0:4318 processors: batch: {} + connectors: + signaltometrics: + spans: + - name: gen_ai.invoke_agent.duration + description: The duration of one agent invocation, derived from its span. + unit: s + conditions: + - span.attributes["gen_ai.operation.name"] == "invoke_agent" and instrumentation_scope.name != "github.com/kagent-dev/kagent/go/adk/pkg/a2a/server" and instrumentation_scope.name != "kagent.core.a2a" + attributes: + - key: gen_ai.agent.name + optional: true + - key: gen_ai.request.model + optional: true + - key: error.type + optional: true + include_resource_attributes: + - key: service.name + - key: service.namespace + optional: true + - key: service.version + optional: true + - key: kagent.runtime + optional: true + - key: gen_ai.main_agent.id + optional: true exporters: otlp_grpc/tempo: endpoint: tempo.telemetry.svc.cluster.local:4317 @@ -273,13 +298,13 @@ Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for H traces: receivers: [otlp] processors: [batch] - exporters: [otlp_grpc/tempo] + exporters: [signaltometrics, otlp_grpc/tempo] logs: receivers: [otlp] processors: [batch] exporters: [otlp_http/loki] metrics: - receivers: [otlp] + receivers: [otlp, signaltometrics] processors: [batch] exporters: [prometheus] EOF @@ -348,6 +373,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als enabled: true logs: enabled: true + metrics: + enabled: true controller: metrics: enabled: true @@ -366,6 +393,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als | `otel.exporter.otlp` | The OTLP endpoint and protocol that every signal exports to. An `http://` endpoint sends plaintext. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). | | `otel.traces` | Exports traces from the controller and from the agent runtimes that it starts. | | `otel.logs` | Exports log records from the agent runtimes. The `kagent` runtime never puts prompts or replies in a log record. For where that content goes, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | + | `otel.metrics` | Exports OTLP metrics from the agent runtimes. The collector also derives `gen_ai.invoke_agent.duration` from `invoke_agent` spans before sampling. | | `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. | | `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. | @@ -483,6 +511,11 @@ Log in to Grafana, and query each backend from the **Explore** view. sum by (method) (rate(kagent_grpc_server_requests_total[5m])) ``` + The following query returns the 95th percentile invocation duration by Agent. + ```text + histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m]))) + ``` + For more metrics, see [Metrics]({{< link path="observability/metrics" >}}). {{% /tab %}} {{< /tabs >}} @@ -497,6 +530,7 @@ Log in to Grafana, and query each backend from the **Explore** view. --namespace kagent --reuse-values \ --set otel.traces.enabled=false \ --set otel.logs.enabled=false \ + --set otel.metrics.enabled=false \ --set controller.metrics.enabled=false \ --set controller.metrics.serviceMonitor.enabled=false helm upgrade substrate \ diff --git a/docs-site/content/kagent/1.x/observability/tracing.md b/docs-site/content/kagent/1.x/observability/tracing.md index e7bbac42..6a77075f 100644 --- a/docs-site/content/kagent/1.x/observability/tracing.md +++ b/docs-site/content/kagent/1.x/observability/tracing.md @@ -43,7 +43,7 @@ Each hop reports itself as a separate OpenTelemetry (OTel) service. A tracing ba - **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on. - **The Agent Substrate router** reports as two services, because its pod runs two containers. The router's own spans, such as its lookup of the Actor for a request, report as `atenet-router`. The spans of the agentgateway proxy that forwards the request to the Worker report as `agentgateway`. Only the `agentgateway` spans join the agent request trace. The `atenet-router` spans form separate [Agent Substrate traces](#agent-substrate-traces). -- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. This is a change from kagent 0.x, where every agent reported under one `kagent` service. A backend that you filter by service now shows one entry for each Agent, and adding an Agent adds a service. +- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. Its resource carries the main agent identity as `gen_ai.main_agent.name` and `gen_ai.main_agent.id`. Request spans carry the invoked agent identity as `gen_ai.agent.name` and `gen_ai.agent.id`. A backend that you filter by service now shows one entry for each Agent, and adding an Agent adds a service. ### Runtime coverage @@ -70,15 +70,16 @@ The `kagent` runtime creates the same spans for every agent, and most span names ### Correlation attributes -A trace tells you which request you are looking at through attributes on its spans, not through the span names. The runtime stamps the following four attributes onto its root span and copies them onto every descendant span. A search on any one of these attributes returns the whole subtree rather than a single span. +A trace tells you which request you are looking at through attributes, not through the span names. A search on a span attribute returns the request subtree. A search on a resource attribute returns all spans and metrics from that runtime process. -| Attribute | Value | -| --------- | ----- | -| `a2a.task.id` | The A2A task ID, which identifies one turn of a conversation. | -| `gen_ai.conversation.id` | The A2A context ID, which identifies the conversation and is stable across its turns. | -| `gen_ai.agent.id` | The Agent, as `/`. The shorter `gen_ai.agent.name` carries the name alone. | -| `enduser.id` | The authenticated caller, such as `admin@kagent.dev`. | -| `kagent.runtime` | The runtime that served the request, such as `adk-go`. | +| Attribute | Location | Value | +| --------- | -------- | ----- | +| `a2a.task.id` | Span | The A2A task ID, which identifies one turn of a conversation. | +| `gen_ai.conversation.id` | Span | The A2A context ID, which identifies the conversation and is stable across its turns. | +| `gen_ai.agent.id` | Span | The invoked Agent, as `/`. The shorter `gen_ai.agent.name` carries the name alone. | +| `enduser.id` | Span | The authenticated caller, such as `admin@kagent.dev`. | +| `gen_ai.main_agent.id` | Resource | The top-level Agent that the runtime was compiled from. The shorter `gen_ai.main_agent.name` carries the name alone. | +| `kagent.runtime` | Resource | The runtime that served the request, such as `adk-go`. | The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.` attribute, so a client can tag a request and search for it later. Unlike the correlation attributes, these tags stay on the `a2a.request` span alone, so a search on one returns that span instead of the whole subtree.