Skip to content

[Automated] Draft docs (kagent): feat(telemetry): Weaver live-check, registry catch-up, reference… #601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
github-actions wants to merge 1 commit into from
Closed

[Automated] Draft docs (kagent): feat(telemetry): Weaver live-check, registry catch-up, reference… #601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,31 @@ config:
endpoint: 0.0.0.0:4318
processors:
batch: {}
connectors:
signaltometrics:
spans:
- name: gen_ai.invoke_agent.duration
description: The duration of one agent invocation, derived from its span.
unit: s
conditions:
- span.attributes["gen_ai.operation.name"] == "invoke_agent" and instrumentation_scope.name != "github.com/kagent-dev/kagent/go/adk/pkg/a2a/server" and instrumentation_scope.name != "kagent.core.a2a"
attributes:
- key: gen_ai.agent.name
optional: true
- key: gen_ai.request.model
optional: true
- key: error.type
optional: true
include_resource_attributes:
- key: service.name
- key: service.namespace
optional: true
- key: service.version
optional: true
- key: kagent.runtime
optional: true
- key: gen_ai.main_agent.id
optional: true
exporters:
debug:
verbosity: basic
Expand All @@ -168,13 +193,13 @@ config:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp_grpc/jaeger]
exporters: [signaltometrics, otlp_grpc/jaeger]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug]
metrics:
receivers: [otlp]
receivers: [otlp, signaltometrics]
processors: [batch]
exporters: [prometheus]
EOF
Expand Down Expand Up @@ -213,6 +238,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
enabled: true
logs:
enabled: true
metrics:
enabled: true
controller:
metrics:
enabled: true
Expand Down Expand Up @@ -303,6 +330,10 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in
```
2. In your browser, open Prometheus at [http://localhost:9090/targets](http://localhost:9090/targets), and confirm that the `otel-collector` and `kagent-controller` targets are `UP`.
3. Open the **Query** page, and run a query such as `ate_workerpool_ready_workers`. For more metrics, see [Metrics]({{< link path="observability/metrics" >}}).
To check agent invocation latency, run the following query.
```text
histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m])))
```

3. Check that the collector receives logs. Each line reports a batch of log records from one source.
```bash
Expand All @@ -319,6 +350,7 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in
--namespace kagent --reuse-values \
--set otel.traces.enabled=false \
--set otel.logs.enabled=false \
--set otel.metrics.enabled=false \
--set controller.metrics.enabled=false
helm upgrade substrate \
{{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
Expand Down
31 changes: 30 additions & 1 deletion docs-site/content/kagent/1.x/observability/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ weight: 40
author: kagent.dev
---

{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent Substrate pushes its metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them.
{{< reuse "kagent-docs/snippets/name-product.md" >}} and Agent Substrate report metrics in two different ways. The kagent controller serves a Prometheus `/metrics` endpoint that Prometheus scrapes. Agent runtimes and Agent Substrate push their metrics over the OpenTelemetry Protocol (OTLP) to a collector, which serves them for Prometheus to scrape. Both are off by default. The [OTel stack]({{< link path="observability/otel-stack" >}}) and [Lightweight OTel stack]({{< link path="observability/lightweight-otel-stack" >}}) guides turn both on and set up Prometheus to collect them.

## kagent controller metrics

Expand Down Expand Up @@ -45,6 +45,30 @@ Besides the standard Go runtime and process metrics, the controller reports the

The Agent Substrate `atecontroller` component reports the same `controller_runtime_*` and `workqueue_*` metrics for its own reconcilers. To keep the two apart in a query, filter by the scrape job, such as `job="kagent-controller-metrics"` in the OTel stack.

## kagent runtime metrics {#runtime-metrics}

### Export the metrics

kagent runtimes export metrics over OTLP when metric export is enabled and an OTLP endpoint is configured. The setting affects new runtime revisions, so create a new Session after the Helm upgrade.

```yaml
otel:
exporter:
otlp:
endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
protocol: grpc
metrics:
enabled: true
```

### Metric reference

The collector's Prometheus exporter converts OpenTelemetry metric names to Prometheus names. For example, `gen_ai.invoke_agent.duration` becomes the `gen_ai_invoke_agent_duration_seconds` histogram.

| Metric | Type | Measures |
| ------ | ---- | -------- |
| `gen_ai.invoke_agent.duration` | Histogram | Time for one execution segment of a kagent Agent. The metric carries `gen_ai.agent.name`, `gen_ai.request.model` when the runtime is compiled against one model, and `error.type` on failed invocations. The stack guides also derive the metric from `invoke_agent` spans that the runtime emits before sampling. |

## Agent Substrate metrics

### Export the metrics
Expand Down Expand Up @@ -109,6 +133,11 @@ Run the following queries in Prometheus or Grafana.
sum by (controller) (rate(controller_runtime_reconcile_errors_total{job="kagent-controller-metrics"}[5m]))
```

- The 95th percentile time for agent invocations, by Agent.
```text
histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m])))
```

## Next steps

{{< cards >}}
Expand Down
40 changes: 37 additions & 3 deletions docs-site/content/kagent/1.x/observability/otel-stack.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ The following table lists what each source sends. For the meaning of each signal

| Source | Traces | Logs | Metrics |
| ------ | ------ | ---- | ------- |
| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. |
| kagent controller and agent runtimes | One trace per agent request, across the controller, the Agent Substrate proxy, and the agent runtime. | Log records from the agent runtimes. Prompts and replies are not among them on the `kagent` runtime, which puts them on spans instead. For how to capture them, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). | Reconciliation, work queue, and gRPC API metrics, scraped from the controller's `/metrics` endpoint. Agent invocation duration metrics, exported by the runtimes over OTLP and derived from `invoke_agent` spans before sampling. |
| Agent Substrate | Separate traces for its own work, such as routing a request to a Worker and resuming an Actor. | A record of every Actor state change, and the router access log. | Actor lifecycle, scheduling, snapshot, image cache, and WorkerPool capacity metrics. |

Agent Substrate traces do not join the trace of the agent request that caused them. To see why a request was slow to start, look up the Agent Substrate trace from the same time window.
Expand Down Expand Up @@ -257,6 +257,31 @@ Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for H
endpoint: 0.0.0.0:4318
processors:
batch: {}
connectors:
signaltometrics:
spans:
- name: gen_ai.invoke_agent.duration
description: The duration of one agent invocation, derived from its span.
unit: s
conditions:
- span.attributes["gen_ai.operation.name"] == "invoke_agent" and instrumentation_scope.name != "github.com/kagent-dev/kagent/go/adk/pkg/a2a/server" and instrumentation_scope.name != "kagent.core.a2a"
attributes:
- key: gen_ai.agent.name
optional: true
- key: gen_ai.request.model
optional: true
- key: error.type
optional: true
include_resource_attributes:
- key: service.name
- key: service.namespace
optional: true
- key: service.version
optional: true
- key: kagent.runtime
optional: true
- key: gen_ai.main_agent.id
optional: true
exporters:
otlp_grpc/tempo:
endpoint: tempo.telemetry.svc.cluster.local:4317
Expand All @@ -273,13 +298,13 @@ Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for H
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp_grpc/tempo]
exporters: [signaltometrics, otlp_grpc/tempo]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp_http/loki]
metrics:
receivers: [otlp]
receivers: [otlp, signaltometrics]
processors: [batch]
exporters: [prometheus]
EOF
Expand Down Expand Up @@ -348,6 +373,8 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
enabled: true
logs:
enabled: true
metrics:
enabled: true
controller:
metrics:
enabled: true
Expand All @@ -366,6 +393,7 @@ Turn on the kagent trace and log exporters, and point both at the collector. Als
| `otel.exporter.otlp` | The OTLP endpoint and protocol that every signal exports to. An `http://` endpoint sends plaintext. For each field, see [Tracing]({{< link path="observability/tracing#enable-tracing" >}}). |
| `otel.traces` | Exports traces from the controller and from the agent runtimes that it starts. |
| `otel.logs` | Exports log records from the agent runtimes. The `kagent` runtime never puts prompts or replies in a log record. For where that content goes, see [Audit prompts]({{< link path="observability/audit-prompts" >}}). |
| `otel.metrics` | Exports OTLP metrics from the agent runtimes. The collector also derives `gen_ai.invoke_agent.duration` from `invoke_agent` spans before sampling. |
| `controller.metrics.enabled` | Serves the controller's Prometheus metrics on port `8443` over HTTPS. |
| `controller.metrics.serviceMonitor` | Creates a ServiceMonitor for the metrics endpoint. The `prometheusServiceAccount` setting binds the metrics reader role to the Prometheus ServiceAccount, which authorizes the scrape. |

Expand Down Expand Up @@ -483,6 +511,11 @@ Log in to Grafana, and query each backend from the **Explore** view.
sum by (method) (rate(kagent_grpc_server_requests_total[5m]))
```

The following query returns the 95th percentile invocation duration by Agent.
```text
histogram_quantile(0.95, sum by (le, gen_ai_agent_name) (rate(gen_ai_invoke_agent_duration_seconds_bucket[5m])))
```

For more metrics, see [Metrics]({{< link path="observability/metrics" >}}).
{{% /tab %}}
{{< /tabs >}}
Expand All @@ -497,6 +530,7 @@ Log in to Grafana, and query each backend from the **Explore** view.
--namespace kagent --reuse-values \
--set otel.traces.enabled=false \
--set otel.logs.enabled=false \
--set otel.metrics.enabled=false \
--set controller.metrics.enabled=false \
--set controller.metrics.serviceMonitor.enabled=false
helm upgrade substrate \
Expand Down
19 changes: 10 additions & 9 deletions docs-site/content/kagent/1.x/observability/tracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Each hop reports itself as a separate OpenTelemetry (OTel) service. A tracing ba

- **The controller** reports as `kagent-controller` in the `kagent` service namespace. Its spans also carry the pod, node, and namespace that the controller runs on.
- **The Agent Substrate router** reports as two services, because its pod runs two containers. The router's own spans, such as its lookup of the Actor for a request, report as `atenet-router`. The spans of the agentgateway proxy that forwards the request to the Worker report as `agentgateway`. Only the `agentgateway` spans join the agent request trace. The `atenet-router` spans form separate [Agent Substrate traces](#agent-substrate-traces).
- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. This is a change from kagent 0.x, where every agent reported under one `kagent` service. A backend that you filter by service now shows one entry for each Agent, and adding an Agent adds a service.
- **Each agent runtime** reports as its own service, named for the {{< gloss "Agent" >}}Agent{{< /gloss >}} it was compiled from. The `my-first-agent` Agent reports as `my-first-agent`. Its resource carries the main agent identity as `gen_ai.main_agent.name` and `gen_ai.main_agent.id`. Request spans carry the invoked agent identity as `gen_ai.agent.name` and `gen_ai.agent.id`. A backend that you filter by service now shows one entry for each Agent, and adding an Agent adds a service.

### Runtime coverage

Expand All @@ -70,15 +70,16 @@ The `kagent` runtime creates the same spans for every agent, and most span names

### Correlation attributes

A trace tells you which request you are looking at through attributes on its spans, not through the span names. The runtime stamps the following four attributes onto its root span and copies them onto every descendant span. A search on any one of these attributes returns the whole subtree rather than a single span.
A trace tells you which request you are looking at through attributes, not through the span names. A search on a span attribute returns the request subtree. A search on a resource attribute returns all spans and metrics from that runtime process.

| Attribute | Value |
| --------- | ----- |
| `a2a.task.id` | The A2A task ID, which identifies one turn of a conversation. |
| `gen_ai.conversation.id` | The A2A context ID, which identifies the conversation and is stable across its turns. |
| `gen_ai.agent.id` | The Agent, as `<namespace>/<name>`. The shorter `gen_ai.agent.name` carries the name alone. |
| `enduser.id` | The authenticated caller, such as `admin@kagent.dev`. |
| `kagent.runtime` | The runtime that served the request, such as `adk-go`. |
| Attribute | Location | Value |
| --------- | -------- | ----- |
| `a2a.task.id` | Span | The A2A task ID, which identifies one turn of a conversation. |
| `gen_ai.conversation.id` | Span | The A2A context ID, which identifies the conversation and is stable across its turns. |
| `gen_ai.agent.id` | Span | The invoked Agent, as `<namespace>/<name>`. The shorter `gen_ai.agent.name` carries the name alone. |
| `enduser.id` | Span | The authenticated caller, such as `admin@kagent.dev`. |
| `gen_ai.main_agent.id` | Resource | The top-level Agent that the runtime was compiled from. The shorter `gen_ai.main_agent.name` carries the name alone. |
| `kagent.runtime` | Resource | The runtime that served the request, such as `adk-go`. |

The runtime also adds each scalar value in the A2A message's metadata as an `a2a.message.metadata.<key>` attribute, so a client can tag a request and search for it later. Unlike the correlation attributes, these tags stay on the `a2a.request` span alone, so a search on one returns that span instead of the whole subtree.

Expand Down