From ca40590890e2aa472f8c17aa88e646094e89af5e Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Tue, 20 Oct 2026 14:28:22 -0500 Subject: [PATCH 1/3] Extract audit guidance into reusable snippets Move the tool-attribute disclosure and trace-verification steps into reusable observability snippets, keeping the audit prompts page concise. Signed-off-by: Rachael Graham --- .../observability/audit-tool-attributes.md | 3 ++ .../observability/audit-verify-steps.md | 35 +++++++++++++++++ .../kagent/1.x/observability/audit-prompts.md | 38 +------------------ 3 files changed, 40 insertions(+), 36 deletions(-) create mode 100644 docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md create mode 100644 docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md new file mode 100644 index 00000000..ab01021f --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md @@ -0,0 +1,3 @@ +A tool's own arguments and reply are recorded separately, on the `execute_tool` span of each tool call, in the `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` attributes. `config.runtimeDefaults.telemetry.captureContent` does not govern those two attributes. The runtime records them whenever tracing is on. + +Treat them as part of the audit record, and as a disclosure to account for on an agent whose tools return sensitive data. diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md new file mode 100644 index 00000000..6c773200 --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md @@ -0,0 +1,35 @@ +1. Send a request to the new Session. + ```bash + export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') + kagent agent invoke --session $SESSION_ID --task "Audit check: what is 2+2?" + ``` + +2. Find the captured request in your tracing backend. + {{< tabs >}} + {{% tab name="Grafana with Tempo" %}} + 1. Forward the Grafana port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 + ``` + 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. {{< reuse "kagent-docs/snippets/observability/grafana-password.md" >}} + 3. Open **Explore**, select the **Tempo** data source, and select the **TraceQL** query type. + 4. Run the following query, which returns the model calls whose request contains the phrase. + ```text + { span.gen_ai.input.messages =~ ".*Audit check.*" } + ``` + 5. Open a trace, and select its `generate_content` span. The **Span Attributes** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. + {{% /tab %}} + {{% tab name="Jaeger" %}} + 1. Forward the Jaeger query port, and leave the command running. + ```bash + kubectl port-forward -n telemetry svc/jaeger 16686:16686 + ``` + 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). + 3. From the **Service** list, select `my-first-agent`. From the **Operation** list, select the `generate_content` operation for your model, such as `generate_content gpt-4.1-mini`, and click **Find Traces**. + 4. Open the most recent trace, and expand the `generate_content` span. The **Tags** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. + {{% /tab %}} + {{< /tabs >}} + + If both attributes read `{}`, the Session started without content capture. Check that the previous section printed `Current`, and create a new Session if it did not. + +3. To collect every model call of one conversation, search by its conversation ID. Every span of the conversation carries the ID in the `gen_ai.conversation.id` attribute. For the other attributes that you can search by, see [Correlation attributes]({{< link path="observability/tracing#correlation-attributes" >}}). diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/1.x/observability/audit-prompts.md index 7799b77c..52c44c74 100644 --- a/docs-site/content/kagent/1.x/observability/audit-prompts.md +++ b/docs-site/content/kagent/1.x/observability/audit-prompts.md @@ -40,7 +40,7 @@ The record holds more than the prompts that your team wrote. The system instruction holds the instructions that the runtime appends as well as the prompt your team wrote, and the message history holds the person's messages, the agent's earlier turns, and the model's tool calls. -A tool's own arguments and reply are recorded separately, on the `execute_tool` span of each tool call, in the `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` attributes. `config.runtimeDefaults.telemetry.captureContent` does not govern those two attributes. The runtime records them whenever tracing is on. Treat them as part of the audit record, and as a disclosure to account for on an agent whose tools return sensitive data. +{{< reuse "kagent-docs/snippets/observability/audit-tool-attributes.md" >}} On the `codex` and `claude` runtimes, the `invoke_agent` span holds only the current turn's prompt and the reply to that prompt. The `chat` spans hold no content. The `execute_tool` span holds tool arguments and successful results in the `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` attributes. @@ -127,41 +127,7 @@ Turn on content capture in the kagent Helm release, then create a Session that p Send a request that contains a distinctive phrase, then find that phrase in the captured request. -1. Send a request to the new Session. - ```bash - export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id') - kagent agent invoke --session $SESSION_ID --task "Audit check: what is 2+2?" - ``` - -2. Find the captured request in your tracing backend. - {{< tabs >}} - {{% tab name="Grafana with Tempo" %}} - 1. Forward the Grafana port, and leave the command running. - ```bash - kubectl port-forward -n telemetry svc/kube-prometheus-stack-grafana 3000:80 - ``` - 2. In your browser, open Grafana at [http://localhost:3000](http://localhost:3000), and log in. {{< reuse "kagent-docs/snippets/observability/grafana-password.md" >}} - 3. Open **Explore**, select the **Tempo** data source, and select the **TraceQL** query type. - 4. Run the following query, which returns the model calls whose request contains the phrase. - ```text - { span.gen_ai.input.messages =~ ".*Audit check.*" } - ``` - 5. Open a trace, and select its `generate_content` span. The **Span Attributes** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. - {{% /tab %}} - {{% tab name="Jaeger" %}} - 1. Forward the Jaeger query port, and leave the command running. - ```bash - kubectl port-forward -n telemetry svc/jaeger 16686:16686 - ``` - 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). - 3. From the **Service** list, select `my-first-agent`. From the **Operation** list, select the `generate_content` operation for your model, such as `generate_content gpt-4.1-mini`, and click **Find Traces**. - 4. Open the most recent trace, and expand the `generate_content` span. The **Tags** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. - {{% /tab %}} - {{< /tabs >}} - - If both attributes read `{}`, the Session started without content capture. Check that the previous section printed `Current`, and create a new Session if it did not. - -3. To collect every model call of one conversation, search by its conversation ID. Every span of the conversation carries the ID in the `gen_ai.conversation.id` attribute. For the other attributes that you can search by, see [Correlation attributes]({{< link path="observability/tracing#correlation-attributes" >}}). +{{< reuse "kagent-docs/snippets/observability/audit-verify-steps.md" >}} > [!CAUTION] > Anyone who can read your tracing or logging backend can now read the prompts and replies of every agent. Restrict access to the backend, and set a retention period that meets your compliance requirements. From e2f284d3f88db7d7665f0d22199e9fde153854cd Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Tue, 20 Oct 2026 14:34:23 -0500 Subject: [PATCH 2/3] conventions check Signed-off-by: Rachael Graham --- .../snippets/observability/audit-tool-attributes.md | 2 +- .../kagent-docs/snippets/observability/audit-verify-steps.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md index ab01021f..123c15d4 100644 --- a/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md @@ -1,3 +1,3 @@ A tool's own arguments and reply are recorded separately, on the `execute_tool` span of each tool call, in the `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` attributes. `config.runtimeDefaults.telemetry.captureContent` does not govern those two attributes. The runtime records them whenever tracing is on. -Treat them as part of the audit record, and as a disclosure to account for on an agent whose tools return sensitive data. +Treat the two tool attributes as part of the audit record, and as a disclosure to account for on an agent whose tools return sensitive data. diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md index 6c773200..e790f553 100644 --- a/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md @@ -17,7 +17,7 @@ ```text { span.gen_ai.input.messages =~ ".*Audit check.*" } ``` - 5. Open a trace, and select its `generate_content` span. The **Span Attributes** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. + 5. Open a trace, and select its `generate_content` span. The **Span Attributes** section shows the three attributes described in [What a record holds](#what-a-record-holds). {{% /tab %}} {{% tab name="Jaeger" %}} 1. Forward the Jaeger query port, and leave the command running. @@ -26,7 +26,7 @@ ``` 2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686). 3. From the **Service** list, select `my-first-agent`. From the **Operation** list, select the `generate_content` operation for your model, such as `generate_content gpt-4.1-mini`, and click **Find Traces**. - 4. Open the most recent trace, and expand the `generate_content` span. The **Tags** section shows the three attributes that [What a record holds](#what-a-record-holds) describes. + 4. Open the most recent trace, and expand the `generate_content` span. The **Tags** section shows the three attributes described in [What a record holds](#what-a-record-holds). {{% /tab %}} {{< /tabs >}} From 307c2b3571da180bc16f67092e01d90be982f5ae Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Tue, 20 Oct 2026 14:55:35 -0500 Subject: [PATCH 3/3] review comments Signed-off-by: Rachael Graham --- .../kagent-docs/snippets/observability/audit-tool-attributes.md | 2 +- .../kagent-docs/snippets/observability/audit-verify-steps.md | 2 +- docs-site/content/kagent/1.x/observability/audit-prompts.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md index 123c15d4..ab912e89 100644 --- a/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-tool-attributes.md @@ -1,3 +1,3 @@ A tool's own arguments and reply are recorded separately, on the `execute_tool` span of each tool call, in the `gcp.vertex.agent.tool_call_args` and `gcp.vertex.agent.tool_response` attributes. `config.runtimeDefaults.telemetry.captureContent` does not govern those two attributes. The runtime records them whenever tracing is on. -Treat the two tool attributes as part of the audit record, and as a disclosure to account for on an agent whose tools return sensitive data. +Treat the two tool attributes as part of the audit record. On an agent whose tools return sensitive data, account for them as a disclosure. diff --git a/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md index e790f553..353372d0 100644 --- a/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md +++ b/docs-site/assets/kagent-docs/snippets/observability/audit-verify-steps.md @@ -30,6 +30,6 @@ {{% /tab %}} {{< /tabs >}} - If both attributes read `{}`, the Session started without content capture. Check that the previous section printed `Current`, and create a new Session if it did not. + If `gen_ai.input.messages` and `gen_ai.output.messages` read `{}`, the Session started without content capture. Check that the previous section printed `Current`, and create a new Session if it did not. 3. To collect every model call of one conversation, search by its conversation ID. Every span of the conversation carries the ID in the `gen_ai.conversation.id` attribute. For the other attributes that you can search by, see [Correlation attributes]({{< link path="observability/tracing#correlation-attributes" >}}). diff --git a/docs-site/content/kagent/1.x/observability/audit-prompts.md b/docs-site/content/kagent/1.x/observability/audit-prompts.md index 52c44c74..c5b385ee 100644 --- a/docs-site/content/kagent/1.x/observability/audit-prompts.md +++ b/docs-site/content/kagent/1.x/observability/audit-prompts.md @@ -9,7 +9,7 @@ Audit the prompts (inputs) and replies (outputs) that pass between your agents a ## About prompt auditing -{{< reuse "kagent-docs/snippets/name-product.md" >}} records prompts and replies in traces. When you turn on content capture, the runtime records the agent's request and the model's reply on its spans. You then find the records in your tracing backend, by agent, by conversation, or by the text that they contain. +{{< reuse "kagent-docs/snippets/name-product.md" >}} records prompts and replies in traces. When you turn on content capture, the runtime records the agent's request and the model's reply on its spans. You then find the records in your tracing backend. The `kagent` runtime does not write prompts or replies to log records, so a logging backend on its own cannot hold a prompt audit trail. Capture is off by default, because the content can include sensitive user and model data.