Skip to content

Add the session and message spans to the trace exporter - #359

Open
esherman-credo wants to merge 6 commits into
GenAI-Security-Project:integrationfrom
esherman-credo:refimpl/trace-session-message-spans
Open

esherman-credo wants to merge 6 commits into
GenAI-Security-Project:integrationfrom
esherman-credo:refimpl/trace-session-message-spans

Conversation

@esherman-credo

Copy link
Copy Markdown

What changed

The trace exporter now writes the four session and message spans that trace/otel-mapping.json defines: acs.session, acs.message.user, acs.message.agent and acs.session.end. Each span carries the required and optional attributes the mapping lists, and the provenance the payload attaches. The sample trace and the READMEs show the four spans.

Which issue does this implement

Part of #322 (slice 1)

  • This is an editorial correction (typo, grammar, link, formatting) with no change in
    meaning, so it needs no issue

Base branch

  • integration, because this touches the specification, schemas, a reference
    implementation, an adapter, tests, or CI
  • main, because every changed path is on the documentation lane allowlist in
    CONTRIBUTING.md

Type of change

  • Specification change (schema, hooks, events, AgBOM)
  • Reference implementation or adapter
  • Documentation
  • Tooling or CI
  • Governance (licensing, security policy, contributor docs)

Specification changes

The only change under docs/spec/ is one count in trace/extend_opentelemetry.md: the sample now holds nine Guardian spans. No normative text or schema changes.

  • I opened a Discussion before this PR
  • Schema changes validate against the JSON Schema spec
  • I described the impact on downstream implementers below

Breaking for implementers? No.

Details

New attribute sources (packages/guardian/src/otel-mapping.ts):

Attribute Span Wire field
acs.session.reason acs.session.end payload.reason
acs.content.types acs.message.user payload.content[].type
acs.tenant_id acs.session params.tenant_id
acs.user.id acs.session payload.user_identity.user_id
acs.policy_mode acs.session payload.policy_mode

acs.session.id, acs.request_id and acs.agent.id already had sources.

Provenance. The exporter now also reads Provenance from payload.content[].provenance (both message steps) and payload.intent.parser_provenance (sessionStart). It uses the same columns, order, "" padding and limit as the tool spans.

Live runs. This Guardian does not dispatch the four steps yet, and #229 keeps them out of methods_evaluated. So a live run shows no new spans. As #322 describes, the tests and the sample generator give the four methods to the exporter through methodsEvaluated. In the sample, each new span has an acs.decision.failure event (guardian_error), because the Guardian answers -32011. When slice 2 adds the "accepted, not evaluated" marker, these spans can appear in live runs.

Questions for the Spec leads

  1. acs.content.types: one entry per content item, in order (this PR), or one entry per distinct type? The spec does not say.
  2. acs.tenant_id on acs.session: otel-mapping.json lists it as optional, and events.md lists it as required "if set". This PR emits it when it is set, which satisfies both.
  3. acs.user.id: this PR reads payload.user_identity.user_id. The envelope also has metadata.user_context.user_id. Which one should the span use?

I tested this

  • I synced my branch with the base branch before opening this (merged integration at b383e40)
  • uv run pytest -v passes on my machine (922 passed, 1 skipped)
  • uv run mkdocs build --strict passes on my machine

Also, in reference-implementations/agt:

  • bun run typecheck passes.
  • bun test: 1307 pass, 1 skip, 1 fail. The failure is hosts/claude-code/test/posture.test.ts and occurs only on Bun 1.4.x, where the error message starts with ECONNREFUSED:. CI uses Bun 1.3.11. The same test fails on integration.
  • The new tests were written first and failed before the code changed. Each test that checks for an absent attribute first checks that the span exists.
  • All 11 changed executable lines in otel-mapping.ts are covered by tests.
  • A local run with a real Guardian over HTTP, with the four methods added to the exporter's list, wrote the four spans with the expected attributes.

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

🤖 Generated with Claude Code

esherman-credo and others added 4 commits October 29, 2026 15:47
The exporter already starts a span for any mapped method in methodsEvaluated,
but it had no sources for the attributes the mapping lists on the session and
message spans, so those spans went out without them. This adds sources for
acs.session.reason, acs.content.types, and the session span's optional
acs.tenant_id, acs.user.id and acs.policy_mode.

acs.content.types holds one entry per content item, in wire order, and is not
deduplicated. The mapping does not say whether it is a set; this keeps the
counts, and is a one-line change if the Spec leads prefer a set.

This Guardian does not evaluate the four methods yet, and PR GenAI-Security-Project#229 keeps them
out of methods_evaluated, so the spans do not appear in a live run. The tests
drive them through the exporter's methodsEvaluated option, as GenAI-Security-Project#322 describes.
Provenance on message content items is left for a later slice.

Part of GenAI-Security-Project#322

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eli Sherman <esherman@credo.ai>
The mapping's provenance rule applies to every hook payload that attaches
Provenance. The message hooks attach it to each content item, and
sessionStart to its Intent as parser_provenance, but the exporter read it
only from tool arguments and outputs, so acs.message.user, acs.message.agent
and acs.session went out without it. provenanceObjects now reads those
carriers too, with the same columns, order, padding and limit.

The session and message tests now check that exactly one span with the
mapped name exists before asserting that an attribute is absent, so a dropped
span can no longer pass them. They also compare acs.request_id and
acs.agent.id against values the test chose, and run the decision and
guardian_error checks on all four methods.

Part of GenAI-Security-Project#322

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eli Sherman <esherman@credo.ai>
GenAI-Security-Project#322 asks for the sample trace and the trace walkthrough to show the new
spans. The sample's exporter now also spans steps/sessionStart,
steps/userMessage, steps/agentResponse and steps/sessionEnd, through
methodsEvaluated as the tests do, and the generator sends one request for
each. This Guardian does not dispatch those steps yet, so each span carries a
guardian_error failure rather than a decision; its name and attributes,
message provenance included, are real exporter output. A sample test pins the
four spans and the attributes their requests carried.

The tree README says why "Produce a trace" shows no session or message span
in a live run and points to the sample, and the Guardian README and the
OpenTelemetry page count the sample's spans again.

Part of GenAI-Security-Project#322

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eli Sherman <esherman@credo.ai>
Brings in GenAI-Security-Project#309, which changed the exporter's start() to take whether the
Guardian trusted the request. The session and message tests now pass TRUSTED,
the sample is regenerated, and the sample test and the two sample
descriptions count the session and message spans beside GenAI-Security-Project#309's rejected
request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eli Sherman <esherman@credo.ai>

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested head 483092e92af54926e0d697162f3b7d2e92f7afc7 merged onto integration 1b8f970370011cbcbb491ffbcafff210d62f6fbd on Windows.

Bun 1.4.4 and 1.3.11 both pass bun run typecheck. Both pass all 21 new cases with bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps'. Both also pass all 11 cases in bun test test/sample-trace.test.ts, including live regeneration. python -m pytest -q tests\test_trace_docs_example.py tests\test_trace_mapping_defaults.py passes all 12 guards.

I removed the added attribute sources, deduplicated content types, and removed message and Intent provenance separately. Each mutation fails its expected new cases on both runtimes. Removing the sample's session-method override fails the regeneration check on both. Restoring every change passes the 21 session tests and 11 sample tests again.

The sample carries all four new spans with failure events, not invented allow decisions, while live supported methods remain unchanged. That matches slice 1 of #322 and the README's non-claim language.

The unfiltered exporter/sample run is not green: 1.4.4 gives 103 pass and eight failures, all reproduced on the pinned baseline. The 1.3.11 run reaches the 120-second timeout seen in the baseline Guardian run. Those limits do not affect the passing isolated session and sample checks.

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at head 057d9bc6cee63f7b68378c90eb501502f3815521. My earlier approval was of 483092e.

What changed between the two heads. 057d9bc is a merge commit with parents 483092e and bba1b84 (current integration). git diff-tree --cc on it is empty, so no conflict was hand-resolved and the merge introduced no content of its own. None of this PR's eight files differ between 483092e and 057d9bc: otel-mapping.ts, otel-trace-exporter.test.ts, make-sample-trace.ts, sample-trace.test.ts, acs-trace-sample.jsonl, docs/spec/trace/extend_opentelemetry.md and the two READMEs are byte-identical. The only things that moved are upstream changes from integration, most visibly the conformance-test cleanup in #345 and three small edits under docs/spec/trace/ that this PR does not touch.

Checks on this head, Windows, Bun 1.4.4 (C:\Users\rdurga\AppData\Roaming\npm\node_modules\bun\bin\bun.exe), fresh bun install --frozen-lockfile:

  • bun run typecheck passes.
  • bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps': 21 pass, 0 fail.
  • bun test test/sample-trace.test.ts: 11 pass, 0 fail, including "the committed sample trace is current". Note that the live regeneration needs ACS_OPA_PATH set to the installed opa.exe in the real process environment under Bun on Windows. Without it the regeneration test errors in agt-bridge before reaching this PR's code. That is an environment requirement, not a defect here.
  • python -m pytest -q tests/test_trace_docs_example.py tests/test_trace_mapping_defaults.py: 12 passed. Current integration's tests/conftest.py now imports mkdocs, so I installed the repository's declared docs dependencies first.
  • Unfiltered exporter file: 92 pass, 8 fail. The eight are the 0600/0700 file-permission case, six keepLoopbackOffProxy cases, and the environment-independence case. They are POSIX permission and proxy-environment behaviour, none sit in the new session/message cases, and they are the same class I reproduced on the pinned baseline last time.

No defect was introduced by the merge, and the earlier findings on this PR still stand as resolved: the sample still carries the four new spans with failure events rather than invented allow decisions, which matches slice 1 of #322 and the README's non-claim language.

Approving 057d9bc.

bar-capsule
bar-capsule previously approved these changes Nov 4, 2026

@bar-capsule bar-capsule left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked each new source against the hook schemas: tenant_id on AcsParams, user_identity.user_id and policy_mode on sessionStart, reason on sessionEnd, content[].type on both message hooks, and intent.parser_provenance. All match, and the required-attribute check reads otel-mapping.json rather than a local list.

Your three questions:

  1. acs.content.types: one entry per item, in order, as here, so a reader can count items of each type.
  2. acs.tenant_id: emit it when set, as here. otel-mapping.json should list it under conditional_attributes, to match events.md.
  3. acs.user.id: payload.user_identity.user_id, as here. It's the session's principal, while metadata.user_context is per request.

I'll put all three into otel-mapping.json in one small spec PR.

Non-blocking: validate each test envelope against request-envelope.json and its hook schema, so a source can't pass on an envelope the schema would reject.

The test gave one content item `origin: "retrieval"`, which provenance.json
does not allow (it is `retrieved`), so both message envelopes in that test
failed their hook schemas. Found by validating every slice-1 test envelope
against request-envelope.json and its hook schema, as the review asked. The
other envelopes are valid, apart from the acs.content.types edge cases, which
are invalid on purpose.

Part of GenAI-Security-Project#322

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eli Sherman <esherman@credo.ai>
@esherman-credo

Copy link
Copy Markdown
Author

Checked each new source against the hook schemas: tenant_id on AcsParams, user_identity.user_id and policy_mode on sessionStart, reason on sessionEnd, content[].type on both message hooks, and intent.parser_provenance. All match, and the required-attribute check reads otel-mapping.json rather than a local list.

Your three questions:

  1. acs.content.types: one entry per item, in order, as here, so a reader can count items of each type.
  2. acs.tenant_id: emit it when set, as here. otel-mapping.json should list it under conditional_attributes, to match events.md.
  3. acs.user.id: payload.user_identity.user_id, as here. It's the session's principal, while metadata.user_context is per request.

I'll put all three into otel-mapping.json in one small spec PR.

Non-blocking: validate each test envelope against request-envelope.json and its hook schema, so a source can't pass on an envelope the schema would reject.

@bar-capsule I did the text envelop validation and found a type in a fixture. The message-provenance test used origin: "retrieval", which provenance.json doesn't allow. Pushed a fix and requesting re-review

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at head b744d4d2ec7153f701ef01ccd4a616e19123ab5d. Approving. My previous approval was of 057d9bc.

What changed between the two heads. One commit, one file: reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts, 2 lines. In the message-provenance test, the third content item's origin changes from retrieval to retrieved, and the expected acs.provenance.origin column changes from ["user_input", "retrieval"] to ["user_input", "retrieved"]. The origin enum in specification/v0.1.0/provenance.json is user_input, system, tool_output, retrieved, agent_generated, a2a_inbound, external, so retrieval was never a legal value and the fixture would have been rejected by the schema it claims to exercise. This is the fixture error the author found by validating the envelopes, which is what I had suggested as a non-blocking follow-up. No production code moved (otel-mapping.ts, the sample generator, the committed sample and the docs are identical to 057d9bc), and the exporter just copies the origin string through, so the assertion is unchanged in meaning. The branch is still based on current integration (bba1b84).

Checks on this head, Windows, Bun 1.4.4 (C:\Users\rdurga\AppData\Roaming\npm\node_modules\bun\bin\bun.exe), clean worktree, bun install --frozen-lockfile, ACS_OPA_PATH set to the installed opa.exe:

  • bun run typecheck passes.
  • bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps': 21 pass, 0 fail.
  • bun test test/sample-trace.test.ts: 11 pass, 0 fail. On the first, cold run the "committed sample trace is current" regeneration case failed after about 15 s while the OPA bundle compiled for the first time. It passes in about 3.5 s both alone and in a full re-run, so this is cold-start latency against the case timeout, not a defect in the PR.
  • python -m pytest -q tests/test_trace_docs_example.py tests/test_trace_mapping_defaults.py: 12 passed.

The earlier findings on this PR still stand as resolved: the sample carries the four new spans with failure events and no invented allow decisions. Approving b744d4d.

@bar-capsule bar-capsule left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. retrieved is in provenance.json's origin list. Non-blocking: if the schema check you ran can live in the test, a future fixture can't drift the same way.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

3 participants