Repository navigation
Add the session and message spans to the trace exporter - #359
esherman-credo wants to merge 6 commits into
Conversation
The exporter already starts a span for any mapped method in methodsEvaluated, but it had no sources for the attributes the mapping lists on the session and message spans, so those spans went out without them. This adds sources for acs.session.reason, acs.content.types, and the session span's optional acs.tenant_id, acs.user.id and acs.policy_mode. acs.content.types holds one entry per content item, in wire order, and is not deduplicated. The mapping does not say whether it is a set; this keeps the counts, and is a one-line change if the Spec leads prefer a set. This Guardian does not evaluate the four methods yet, and PR GenAI-Security-Project#229 keeps them out of methods_evaluated, so the spans do not appear in a live run. The tests drive them through the exporter's methodsEvaluated option, as GenAI-Security-Project#322 describes. Provenance on message content items is left for a later slice. Part of GenAI-Security-Project#322 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eli Sherman <esherman@credo.ai>
The mapping's provenance rule applies to every hook payload that attaches Provenance. The message hooks attach it to each content item, and sessionStart to its Intent as parser_provenance, but the exporter read it only from tool arguments and outputs, so acs.message.user, acs.message.agent and acs.session went out without it. provenanceObjects now reads those carriers too, with the same columns, order, padding and limit. The session and message tests now check that exactly one span with the mapped name exists before asserting that an attribute is absent, so a dropped span can no longer pass them. They also compare acs.request_id and acs.agent.id against values the test chose, and run the decision and guardian_error checks on all four methods. Part of GenAI-Security-Project#322 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eli Sherman <esherman@credo.ai>
GenAI-Security-Project#322 asks for the sample trace and the trace walkthrough to show the new spans. The sample's exporter now also spans steps/sessionStart, steps/userMessage, steps/agentResponse and steps/sessionEnd, through methodsEvaluated as the tests do, and the generator sends one request for each. This Guardian does not dispatch those steps yet, so each span carries a guardian_error failure rather than a decision; its name and attributes, message provenance included, are real exporter output. A sample test pins the four spans and the attributes their requests carried. The tree README says why "Produce a trace" shows no session or message span in a live run and points to the sample, and the Guardian README and the OpenTelemetry page count the sample's spans again. Part of GenAI-Security-Project#322 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eli Sherman <esherman@credo.ai>
Brings in GenAI-Security-Project#309, which changed the exporter's start() to take whether the Guardian trusted the request. The session and message tests now pass TRUSTED, the sample is regenerated, and the sample test and the two sample descriptions count the session and message spans beside GenAI-Security-Project#309's rejected request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eli Sherman <esherman@credo.ai>
drk-ram
left a comment
There was a problem hiding this comment.
I tested head 483092e92af54926e0d697162f3b7d2e92f7afc7 merged onto integration 1b8f970370011cbcbb491ffbcafff210d62f6fbd on Windows.
Bun 1.4.4 and 1.3.11 both pass bun run typecheck. Both pass all 21 new cases with bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps'. Both also pass all 11 cases in bun test test/sample-trace.test.ts, including live regeneration. python -m pytest -q tests\test_trace_docs_example.py tests\test_trace_mapping_defaults.py passes all 12 guards.
I removed the added attribute sources, deduplicated content types, and removed message and Intent provenance separately. Each mutation fails its expected new cases on both runtimes. Removing the sample's session-method override fails the regeneration check on both. Restoring every change passes the 21 session tests and 11 sample tests again.
The sample carries all four new spans with failure events, not invented allow decisions, while live supported methods remain unchanged. That matches slice 1 of #322 and the README's non-claim language.
The unfiltered exporter/sample run is not green: 1.4.4 gives 103 pass and eight failures, all reproduced on the pinned baseline. The 1.3.11 run reaches the 120-second timeout seen in the baseline Guardian run. Those limits do not affect the passing isolated session and sample checks.
drk-ram
left a comment
There was a problem hiding this comment.
Re-verified at head 057d9bc6cee63f7b68378c90eb501502f3815521. My earlier approval was of 483092e.
What changed between the two heads. 057d9bc is a merge commit with parents 483092e and bba1b84 (current integration). git diff-tree --cc on it is empty, so no conflict was hand-resolved and the merge introduced no content of its own. None of this PR's eight files differ between 483092e and 057d9bc: otel-mapping.ts, otel-trace-exporter.test.ts, make-sample-trace.ts, sample-trace.test.ts, acs-trace-sample.jsonl, docs/spec/trace/extend_opentelemetry.md and the two READMEs are byte-identical. The only things that moved are upstream changes from integration, most visibly the conformance-test cleanup in #345 and three small edits under docs/spec/trace/ that this PR does not touch.
Checks on this head, Windows, Bun 1.4.4 (C:\Users\rdurga\AppData\Roaming\npm\node_modules\bun\bin\bun.exe), fresh bun install --frozen-lockfile:
bun run typecheckpasses.bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps': 21 pass, 0 fail.bun test test/sample-trace.test.ts: 11 pass, 0 fail, including "the committed sample trace is current". Note that the live regeneration needsACS_OPA_PATHset to the installedopa.exein the real process environment under Bun on Windows. Without it the regeneration test errors inagt-bridgebefore reaching this PR's code. That is an environment requirement, not a defect here.python -m pytest -q tests/test_trace_docs_example.py tests/test_trace_mapping_defaults.py: 12 passed. Current integration'stests/conftest.pynow importsmkdocs, so I installed the repository's declared docs dependencies first.- Unfiltered exporter file: 92 pass, 8 fail. The eight are the 0600/0700 file-permission case, six
keepLoopbackOffProxycases, and the environment-independence case. They are POSIX permission and proxy-environment behaviour, none sit in the new session/message cases, and they are the same class I reproduced on the pinned baseline last time.
No defect was introduced by the merge, and the earlier findings on this PR still stand as resolved: the sample still carries the four new spans with failure events rather than invented allow decisions, which matches slice 1 of #322 and the README's non-claim language.
Approving 057d9bc.
bar-capsule
left a comment
There was a problem hiding this comment.
Checked each new source against the hook schemas: tenant_id on AcsParams, user_identity.user_id and policy_mode on sessionStart, reason on sessionEnd, content[].type on both message hooks, and intent.parser_provenance. All match, and the required-attribute check reads otel-mapping.json rather than a local list.
Your three questions:
acs.content.types: one entry per item, in order, as here, so a reader can count items of each type.acs.tenant_id: emit it when set, as here.otel-mapping.jsonshould list it underconditional_attributes, to matchevents.md.acs.user.id:payload.user_identity.user_id, as here. It's the session's principal, whilemetadata.user_contextis per request.
I'll put all three into otel-mapping.json in one small spec PR.
Non-blocking: validate each test envelope against request-envelope.json and its hook schema, so a source can't pass on an envelope the schema would reject.
The test gave one content item `origin: "retrieval"`, which provenance.json does not allow (it is `retrieved`), so both message envelopes in that test failed their hook schemas. Found by validating every slice-1 test envelope against request-envelope.json and its hook schema, as the review asked. The other envelopes are valid, apart from the acs.content.types edge cases, which are invalid on purpose. Part of GenAI-Security-Project#322 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eli Sherman <esherman@credo.ai>
@bar-capsule I did the text envelop validation and found a type in a fixture. The |
drk-ram
left a comment
There was a problem hiding this comment.
Re-verified at head b744d4d2ec7153f701ef01ccd4a616e19123ab5d. Approving. My previous approval was of 057d9bc.
What changed between the two heads. One commit, one file: reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts, 2 lines. In the message-provenance test, the third content item's origin changes from retrieval to retrieved, and the expected acs.provenance.origin column changes from ["user_input", "retrieval"] to ["user_input", "retrieved"]. The origin enum in specification/v0.1.0/provenance.json is user_input, system, tool_output, retrieved, agent_generated, a2a_inbound, external, so retrieval was never a legal value and the fixture would have been rejected by the schema it claims to exercise. This is the fixture error the author found by validating the envelopes, which is what I had suggested as a non-blocking follow-up. No production code moved (otel-mapping.ts, the sample generator, the committed sample and the docs are identical to 057d9bc), and the exporter just copies the origin string through, so the assertion is unchanged in meaning. The branch is still based on current integration (bba1b84).
Checks on this head, Windows, Bun 1.4.4 (C:\Users\rdurga\AppData\Roaming\npm\node_modules\bun\bin\bun.exe), clean worktree, bun install --frozen-lockfile, ACS_OPA_PATH set to the installed opa.exe:
bun run typecheckpasses.bun test packages/guardian/test/otel-trace-exporter.test.ts -t 'the session and message steps': 21 pass, 0 fail.bun test test/sample-trace.test.ts: 11 pass, 0 fail. On the first, cold run the "committed sample trace is current" regeneration case failed after about 15 s while the OPA bundle compiled for the first time. It passes in about 3.5 s both alone and in a full re-run, so this is cold-start latency against the case timeout, not a defect in the PR.python -m pytest -q tests/test_trace_docs_example.py tests/test_trace_mapping_defaults.py: 12 passed.
The earlier findings on this PR still stand as resolved: the sample carries the four new spans with failure events and no invented allow decisions. Approving b744d4d.
bar-capsule
left a comment
There was a problem hiding this comment.
Approved. retrieved is in provenance.json's origin list. Non-blocking: if the schema check you ran can live in the test, a future fixture can't drift the same way.
What changed
The trace exporter now writes the four session and message spans that
trace/otel-mapping.jsondefines:acs.session,acs.message.user,acs.message.agentandacs.session.end. Each span carries the required and optional attributes the mapping lists, and the provenance the payload attaches. The sample trace and the READMEs show the four spans.Which issue does this implement
Part of #322 (slice 1)
meaning, so it needs no issue
Base branch
integration, because this touches the specification, schemas, a referenceimplementation, an adapter, tests, or CI
main, because every changed path is on the documentation lane allowlist inCONTRIBUTING.md
Type of change
Specification changes
The only change under
docs/spec/is one count intrace/extend_opentelemetry.md: the sample now holds nine Guardian spans. No normative text or schema changes.Breaking for implementers? No.
Details
New attribute sources (
packages/guardian/src/otel-mapping.ts):acs.session.reasonacs.session.endpayload.reasonacs.content.typesacs.message.userpayload.content[].typeacs.tenant_idacs.sessionparams.tenant_idacs.user.idacs.sessionpayload.user_identity.user_idacs.policy_modeacs.sessionpayload.policy_modeacs.session.id,acs.request_idandacs.agent.idalready had sources.Provenance. The exporter now also reads Provenance from
payload.content[].provenance(both message steps) andpayload.intent.parser_provenance(sessionStart). It uses the same columns, order,""padding and limit as the tool spans.Live runs. This Guardian does not dispatch the four steps yet, and #229 keeps them out of
methods_evaluated. So a live run shows no new spans. As #322 describes, the tests and the sample generator give the four methods to the exporter throughmethodsEvaluated. In the sample, each new span has anacs.decision.failureevent (guardian_error), because the Guardian answers-32011. When slice 2 adds the "accepted, not evaluated" marker, these spans can appear in live runs.Questions for the Spec leads
acs.content.types: one entry per content item, in order (this PR), or one entry per distinct type? The spec does not say.acs.tenant_idonacs.session:otel-mapping.jsonlists it as optional, andevents.mdlists it as required "if set". This PR emits it when it is set, which satisfies both.acs.user.id: this PR readspayload.user_identity.user_id. The envelope also hasmetadata.user_context.user_id. Which one should the span use?I tested this
integrationat b383e40)uv run pytest -vpasses on my machine (922 passed, 1 skipped)uv run mkdocs build --strictpasses on my machineAlso, in
reference-implementations/agt:bun run typecheckpasses.bun test: 1307 pass, 1 skip, 1 fail. The failure ishosts/claude-code/test/posture.test.tsand occurs only on Bun 1.4.x, where the error message starts withECONNREFUSED:. CI uses Bun 1.3.11. The same test fails onintegration.otel-mapping.tsare covered by tests.Checklist
git commit -s(required by the DCO)Security
🤖 Generated with Claude Code