Record every decision failure and count them at session end by bar-capsule · Pull Request #273 · GenAI-Security-Project/agent-control-standard

bar-capsule · 2026-10-13T16:18:06Z

What changed

The rest of the #37 decision, the parts that live outside the Trace event #275 added. §6.4 now requires an audit record for every decision failure under either posture: in the deployment's own audit log, and as an acs.decision.failure event on Trace when the deployment claims ACS-Trace. §4.1 surfaces a session started unguarded as the same event with class handshake_failure. sessionEnd.summary gains decision_failures, a count per failure class that the Observed Agent SHOULD report, and a session that ends without it reads as unmeasurable.

It also closes two gaps from the #275 review. A request the Guardian rejects before trusting it can carry a genuine request's pair, so it produces no failure or decision event under that pair; a deployment that records rejected requests uses other attribute names, as decided on #283. And handshake_failure gets a posture: it comes from the startup posture, since on_decision_failure is never negotiated.

Which issue does this implement

Closes #37
Refs #283, which decided how the reference Guardian records requests it can't vouch for

This is an editorial correction (typo, grammar, link, formatting) with no change in
meaning, so it needs no issue

Base branch

integration, because this touches the specification, schemas, a reference
implementation, an adapter, tests, or CI
main, because every changed path is on the documentation lane allowlist in
CONTRIBUTING.md

Type of change

Specification change (schema, hooks, events, AgBOM)
Reference implementation or adapter
Documentation
Tooling or CI
Governance (licensing, security policy, contributor docs)

Specification changes

I opened a Discussion before this PR
Schema changes validate against the JSON Schema spec
I described the impact on downstream implementers below

No Discussion was opened. The decisions are on #37 and #283.

File	Change
`docs/spec/instrument/specification.md`	§6.4: every decision failure is recorded, fail-open or fail-closed, with the audit log as the MUST and Trace as the SHOULD; the `sessionEnd` count is a SHOULD. §4.1: an unguarded session becomes `acs.decision.failure` with class `handshake_failure` and posture `fail_open` on the session span.
`specification/v0.1.0/hooks/session-end.json`	`summary.decision_failures`: an object keyed by failure class, each value an integer of 0 or more. The keys are limited to the eight `acs.failure.class` values.
`specification/v0.1.0/trace/otel-mapping.json`	No failure or decision event under the pair of a request rejected before it is trusted (failed envelope validation, `SIGNATURE_INVALID`, `REPLAY_DETECTED`, `TIMESTAMP_OUT_OF_WINDOW`). `handshake_failure` covers both startup postures, and its `acs.failure.posture` comes from the startup posture.
`docs/spec/trace/events.md`	The same rejected-request rule.
`docs/spec/conformance.md`	The decision-honoring line follows §6.4.
`docs/spec/instrument/hooks.md`	The `sessionEnd` payload line names the real fields (`reason`, not `session_reason`, plus the missing `abandoned` value) and the new count.
`tests/test_decision_failure_classes.py`	Fails when the classes `session-end.json` counts differ from the `acs.failure.class` values in `otel-mapping.json`.

Breaking for implementers? The schema change is additive: decision_failures is optional. Two rules tighten. §6.4 previously required a record only for a step that proceeded without a decision, and now a fail-closed decision failure needs one too. And a Guardian that used to record a rejected replay, or any other untrusted request, under its claimed pair now records it under other names or not at all.

I tested this

I synced my branch with the base branch before opening this
uv run pytest -v passes on my machine
uv run mkdocs build --strict passes on my machine

pytest reports 919 passed and 1 skipped (a case-insensitive filesystem skip). From reference-implementations/agt/, bun run typecheck is clean and bun test reports 1234 passed, 1 skipped, 0 failed. session-end.json was checked with jsonschema (Draft 2020-12): a count for known classes, a zero, and a payload with no summary are valid; an unknown class, a negative count and a non-integer are rejected.

Merging

Merge with an explicit squash message that keeps the Signed-off-by line.

Checklist

Commits are signed off with git commit -s (required by the DCO)
Prose follows STYLE.md
No secrets, tokens, or internal URLs in the diff

Security

This change has no security impact

It adds detection rather than surface: a blocked handshake now produces an event and a count, and a forged or replayed request can no longer make a genuine decision read as unenforced.

github-actions · 2026-10-13T16:18:18Z

This pull request's description says it closes #37, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging.

…n issue #37 signature_invalid keeps a tampered response apart from organic failures. handshake_failure counts a session that started without a completed handshake, and its event goes on the acs.session span, not a step span. PR #273's sessionEnd count keys on this set of eight classes. Signed-off-by: rocklambros <rock@rockcyber.com>

rocklambros · 2026-10-14T23:04:34Z

@bar-capsule, both classes from your issue #37 answer are in PR #275 now (9cfa988 and 199af89). signature_invalid sits apart from malformed_response, and handshake_failure goes on the acs.session span, counted under the handshake's own request_id and session_id. I merged this PR on top of PR #275 in a scratch tree, and tests/test_decision_failure_classes.py passes, so this one goes green as soon as PR #275 lands. Sorry it took me a day to pick up your answer.

Section 6.4 now requires an audit record for every decision failure under either posture, kept in the deployment's own audit log and emitted on Trace as acs.decision.failure when the deployment claims ACS-Trace. Section 4.1 surfaces a session started unguarded as the same event with class handshake_failure. sessionEnd.summary gains decision_failures, a per-class count the Observed Agent SHOULD report, and a session that ends without it reads as unmeasurable. conformance.md and the hooks.md sessionEnd summary follow, and a test keeps the counted classes equal to the acs.failure.class values in the OTel mapping. Signed-off-by: Bar Kaduri <bar@capsule.security>

…ke_failure its posture A Guardian emits no acs.decision.failure or acs.decision event under the pair of a request it rejects before trusting it: an envelope that fails validation, a failed signature, a replay, or a timestamp outside the window. Such a request can carry a genuine request's pair. handshake_failure now covers the refuse startup posture too, and its acs.failure.posture comes from the startup posture, since on_decision_failure is never negotiated. Section 4.1 records an unguarded session with posture fail_open. Signed-off-by: Bar Kaduri <bar@capsule.security>

bar-capsule · 2026-10-17T20:30:43Z

@rocklambros @afogel ready for review - the rest of #37 on top of #275, plus the two gaps from the #275 review: requests the Guardian rejects stay off the genuine request's IDs, and handshake_failure takes its posture from the startup posture. Small spec PR, all checks green.

rocklambros

@bar-capsule, thanks, this reads well, and both gaps from the PR #275 review are in. I merged it into current integration locally. It merges cleanly, pytest passes and mkdocs build --strict is clean.

Two small asks, both one-line changes in files outside your diff that still state the old rule:

specification/v0.1.0/handshake.json:131, the on_decision_failure description, says: "Every fail-open proceed MUST be recorded as an audit event (Specification 6.4)." Since §6.4 now says every decision failure, could it read: "Every decision failure MUST be recorded as an audit event, fail-open or fail-closed (Specification 6.4)."
docs/identity/overview.md:124 says the Observed Agent "MUST record every fail-open proceed as an audit event". The same change works there: "MUST record every decision failure as an audit event".

One more thing, so you know it's covered: the new rule in events.md (no acs.decision under the pair of a request the Guardian rejects before trusting it) is something our reference Guardian breaks today. I sent it a request that fails schema validation with a forged request_id, and it exported an acs.decision span under that request_id. That's on the reference implementation, not this PR. I'm opening a small Guardian fix to land alongside it, so nothing to do here for that.

…never under its claimed ids PR #273 adds a rule to Trace Events: a Guardian MUST NOT emit acs.decision or acs.decision.failure under the pair of a request it rejects before trusting it. The reference Guardian broke it: an envelope that failed the schema, carrying a forged request_id, got an acs.decision span under that request_id. Following Bar's ruling on issue #283, a request the Guardian cannot vouch for is now a guardian.rejected_request span, with the ids it claimed under guardian.rejected.* and no ACS pair, step attribute or decision event. That covers an envelope that fails validation, a Guardian fault, and a step answered with SIGNATURE_INVALID, REPLAY_DETECTED or TIMESTAMP_OUT_OF_WINDOW. handleAcsRequest validates once before the span opens. dispatch returns its trust verdict beside the response, and a step span gets its pair and step attributes only when it ends as a trusted step. Rebuilt on integration after PR #272: dispatch also takes the envelope log, and the step findings PR #272 added still reach the chain entry. Signed-off-by: rocklambros <rock@rockcyber.com>

Signed-off-by: rocklambros <rock@rockcyber.com>

…tity overview Both still said only a fail-open proceed is recorded, which section 6.4 no longer says after this PR. They now match: every decision failure is recorded, fail-open or fail-closed. Signed-off-by: rocklambros <rock@rockcyber.com>

Addressed in 6e1ec9e: handshake.json and the identity overview now say every decision failure is recorded.

PR GenAI-Security-Project#273 widened section 6.4 to record every decision failure and added sessionEnd's summary.decision_failures, and it closed issue GenAI-Security-Project#37. The template now asks for decision failures by class and drops the issue GenAI-Security-Project#37 caveat. PR GenAI-Security-Project#272 made section 8.6 say a signed HMAC decision is not attributable to the Guardian against the Observed Agent, and the trust-topology block now says so. Signed-off-by: rocklambros <rock@rockcyber.com>

Remove agtbridge/goeval, the port's Go reimplementation of AGT's policy runtime. The README merged in GenAI-Security-Project#256 requires maintained implementations to ship only AGT's runtime as their evaluator. AGT is now the only evaluator. Remove OPA from go.mod. The README allows a second evaluator in tests as a drift check; do not keep goeval for that, because it would have to follow every AGT update. A build without the agteval tag still compiles, and the command then refuses to start. Leave policy.evaluator, EVALUATOR and the agtbridge.Evaluator interface in place, with "agt" as the only value, so another evaluator is one new case. The bridge tests also use the interface to plug in fake evaluators. Switch the bridge tests that used the Go evaluator to AGT. They used it because it was fast and needed no Rust. AGT already ran all 49 recorded sessions before this change. Drop TestNewRefusesAnUnsupportedManifest: it tested which manifest features the Go evaluator refused. Drop the load measurements that only the Go evaluator produced. Regenerate the spec's JSON schema copies and re-record the TypeScript Guardian's answers. No recorded decision changed. Add a signer test against the example key in spec section 10. GenAI-Security-Project#272 defines the HKDF key-derivation parameters there. The Go signer already used exactly those parameters. In docs/conformance.md, mark three expectations of the GenAI-Security-Project#256 README as not met yet: - The command accepts one HMAC key. The README requires at least two keys at once to support key rotation. - A full audit queue, or a write failure, loses audit events the spec requires while decisions continue. The README requires keeping them and denying instead. - An oversized or unreadable request body gets an error, and the engine's deadline equals the declared timeout. The README requires a deny and a lower deadline. Mark the interim verification as met: make check runs the TypeScript conformance probes against this port. GenAI-Security-Project#273 needs no change. TestDecisionFailureRecordsItsClass and TestHandshakeFailureRecordsItsClass in internal/hostadapter already cover recording every decision failure. Verified locally: make check passes, including all 10 external wire checks. The container image and other CI platforms were not tested locally. CI runs those checks. Claude-Session: https://claude.ai/code/session_01SzmLMNjG3KSaCFfg8aKB11 Signed-off-by: Konstantin Trunin <konstantin@trunin.com>

@bar-capsule

…never under its claimed ids (#309) ## What changed PR #273 adds a rule to Trace Events: a Guardian MUST NOT emit `acs.decision` or `acs.decision.failure` under the pair of a request it rejects before trusting it. Our reference Guardian broke that rule. An envelope that failed the schema, carrying a forged `request_id`, got an `acs.decision` span under that `request_id`. This fixes the reference Guardian, following @bar-capsule's ruling on issue #283: keep the rejected request visible, under separate names, never under the genuine request's ids. - **Rejected requests get their own span.** A request the Guardian can't vouch for becomes a `guardian.rejected_request` span. It carries no `acs.session.id`, `acs.request_id`, step attribute, or `acs.decision` event. The ids it claimed go under `guardian.rejected.claimed_session_id` and `guardian.rejected.claimed_request_id`, with `guardian.rejected.reason` and the error code. - **What counts as rejected:** an envelope that fails validation, a request the Guardian couldn't check at all (`guardian_fault`, which still records its `guardian_error`), and a step answered with `SIGNATURE_INVALID`, `REPLAY_DETECTED` or `TIMESTAMP_OUT_OF_WINDOW`. - **The pair is added last.** A trusted step's pair and step attributes go on only when its span ends. `dispatch` now returns its trust verdict next to the response. So a check that runs after the span opens can't leak a rejected request's ids onto a step span, whether it answers with an error or with a deny. That's where PR #229's checks will run. - **Sample trace:** it now includes a forged request that reuses the allowed call's ids. A test checks that the allowed call keeps the only span under its pair. - **Docs:** the READMEs and `extend_opentelemetry.md` say what a rejected span carries. The known limits say plainly that this Guardian checks nothing but the schema yet, so a well-formed forgery or replay is still recorded as a step until PR #229 lands. @sabasiddique1, this implements item 6 of issue #283, so you can skip that item. Items 1 to 5 and 7 are still yours. **Merge order:** this is safe to merge before PR #273. Before that rule exists, the only effect is that a rejected request moves off the step spans. Please don't promote it to `main` without PR #273, or the other way round. ## Which issue does this implement Part of #283 Refs #37 - [ ] This is an editorial correction (typo, grammar, link, formatting) with no change in meaning, so it needs no issue ## Base branch - [x] `integration`, because this touches the specification, schemas, a reference implementation, an adapter, tests, or CI - [ ] `main`, because every changed path is on the documentation lane allowlist in CONTRIBUTING.md ## Type of change - [ ] Specification change (schema, hooks, events, AgBOM) - [x] Reference implementation or adapter - [x] Documentation - [ ] Tooling or CI - [ ] Governance (licensing, security policy, contributor docs) ## Specification changes No normative text changes. `docs/spec/trace/extend_opentelemetry.md` changes only where it describes what the reference Guardian exports. **Breaking for implementers?** No ACS change. Anyone reading the reference Guardian's trace sees a schema-invalid envelope as a `guardian.rejected_request` span. Before, it was a `gen_ai.tool.call` with an `envelope_invalid` deny. ## I tested this - [x] I synced my branch with the base branch before opening this - [x] `uv run pytest -v` passes on my machine - [x] `uv run mkdocs build --strict` passes on my machine From `reference-implementations/agt/`, `bun run typecheck` is clean and `bun test` passes (1245). I checked the new tests by breaking each behavior on purpose. With trust always true, a late rejection ignored, the pair attached early, `dispatch`'s verdict dropped, the fault event removed, or the arrival time ignored, the matching tests fail. Before writing this I ran a six-perspective premortem and a code review, and their findings are fixed here. ## Checklist - [x] Commits are signed off with `git commit -s` (required by the DCO) - [x] Prose follows [STYLE.md](../STYLE.md) - [x] No secrets, tokens, or internal URLs in the diff ## Security - [ ] This change has no security impact Without this, anyone who can reach the Guardian's port could write a decision into the trace under a live session's ids by sending a malformed envelope with those ids. A reader joining the Guardian's and the host's streams by pair would then see a step that never happened. A well-formed forgery or replay still gets through until PR #229 adds signature, replay and timestamp checks. The README says so. Signed-off-by: rocklambros <rock@rockcyber.com>

…-Project#273 and GenAI-Security-Project#168 Every decision failure is recorded under either posture, with a per-class count at session end (GenAI-Security-Project#273). A receipt a third party can attribute needs ACS-Crypto, because an HMAC binds neither party (GenAI-Security-Project#272). The conformance claim template is published (GenAI-Security-Project#168), and the Guardian step links the reference implementations. Signed-off-by: Bar Kaduri <bar@capsule.security>

Brings in GenAI-Security-Project#309, GenAI-Security-Project#273, GenAI-Security-Project#168 and GenAI-Security-Project#323. GenAI-Security-Project#309 moves a request the Guardian rejects before trusting it off the step spans, and leaves dispatch a `rejection` to report for this PR's checks. This merge wires them in: - dispatch keeps this PR's checks and returns them as Dispatched. A SIGNATURE_INVALID, TIMESTAMP_OUT_OF_WINDOW or REPLAY_DETECTED refusal reports signature_invalid, timestamp_out_of_window or replay_detected, and a payload that fails validation reports envelope_invalid, as it did in GenAI-Security-Project#309. - validateForDispatch checks the envelope's shape only. The method payload is still validated inside dispatch after the signature, so an unsigned request with a malformed payload is refused as unauthenticated, not answered with a trusted deny. - A method with no step span (handshake/hello, system/ping) opens no span, so a rejection dispatch reports for one is recorded as a rejected request when the response is sent. - The "until PR GenAI-Security-Project#229" notes from GenAI-Security-Project#309 are gone, the sample trace is regenerated, and the denied span is copied into the OpenTelemetry page. GenAI-Security-Project#309's "no span for handshake/hello" test sends a schema-valid ClientHello, because an invalid one is now a rejected request. - New end-to-end tests: a replayed request, an unsigned request and a forged hello are each recorded apart, and the genuine request keeps the only span under its ids. Signed-off-by: Jason Crittenden <jrcrittenden@gmail.com>

github-project-automation Bot added this to ACS Project Tracker Oct 13, 2026

github-project-automation Bot moved this to Needs triage in ACS Project Tracker Oct 13, 2026

bar-capsule mentioned this pull request Oct 13, 2026

[Bug] §6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable #37

Closed

acs-board-reconciler Bot moved this from Needs triage to In progress in ACS Project Tracker Oct 14, 2026

rocklambros mentioned this pull request Oct 15, 2026

[RefImpl] Host-side trace follow-ups that wait on PR #229 #285

Open

bar-capsule added 2 commits October 17, 2026 23:20

bar-capsule force-pushed the spec/37-decision-failure-recording branch from 2d48e5d to dffca95 Compare October 17, 2026 20:24

bar-capsule marked this pull request as ready for review October 17, 2026 20:24

bar-capsule requested review from GangGreenTemperTatum, Lrd0036, afogel, almogbhl, artmaro, fewdisc, mamicidal, rocklambros, sclintonowasp and stefanoamorelli as code owners October 17, 2026 20:24

rocklambros previously requested changes Oct 20, 2026

View reviewed changes

rocklambros mentioned this pull request Oct 20, 2026

Record a request the Guardian cannot vouch for apart from the steps, never under its claimed ids #309

Merged

9 of 15 tasks

rocklambros added 2 commits October 22, 2026 09:52

Merge origin/integration into spec/37-decision-failure-recording

5c2cd36

Signed-off-by: rocklambros <rock@rockcyber.com>

rocklambros merged commit 61993cb into GenAI-Security-Project:integration Oct 22, 2026
5 checks passed

github-project-automation Bot moved this from In progress to Done in ACS Project Tracker Oct 22, 2026

bar-capsule mentioned this pull request Oct 28, 2026

Promote integration to main #350

Merged

bar-capsule mentioned this pull request Oct 28, 2026

Add FAQ to docs/topics #20

Merged

3 tasks

rocklambros mentioned this pull request Oct 29, 2026

refimpl: add first-stage Rust AGT Guardian #187

Open

This was referenced Nov 4, 2026

Add adapters/ with Claude Code, Cursor, and NAT reference implementations #22

Merged

Move the example policy to acs.examples and check the import direction #486

Closed

Align the adapters with the current spec #488

Closed

Navigation Menu

Repository navigation

Record every decision failure and count them at session end - #273

Record every decision failure and count them at session end#273
rocklambros merged 4 commits into
GenAI-Security-Project:integrationfrom
bar-capsule:spec/37-decision-failure-recording

bar-capsule commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

rocklambros commented Oct 14, 2026

Uh oh!

bar-capsule commented Oct 17, 2026

Uh oh!

rocklambros left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bar-capsule commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Which issue does this implement

Base branch

Type of change

Specification changes

I tested this

Merging

Checklist

Security

Uh oh!

github-actions Bot commented Oct 13, 2026

Uh oh!

rocklambros commented Oct 14, 2026

Uh oh!

bar-capsule commented Oct 17, 2026

Uh oh!

rocklambros left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bar-capsule commented Oct 13, 2026 •

edited

Loading