Skip to content

Match the Hooks page's payload summaries to the schemas - #498

Open
Lrd0036 wants to merge 3 commits into
GenAI-Security-Project:integrationfrom
Lrd0036:docs/hooks-payload-summaries
Open

Lrd0036 wants to merge 3 commits into
GenAI-Security-Project:integrationfrom
Lrd0036:docs/hooks-payload-summaries

Conversation

@Lrd0036

@Lrd0036 Lrd0036 commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Sixteen of the Hooks page's nineteen payload rows disagreed with their schemas. Each row now names every top-level field and enum value its schema defines, marks the optional ones, and names nothing the schemas don't. A new guard keeps it that way.

Which issue does this implement

Closes #389

Base branch

  • integration, because this touches tests

Type of change

  • Documentation
  • Tooling or CI (documentation guard in tests/)

Changes

docs/spec/instrument/hooks.md. Only the **Payload:** rows change. The fixes, by kind:

  • Required fields left out: toolCallResult's tool, exit_status and outputs; postCompact's two chain hashes; content on userMessage, agentResponse and memoryStore; results on knowledgeRetrieval and memoryContextRetrieval.
  • Enum values left out: turnStart's scheduled and external_event, preCompact's framework_initiated, and turnEnd's interrupted and denied_at_start.
  • Optional fields left out: sessionEnd's final_chain_hash, subagentStart's subagent_descriptor, skillRegister's skill, turnStart's expected_inputs, and the token counts on preCompact, among others.
  • Fields shown as required that the schema leaves optional: agentTrigger's trigger_source, turnEnd's step_count, subagentStart's subagent_intent, and skillRegister's definition.ref and registration_provenance.
  • Names no schema defines: keywords and mime type on knowledgeRetrieval, TTL on memoryStore, and citations and user context on userMessage and agentResponse. preCompact's provenance_summary is now the payload's actual field, pre_compact_provenance_summary.
  • sessionStart listed descriptions rather than fields, so it now names user_identity, policy_mode, intent and platform_context, all optional.

subagentStop, skillLoad and skillUnload already matched and are unchanged.

toolCallRequest is #253's line verbatim. #253 fixes that row and is approved. This PR copies it so the guard passes whichever merges first, and identical lines merge without a conflict in either order.

Normative text is unchanged. postCompact's MUST sentence stays word for word, and the other fields follow it. The new text adds no requirement keyword. Every sentence on the page that contains one matches integration exactly: 25 before, 25 after.

tests/test_hook_payload_summaries.py. For each of the 19 steps/* hooks, the guard reads the schema named on its Schema: line and fails when:

  • the row leaves out a top-level field or a value of a top-level enum, or
  • the row puts a name in backticks that nothing defines: not the hook's schema, a schema it references, the request envelope's metadata, a ContextEntry, or another hook's name.

Sections whose names keep their namespace, such as agbom/snapshot and #496's defer/subscribe, are skipped, and a test adds a defer/subscribe section to the page and checks that the guard skips it. One deliberate exception sits in an allowlist with its reason: skillRegister names model, the AgBOM component type. Mutation cases replay five of the rows #389 reported and check that the guard catches each one. Whether a row marks a field optional is not checked, so the agentTrigger fix above was found by review, not by the guard.

Why

Implementers read this page first. A row that leaves out toolCallResult's required tool, or names a keywords field the schema rejects, sends them to the schema after their first failed message instead of before it.

Specification changes

  • I opened a Discussion before this PR. This changes no schema, hook or event, and no normative text: the rows now describe what the schemas already define. [Docs] hooks.md payload summaries drift from the schemas #389 is an accepted issue a project lead filed.
  • No schema changes. The schemas are the source these rows now follow.
  • Impact on downstream implementers: none on the wire. The page now describes what the schemas already require.

Breaking for implementers? No.

I tested this

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

The page is descriptive. The schemas and the normative text are unchanged.

Implementers read the Hooks page first, and fifteen of its nineteen
payload rows had drifted from the schemas: required fields and enum
values left out, optional fields shown as required, and fields no schema
defines (keywords, TTL, citations). Each row now names every top-level
field and enum value its schema defines, marks the optional ones, and
names nothing the schemas do not.

The toolCallRequest row is GenAI-Security-Project#253's line verbatim, so the two pull
requests merge cleanly in either order.

A new guard reads the schemas and fails when a row leaves out a
top-level field or enum value, or names something undefined. Its
mutation cases replay five of the rows GenAI-Security-Project#389 reported.

Closes GenAI-Security-Project#389

Signed-off-by: Lrd0036 <lrd0036@auburn.edu>
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

This pull request's description says it closes #389, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging.

@Lrd0036

Lrd0036 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Confirmed: all 19 payload rows now match their schemas, which is what #389 asks for, and tests/test_hook_payload_summaries.py keeps them matched. The toolCallRequest row is #253's line verbatim, so either can merge first.

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head d3966f3b5978e8163f165e6d1bb079214e7f49b8. Approving. Please read note 1: it concerns a PR that has not merged yet, and it is cheap to fix here.

What I checked, independently of the guard. The guard is permissive in one direction (it checks that names are present and defined, not whether a row marks a field optional), so I did not rely on it. I dumped every specification/v0.1.0/hooks/*.json with its required set and its nested properties and compared all 19 payload rows by hand:

  • Top-level fields: every row names all of them, and none names a field that is not there. The keywords, the TTL and "agent reasoning" that the old rows invented are gone.
  • Required versus optional: sessionStart ("every field is optional", and the schema has no required), skillLoad, skillUnload, subagentStop, subagentStart, skillRegister, turnStart, turnEnd, preCompact, postCompact, sessionEnd, toolCallRequest, toolCallResult, knowledgeRetrieval, memoryContextRetrieval and memoryStore all mark exactly the fields the schema leaves out of required as optional. agentTrigger is the one exception, in note 3.
  • Nested objects and enums: userMessage and agentResponse parts (type text/image/audio/file, required value), knowledgeRetrieval (query.value, results[].value, source), memoryContextRetrieval and memoryStore (memory_store.scope is session/user/tenant/global in both schemas), skillRegister (definition.digest required, ref and body optional), and postCompact (summary with required value and provenance, the two chain hashes required, lineage_depth_after optional).
  • expected_inputs ("a hint listing the hooks the framework expects this turn to fire") and final_chain_hash ("the agent's own final chain hash, for the Guardian to check against its chain") match the schemas' own descriptions.
  • The toolCallRequest row is byte-for-byte the line #253 adds. Both orders merge cleanly (git merge-tree against f033a63, and against #247, #271 and #496), so the "either can merge first" claim holds.

The only disagreement I found is one unmarked optional field on a row this PR left as it was (note 3).

Validation at this head (Windows host)

  • uv run pytest -q tests/test_hook_payload_summaries.py: 45 passed.
  • Full uv run pytest -q: 1230 passed, 1 skipped, 10 failed, 20 errors. The 30 are in test_closing_choice.py, test_fetch_roadmap.py, test_overrides.py, test_pr_intake_workflow.py and test_publish_schemas.py, which fail identically on integration at bba1b84 and are not touched here.
  • uv run --no-dev mkdocs build --strict -d _site/docs: exit 0.
  • Mutation checks on the guard: removing scheduled from the turnStart row makes one test fail, and adding a keywords field to the knowledgeRetrieval row makes one fail. Both are the drift the guard exists to catch.

Notes

  1. The guard breaks when #496 merges. #496 adds a ## defer/subscribe section with a Schema: link and a **Payload:** row. This test collects every such section into ROWS, then asserts len(ROWS) == 19, and checks each row's names against its schema. I merged #496 (034f52a) into this branch locally. The merge is textually clean, and two tests fail: test_every_steps_hook_has_a_payload_row (20 rows, not 19) and test_payload_row_names_only_what_the_schemas_define[defer/subscribe] (request_id is not defined by defer-subscribe.json). Whichever PR lands second goes red, and it is the other author who has to edit this guard. The least effort is to build ROWS only from sections without a / in the name (every steps/* hook is a bare name, while agbom/…, system/ping and defer/subscribe all have one), which also matches what the assertion's message says it counts. If you would rather check defer/subscribe as well, add request_id to ALSO_NAMED["defer/subscribe"] with its reason, and change 19 to the count of steps/* hooks.
  2. The rows say provenance is optional, which is what the base schemas say. The *.acs-provenance.json variants make it required under provenance_producer: deterministic, and hooks.md never mentions that. A single sentence above the first payload row, or a pointer to the profile, would stop an implementer of that profile from reading "optional" literally. It is not in scope for #389, so a follow-up is fine.
  3. The guard does not check optional markers, as its docstring says, and that is where one row slips through. The agentTrigger row, which this PR did not change, lists trigger_source as if it were required. agent-trigger.json requires only trigger_type, so it should read "optional trigger_source". It is a two-word fix on a row you are already rewriting the neighbours of. If you want the guard to catch this class, the schema's required list is all it needs to compare against the words "optional" and "optionally" in the row.

Note 1 is the only thing I would change before this merges, and note 3 is cheap to include. Every other row matches its schema. Approving.

The payload guard read every section with a schema link and a payload
row, so GenAI-Security-Project#496's defer/subscribe section would make two of its tests fail:
it counts 20 rows, and request_id is not in defer-subscribe.json. The
guard now reads only the sections with bare names, which are the 19
steps/* hooks, and a new test adds a defer/subscribe section to the page
and checks that it is skipped.

The agentTrigger row listed trigger_source as required.
agent-trigger.json requires only trigger_type, so the row now marks
trigger_source optional.

Both come from drk-ram's review of GenAI-Security-Project#498.

Signed-off-by: Lrd0036 <lrd0036@auburn.edu>
@Lrd0036

Lrd0036 commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @drk-ram. 86724e4 takes notes 1 and 3. The guard now reads only the 19 steps/* sections, and a new test adds a defer/subscribe section to the page and checks that it's skipped. With #496 merged in locally, both test files pass. The agentTrigger row now marks trigger_source optional. Note 2 is a fair point. I'd rather take it in a follow-up than widen this PR.

@drk-ram drk-ram left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified at head 86724e41259239dd3fb3f7fe053b701ecfea7ae2. Approving.

Delta since d3966f3 (2 files).

  • tests/test_hook_payload_summaries.py: hook_sections now takes the page text, and the new steps_hooks filter keeps only sections with bare names (no /). ROWS, the "every steps hook has a payload row" check and the == 19 count all run over that set. This replaces the old startswith(("agbom/", "system/")) exclusion, which would have counted defer/subscribe as a missing row the moment #496 landed. The new test_a_namespaced_section_with_a_payload_row_is_not_counted appends a ## defer/subscribe section carrying a schema link and a payload row and asserts the steps/* key set is unchanged. That is the failure mode the guard has to survive, and the test exercises it directly.
  • docs/spec/instrument/hooks.md: the agentTrigger row now reads "and optionally trigger_source … and intent", which matches hooks/agent-trigger.json, where only trigger_type is required.

Verification (detached worktree).

  • 86724e4 alone: pytest tests/test_hook_payload_summaries.py gives 46 passed.
  • 86724e4 with #496 (034f52a) merged in: the merge is clean, and the page then has ## defer/subscribe alongside agbom/snapshot and agbom/changed. pytest tests/test_hook_payload_summaries.py tests/test_defer_subscribe.py gives 64 passed.

So the guard holds in both merge orders, and my earlier notes 1 and 3 are resolved. Nothing further from me.

@acs-board-reconciler acs-board-reconciler Bot moved this from Needs triage to In progress in ACS Project Tracker Nov 5, 2026
The steps/* hook set now comes from the schemas' own titles, through
test_hook_taxonomy's native_hook_methods(), instead of a count of 19. A
section whose Schema line is missing or mistyped now fails the check
instead of dropping out.

resolve() follows a $ref chain from each file's root, including refs into
another file, so a top-level enum defined elsewhere is still required in
the row. vocabulary() takes only the part a fragment $ref names. A payload
row may wrap to the next blank line, and a namespaced hook name in
backticks, such as `agbom/changed`, is no longer flagged as undefined.

The subagentStop, skillLoad and skillUnload rows now write "and optionally"
like the other sixteen.

Signed-off-by: Lrd0036 <lrd0036@auburn.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

[Docs] hooks.md payload summaries drift from the schemas

2 participants