Repository navigation
Conversation
Implementers read the Hooks page first, and fifteen of its nineteen payload rows had drifted from the schemas: required fields and enum values left out, optional fields shown as required, and fields no schema defines (keywords, TTL, citations). Each row now names every top-level field and enum value its schema defines, marks the optional ones, and names nothing the schemas do not. The toolCallRequest row is GenAI-Security-Project#253's line verbatim, so the two pull requests merge cleanly in either order. A new guard reads the schemas and fails when a row leaves out a top-level field or enum value, or names something undefined. Its mutation cases replay five of the rows GenAI-Security-Project#389 reported. Closes GenAI-Security-Project#389 Signed-off-by: Lrd0036 <lrd0036@auburn.edu>
|
This pull request's description says it closes #389, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging. |
drk-ram
left a comment
There was a problem hiding this comment.
Reviewed at head d3966f3b5978e8163f165e6d1bb079214e7f49b8. Approving. Please read note 1: it concerns a PR that has not merged yet, and it is cheap to fix here.
What I checked, independently of the guard. The guard is permissive in one direction (it checks that names are present and defined, not whether a row marks a field optional), so I did not rely on it. I dumped every specification/v0.1.0/hooks/*.json with its required set and its nested properties and compared all 19 payload rows by hand:
- Top-level fields: every row names all of them, and none names a field that is not there. The
keywords, theTTLand "agent reasoning" that the old rows invented are gone. - Required versus optional:
sessionStart("every field is optional", and the schema has norequired),skillLoad,skillUnload,subagentStop,subagentStart,skillRegister,turnStart,turnEnd,preCompact,postCompact,sessionEnd,toolCallRequest,toolCallResult,knowledgeRetrieval,memoryContextRetrievalandmemoryStoreall mark exactly the fields the schema leaves out ofrequiredas optional.agentTriggeris the one exception, in note 3. - Nested objects and enums:
userMessageandagentResponseparts (typetext/image/audio/file, requiredvalue),knowledgeRetrieval(query.value,results[].value,source),memoryContextRetrievalandmemoryStore(memory_store.scopeissession/user/tenant/globalin both schemas),skillRegister(definition.digestrequired,refandbodyoptional), andpostCompact(summarywith requiredvalueandprovenance, the two chain hashes required,lineage_depth_afteroptional). expected_inputs("a hint listing the hooks the framework expects this turn to fire") andfinal_chain_hash("the agent's own final chain hash, for the Guardian to check against its chain") match the schemas' own descriptions.- The
toolCallRequestrow is byte-for-byte the line #253 adds. Both orders merge cleanly (git merge-treeagainstf033a63, and against #247, #271 and #496), so the "either can merge first" claim holds.
The only disagreement I found is one unmarked optional field on a row this PR left as it was (note 3).
Validation at this head (Windows host)
uv run pytest -q tests/test_hook_payload_summaries.py: 45 passed.- Full
uv run pytest -q: 1230 passed, 1 skipped, 10 failed, 20 errors. The 30 are intest_closing_choice.py,test_fetch_roadmap.py,test_overrides.py,test_pr_intake_workflow.pyandtest_publish_schemas.py, which fail identically onintegrationatbba1b84and are not touched here. uv run --no-dev mkdocs build --strict -d _site/docs: exit 0.- Mutation checks on the guard: removing
scheduledfrom theturnStartrow makes one test fail, and adding akeywordsfield to theknowledgeRetrievalrow makes one fail. Both are the drift the guard exists to catch.
Notes
- The guard breaks when #496 merges. #496 adds a
## defer/subscribesection with aSchema:link and a**Payload:**row. This test collects every such section intoROWS, then assertslen(ROWS) == 19, and checks each row's names against its schema. I merged #496 (034f52a) into this branch locally. The merge is textually clean, and two tests fail:test_every_steps_hook_has_a_payload_row(20 rows, not 19) andtest_payload_row_names_only_what_the_schemas_define[defer/subscribe](request_idis not defined bydefer-subscribe.json). Whichever PR lands second goes red, and it is the other author who has to edit this guard. The least effort is to buildROWSonly from sections without a/in the name (everysteps/*hook is a bare name, whileagbom/…,system/pinganddefer/subscribeall have one), which also matches what the assertion's message says it counts. If you would rather checkdefer/subscribeas well, addrequest_idtoALSO_NAMED["defer/subscribe"]with its reason, and change 19 to the count ofsteps/*hooks. - The rows say
provenanceis optional, which is what the base schemas say. The*.acs-provenance.jsonvariants make it required underprovenance_producer: deterministic, andhooks.mdnever mentions that. A single sentence above the first payload row, or a pointer to the profile, would stop an implementer of that profile from reading "optional" literally. It is not in scope for #389, so a follow-up is fine. - The guard does not check optional markers, as its docstring says, and that is where one row slips through. The
agentTriggerrow, which this PR did not change, liststrigger_sourceas if it were required.agent-trigger.jsonrequires onlytrigger_type, so it should read "optionaltrigger_source". It is a two-word fix on a row you are already rewriting the neighbours of. If you want the guard to catch this class, the schema'srequiredlist is all it needs to compare against the words "optional" and "optionally" in the row.
Note 1 is the only thing I would change before this merges, and note 3 is cheap to include. Every other row matches its schema. Approving.
The payload guard read every section with a schema link and a payload row, so GenAI-Security-Project#496's defer/subscribe section would make two of its tests fail: it counts 20 rows, and request_id is not in defer-subscribe.json. The guard now reads only the sections with bare names, which are the 19 steps/* hooks, and a new test adds a defer/subscribe section to the page and checks that it is skipped. The agentTrigger row listed trigger_source as required. agent-trigger.json requires only trigger_type, so the row now marks trigger_source optional. Both come from drk-ram's review of GenAI-Security-Project#498. Signed-off-by: Lrd0036 <lrd0036@auburn.edu>
|
Thanks @drk-ram. |
drk-ram
left a comment
There was a problem hiding this comment.
Re-verified at head 86724e41259239dd3fb3f7fe053b701ecfea7ae2. Approving.
Delta since d3966f3 (2 files).
tests/test_hook_payload_summaries.py:hook_sectionsnow takes the page text, and the newsteps_hooksfilter keeps only sections with bare names (no/).ROWS, the "every steps hook has a payload row" check and the== 19count all run over that set. This replaces the oldstartswith(("agbom/", "system/"))exclusion, which would have counteddefer/subscribeas a missing row the moment #496 landed. The newtest_a_namespaced_section_with_a_payload_row_is_not_countedappends a## defer/subscribesection carrying a schema link and a payload row and asserts thesteps/*key set is unchanged. That is the failure mode the guard has to survive, and the test exercises it directly.docs/spec/instrument/hooks.md: theagentTriggerrow now reads "and optionallytrigger_source… andintent", which matcheshooks/agent-trigger.json, where onlytrigger_typeis required.
Verification (detached worktree).
86724e4alone:pytest tests/test_hook_payload_summaries.pygives 46 passed.86724e4with #496 (034f52a) merged in: the merge is clean, and the page then has## defer/subscribealongsideagbom/snapshotandagbom/changed.pytest tests/test_hook_payload_summaries.py tests/test_defer_subscribe.pygives 64 passed.
So the guard holds in both merge orders, and my earlier notes 1 and 3 are resolved. Nothing further from me.
The steps/* hook set now comes from the schemas' own titles, through test_hook_taxonomy's native_hook_methods(), instead of a count of 19. A section whose Schema line is missing or mistyped now fails the check instead of dropping out. resolve() follows a $ref chain from each file's root, including refs into another file, so a top-level enum defined elsewhere is still required in the row. vocabulary() takes only the part a fragment $ref names. A payload row may wrap to the next blank line, and a namespaced hook name in backticks, such as `agbom/changed`, is no longer flagged as undefined. The subagentStop, skillLoad and skillUnload rows now write "and optionally" like the other sixteen. Signed-off-by: Lrd0036 <lrd0036@auburn.edu>
What changed
Sixteen of the Hooks page's nineteen payload rows disagreed with their schemas. Each row now names every top-level field and enum value its schema defines, marks the optional ones, and names nothing the schemas don't. A new guard keeps it that way.
Which issue does this implement
Closes #389
Base branch
integration, because this touches testsType of change
tests/)Changes
docs/spec/instrument/hooks.md. Only the**Payload:**rows change. The fixes, by kind:toolCallResult'stool,exit_statusandoutputs;postCompact's two chain hashes;contentonuserMessage,agentResponseandmemoryStore;resultsonknowledgeRetrievalandmemoryContextRetrieval.turnStart'sscheduledandexternal_event,preCompact'sframework_initiated, andturnEnd'sinterruptedanddenied_at_start.sessionEnd'sfinal_chain_hash,subagentStart'ssubagent_descriptor,skillRegister'sskill,turnStart'sexpected_inputs, and the token counts onpreCompact, among others.agentTrigger'strigger_source,turnEnd'sstep_count,subagentStart'ssubagent_intent, andskillRegister'sdefinition.refandregistration_provenance.keywordsand mime type onknowledgeRetrieval, TTL onmemoryStore, and citations and user context onuserMessageandagentResponse.preCompact'sprovenance_summaryis now the payload's actual field,pre_compact_provenance_summary.sessionStartlisted descriptions rather than fields, so it now namesuser_identity,policy_mode,intentandplatform_context, all optional.subagentStop,skillLoadandskillUnloadalready matched and are unchanged.toolCallRequestis #253's line verbatim. #253 fixes that row and is approved. This PR copies it so the guard passes whichever merges first, and identical lines merge without a conflict in either order.Normative text is unchanged.
postCompact's MUST sentence stays word for word, and the other fields follow it. The new text adds no requirement keyword. Every sentence on the page that contains one matchesintegrationexactly: 25 before, 25 after.tests/test_hook_payload_summaries.py. For each of the 19steps/*hooks, the guard reads the schema named on itsSchema:line and fails when:Sections whose names keep their namespace, such as
agbom/snapshotand #496'sdefer/subscribe, are skipped, and a test adds adefer/subscribesection to the page and checks that the guard skips it. One deliberate exception sits in an allowlist with its reason:skillRegisternamesmodel, the AgBOM component type. Mutation cases replay five of the rows #389 reported and check that the guard catches each one. Whether a row marks a field optional is not checked, so theagentTriggerfix above was found by review, not by the guard.Why
Implementers read this page first. A row that leaves out
toolCallResult's requiredtool, or names akeywordsfield the schema rejects, sends them to the schema after their first failed message instead of before it.Specification changes
hooks.mdpayload summaries drift from the schemas #389 is an accepted issue a project lead filed.Breaking for implementers? No.
I tested this
integrationcommit,bba1b84(checked 2026-11-04).uv run --locked pytest -q: 1251 passed, 1 skipped, including the 46 new tests.uv run --locked mkdocs build --strictpasses, and the new#usermessagelink resolves in the built page.git diff --checkpasses.hooks.mdfromintegration, the guard flags fifteen drifted rows, including every row [Docs]hooks.mdpayload summaries drift from the schemas #389 lists. It passesagentTrigger, whose only gap was the optional marker, and the three that already matched.034f52a) locally: no conflict, and this guard and Define defer/subscribe for DEFER resolution #496'stests/test_defer_subscribe.pypass together (64 tests).Checklist
git commit -s(required by the DCO)Security
The page is descriptive. The schemas and the normative text are unchanged.