feat(openai_sdk): add OAI-027, OAI-028 TypeScript description quality rules - #92
feat(openai_sdk): add OAI-027, OAI-028 TypeScript description quality rules#92bradAGI wants to merge 1 commit into
Conversation
… rules The TypeScript counterpart to OAI-025/026. OAI-022 only checks that a description exists, so a tool described "TODO: describe this tool." or "Gets data." passes today while leaving the tool in exactly the state OAI-022 exists to prevent. The Zod parameters schema does not compensate: it constrains the shape of the arguments once the model has decided to call this tool, never whether that decision was right, so a placeholder yields a well-formed call to the wrong tool. With handoffs the reach is wider — an agent picks between its own tools and its peers' from these strings, so the stub can route the conversation to the wrong agent rather than merely the wrong function.
|
Thanks @bradAGI. Checked this against The engine half in trustabl/trustabl#170 mirrors this file byte for byte and carries fire and silent cases for both rules, so the pair is complete and neither should land alone. One note rather than a request: the needles are plain substrings, so a genuine tool described as adding a New rules also need a rationale doc in |
The TypeScript counterpart to OAI-025/026 (#87). OAI-022 only checks that a description exists, so a tool described
"TODO: describe this tool."or"Gets data."passes today — while sitting in exactly the state OAI-022 exists to prevent. OAI-022's own text calls the description "the model's primary routing signal alongside the tool name," which is precisely why a placeholder in that slot matters.Two points, both aimed at mitigations people assume already cover this:
parametersschema doesn't compensate. It constrains the shape of the arguments once the model has decided to call this tool, never whether that decision was right. A placeholder description yields a well-formed call to the wrong tool — validation passes, the run is still wrong.OAI-028 adds the cost that's hardest to trace: each mis-selection spends a turn against the agent's max-turns budget, and nothing in a trace attributes the wasted turn to the description that caused it — the run just looks slow or confused.
Verification — engine built at
main:Fire (one
tool({...})described"TODO: describe this tool.", one"Gets data."):OAI-027, OAI-028, OAI-202Silent (full description naming when to prefer the neighboring tool):
OAI-202(OAI-202 is the pre-existing missing-CLAUDE.md repo rule.)
OAI-028 pairs
description_length_lt: 40withhas_docstring: trueso it doesn't double-report against OAI-022 on an empty description, same as CSDK-018.No new predicates, so no
schema_versionbump.Merge note: this and #87 both append to
openai_sdk/tool_definition.yaml, so whichever lands second needs a trivial rebase — the rule blocks are independent and the resolution is "keep both." Happy to rebase on request.