Skip to content

Validate streamed tool calls before execution - #73

Merged
zhanghanduo merged 2 commits into
mainfrom
codex/streamed-tool-validation
Oct 7, 2026
Merged

zhanghanduo merged 2 commits into
mainfrom
codex/streamed-tool-validation

Conversation

@zhanghanduo

@zhanghanduo zhanghanduo commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add LoopConfig.stream_transport and call_llm(stream=...) so a complete response can use streaming transport without a delta observer. Existing automatic selection and stream_llm_tokens remain compatible.
  • Assemble native tool calls, check them before execution, and retry an invalid streamed response once using streaming within the original attempt budget. Keep each physical request's usage and attempt record separate.
  • Return an error tool result without invoking the tool when a native call is still invalid after the retry, and repair a named native call that arrived without an id instead of failing the LLM call.

How tool calls are checked

LoopConfig.tool_argument_validation selects the level. It is keyword-only, and the shared text-mode parsers are untouched.

Mode Checks Use
"structural" (default) Arguments decode to a JSON object carrying every top-level required property Default; a blank argument string counts as {}, and property types are left to the tool
"strict" The structural checks plus full JSON Schema Deployments that want typed arguments enforced
"off" Only the legacy empty-required-arguments retry Previously the only behaviour

Property types are deliberately not enforced in the default mode: tools coerce values themselves, and text-mode parameters are JSON-decoded (a 600519 parameter arrives as an int), so a strict type check would reject calls that work today. A tool schema that is not itself valid JSON Schema never fails a call — validation is skipped for that tool with a warning.

At execution time, only provider-native calls are checked. Text-mode calls keep their lenient handling, and an observer's rewrite_args always wins over validation.

Recovery and diagnostics

  • An invalid streamed call triggers at most one streaming retry inside the attempt's remaining time budget. The discarded request keeps its own usage and attempt record.
  • Calls matching the previous empty-required-arguments condition keep the stream_empty_tool_arguments attempt reason (and stream_empty_args_replay when the retry fails), plus the stream_empty_args_* response metadata keys. Other invalid calls use stream_invalid_tool_call. The retry's recovery_action is retry_streaming.
  • A retry that remains invalid returns invalid_tool_calls diagnostics, raw arguments included, and the attempt is accepted_degraded.
  • A named native call without an id gets a generated id before the assistant turn reaches history, so the tool reply always matches its call. Nameless calls keep their existing dropped-call handling.

Compatibility

stream_transport is keyword-only, so positional LoopConfig construction is unaffected, and leaving it unset retains the legacy selection. Existing stream_empty_args_* response metadata remains available for calls matching the prior condition.

Two behaviour changes are intentional and downstream-facing: the retry uses streaming rather than a non-streaming replay, and an invalid call that survives the retry is returned as accepted_degraded with an error tool result rather than executing. Consumers asserting the old non-streaming replay or its event names must update those assertions when they adopt this release.

Verification

  • Full suite: 2306 passed, 2 skipped.
  • Ruff, Pyright and git diff --check pass on the changed runtime code.
  • Regression tests cover both transports for zero-argument tools, structural versus strict validation, text-mode calls, a malformed tool schema, id repair, and observer-visible fields.

@zhanghanduo
zhanghanduo force-pushed the codex/streamed-tool-validation branch 2 times, most recently from 1c0fecb to 79a8c41 Compare October 7, 2026 09:32
@zhanghanduo
zhanghanduo force-pushed the codex/streamed-tool-validation branch from 79a8c41 to 8be5e34 Compare October 7, 2026 09:37
- Add LoopConfig.tool_argument_validation ("structural" default, "strict",
  "off"). Structural checks JSON object shape and top-level required
  properties only, so tool-side type coercion keeps working.
- Treat blank/whitespace arguments as {} so zero-argument tools (Anthropic
  streaming encodes them as "") run without a wasted retry.
- Block only provider-native calls at execution; text-mode calls are never
  rejected by schema validation. Drop the private parser keys that leaked
  to observers and derive diagnostics from response.tool_calls instead.
- Skip validation for tools whose schema is not valid JSON Schema instead
  of raising; cache compiled validators.
- Generate ids for named native calls that lack one instead of failing the
  LLM call; nameless calls keep their dropped-call handling.
- Keep the legacy empty-arguments trigger and its attempt reasons
  (stream_empty_tool_arguments / stream_empty_args_replay) as aliases.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@zhanghanduo
zhanghanduo merged commit 4557ab2 into main Oct 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant