Skip to content

FIX: Render a JSON-string schema as a schema in JsonSchemaNormalizer - #3352

Merged
Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
bIackr0se:fix/json-schema-normalizer-string-schema
Oct 25, 2026
Merged

Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
bIackr0se:fix/json-schema-normalizer-string-schema

Conversation

@bIackr0se

Copy link
Copy Markdown
Contributor

Description

When a target cannot enforce a JSON schema natively, JsonSchemaNormalizer writes the schema into the prompt instead. It renders prompt_metadata["json_schema"] with json.dumps(schema, indent=2). That is right for a dict, but the schema can also be stored as a JSON string: the JSON examples in doc/code/targets/1_openai_chat_target.py and 2_openai_responses_target.py pass json.dumps(person_schema), and JsonResponseConfig.from_metadata parses a string schema for targets that enforce it natively. The normalizer encodes the string a second time, so the model is told to conform to a quoted, escaped string literal and not to a schema.

This PR moves the string handling out of from_metadata into JsonResponseConfig.parse_schema, and the normalizer calls it before rendering. Both paths now read the stored value the same way: a JSON string is parsed, an invalid one raises the same ValueError the native path raises, and an empty string means no schema (the key is stripped and nothing is appended). Non-text pieces are unchanged; their schema key is still only removed.

Real run: the chat-target doc example (same metadata, OpenAIChatTarget with no underlying_model, so the default configuration adapts the schema) sent to a local OpenAI-compatible server that records the request. The user message the server received:

main
The response should conform to the following JSON schema:
"{\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"}, \"age\": {\"type\": \"integer\", \"minimum\": 0, \"maximum\": 150}}, \"required\": [\"name\", \"age\"], \"additionalProperties\": false}"

this PR
The response should conform to the following JSON schema:
{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    ...

With underlying_model="gpt-5" (native schema support) the request is byte-identical on main and on this branch: response_format carries the parsed schema and the prompt is not changed.

Tests and Documentation

  • test_json_schema_normalizer.py: a string schema renders the same text as the dict schema; an invalid string raises ValueError; an empty string strips the key without adding instructions. All three fail on main and pass here.
  • tests/unit/models, tests/unit/prompt_target, tests/unit/message_normalizer and tests/unit/score: 8210 passed, 365 skipped. The existing JsonResponseConfig.from_metadata tests pass unchanged.
  • ruff check, ruff format and ty are clean on the changed files.

@richlundeen

Copy link
Copy Markdown
Contributor

Great find Jaafer Rahmani (@bIackr0se) thank you!

@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 25, 2026
Merged via the queue into microsoft:main with commit 65c1007 Oct 25, 2026
65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants