Skip to content

fix(providers/openai): replay web searches and computer calls without item IDs - #77

Open
ethanndickson wants to merge 1 commit into
mafredri/responses-web-search-unstored-replayfrom
ethan/openai-web-search-replay-record
Open

ethanndickson wants to merge 1 commit into
mafredri/responses-web-search-unstored-replayfrom
ethan/openai-web-search-replay-record

Conversation

@ethanndickson

@ethanndickson ethanndickson commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Relates to CODAGT-1425

Stacked on #78.

#78 sends a hosted web search inline, without its id, whenever it doesn't get an item_reference. This PR takes that one step further: we'll never reference a web search, so every search goes inline. After this, reasoning items are the only replayed output items that carry an ID.

The API pairs any item that has a stored ID with the reasoning item right before it, and rejects the whole request if that reasoning is missing. That's coder/coder#30528. An item without an ID is never paired. I probed this through the AI Gateway:

  • A search sent inline with its stored ws_ id, but without its reasoning item, gets a 400, even with store: false. The same item without an id is accepted with or without reasoning, and with store true or false.
  • The reasoning item the API wants is the last one before the search in the original response, and it has to come before the search in the input.
  • An item_reference and an inline item cost the same input tokens (~35 in my probes), and neither brings back the page contents. Mathias saw the same on fix(providers/openai): replay web searches and computer calls without item IDs #77 (the one exception being gpt-4.1-mini getting the source list). So a reference buys us nothing, whilst it can still resolve another org's stored item after a credential switch.

computer_call has the same problem. A replayed cu_ id without its reasoning item gets Item 'cu_...' of type 'computer_call' was provided without its required 'reasoning' item, with store true or false. We replay computer calls from their raw JSON, so we'll just drop the id from it. A computer_call without an id is accepted.

As a result we can delete the reference path: referenceAt, isResponsesWebSearchToolCall, validateResponsesItemReferences, effectiveResponseStore, and ResponsesReasoningMetadata.SourceStoreEnabled, which only existed to gate references. Rows saved with source_store_enabled still decode; the field is just ignored. The computer use cassettes now expect the calls without an id.

I also ran live two-turn agent.Generate runs through the gateway, with the turn 1 messages round-tripped through JSON: gpt-5-mini with store true then true, false then true, and false then false, and gpt-4.1-mini with store true then true, i.e. a search with no reasoning before it. Every turn 2 sent the searches inline without an id, got a 200, and the model listed the searches it made.

Since searches are never looked up server side any more, coder/coder doesn't need to strip web searches on a credential switch before it bumps fantasy.

@linear-code

linear-code Bot commented Oct 27, 2026

Copy link
Copy Markdown

CODAGT-1425

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T06:38:42.134496Z 688d5fb Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 688d5fbf5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson
ethanndickson marked this pull request as ready for review October 27, 2026 06:36
@mafredri

mafredri commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

@ethanndickson, about the store: false gap your description mentions. I probed replay of hosted web searches live through dogfood's AI Gateway: 249 requests on gpt-6.1-sol, gpt-5-mini and gpt-4.1-mini, covering search, open_page, find_in_page, several searches per turn, and the ws, rs, ws order. Turn 2 asked neutrally what the model did before its answer, and separately asked for a detail that was only on the fetched page.

Turn 2 input store: false store: true
Search left out 200. The model says it made no tool calls, or makes up a different action (0 of every run) same
item_reference 404 "Items are not persisted when store is set to false" 200. The model knows the search (3/3)
Inline web_search_call with id 200. The model knows the action it sent (3/3) 200, but the server swaps in the stored item: same tokens as the reference, and a changed action is ignored
Inline web_search_call without id 200. Same as with id 200, no stored lookup
  • Page content never comes back. Even a store: true reference doesn't carry it: turn 2 tokens stay within about 120 of the "left out" variant, while turn 1 read about 12k tokens of the page. All 36 page-detail questions were refused. The one exception is gpt-4.1-mini with store: true, which gets the search's source list (+190 tokens).
  • So with store: false an inline item is the only way to keep the search record. Client-sent action.sources are ignored. Replaying chat 6a4a3462 from the dogfood DB with its real system prompt, tools and encrypted reasoning: left out, it says it never fetched slashdot (6/6); inline, it lists both opens (3/3).
  • Pairing rule: with store: true, any item with an ID (inline or reference) whose search followed a reasoning item needs that reasoning item directly before it, or it fails with 400 "…provided without its required 'reasoning' item". With store: false there is no such check.
  • A reference to a search with no reasoning before it is accepted with store: true, on reasoning and non-reasoning models. Today's rule drops it: on ws, rs, ws it keeps only the second search, so the model sees a lone click.
  • The double open is open, then click. gpt-6.1-sol's second open_page of the same URL is, in the stored item, a click on links from the first page. The public action reports it as open_page.

Would you add an inline item for searches that can't be referenced with store: false, here or as a follow-up? I can take it on top of your branch otherwise. Tracked as CODAGT-1482. Not tested: streamed turn 2, and whether the gateway changes requests.

🤖 This comment was generated with Coder Agents.

@mafredri

mafredri commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Update: #78 now sends a search inline (no id) whenever its tool call got no item_reference, not only with store: false. Reviewers pointed out that a search from an unstored response is still dropped when a later request is stored: a chat switching model configs, or a zero data retention org, which sends store: true but gets store: false echoed. This PR doesn't reference those, because their source wasn't stored. I probed that case live (turn 1 store: false, turn 2 store: true, gpt-6.1-sol and gpt-5-mini): inline items without id returned 200 and the model recalled each action 12/12, while an item_reference to the same search returned 404.

So the two compose: this PR decides which searches get a reference, and #78 sends whatever is left inline. The rebase is in the same assistant-message loop: #78 records which ids got a reference and skips those in the tool-result branch.

🤖 This comment was generated with Coder Agents.

@ethanndickson
ethanndickson force-pushed the ethan/openai-web-search-replay-record branch from 688d5fb to 9b90448 Compare October 28, 2026 00:49
@ethanndickson ethanndickson changed the title fix(providers/openai): record web search replay dependencies fix(providers/openai): replay web searches and computer calls without item IDs Oct 28, 2026
@ethanndickson
ethanndickson changed the base branch from coder_2_33 to mafredri/responses-web-search-unstored-replay October 28, 2026 00:49
@ethanndickson
ethanndickson added this pull request to stack #79 October 28, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants