[jpr5]

Closes #547
Closes #548

#550 added citations on the OpenAI Responses API and generated files on Gemini. This PR covers the remaining providers, adds file citations on Responses, and makes malformed citations and files fail loudly instead of being served or dropped silently. It reuses the same fixture fields (citations, files) and the same defaults: the title falls back to the URL, and URL citation offsets default to the whole text.

What each provider does now

citations (#547)

• Chat Completions: message.annotations gets {type: "url_citation", url_citation: {url, title, start_index, end_index}}, the shape of ChatCompletionMessage.Annotation in the openai SDK. When streaming, one chunk with delta.annotations is sent after the content and before the finish chunk. The SDK's ChatCompletionStream copies it into message.annotations.
• Anthropic Messages: the text block gets citations: [{type: "web_search_result_location", cited_text, url, title, encrypted_index}] (CitationsWebSearchResultLocation in @anthropic-ai/sdk). The real encrypted_index is opaque, so aimock fills in a stable base64 placeholder. When streaming, one citations_delta per citation comes after the text deltas and before content_block_stop, in the same order as the streaming example in the Anthropic citations docs. On a blocks turn with several text blocks, each citation goes to the text block that holds its start.
• Gemini (including Vertex AI): the candidate gets groundingMetadata with one groundingChunks[].web {uri, title} per citation and groundingSupports[] {segment {partIndex, startIndex, endIndex, text}, groundingChunkIndices}. Segment offsets are UTF-8 byte offsets into the text of the part named by partIndex, as the Segment docs in @google/genai require. A citation that spans several text parts becomes one support per part. When streaming, each chunk that holds cited text carries the supports for its own text.
• OpenAI Responses: url_citation annotations as in feat: scriptable citations (Responses) and model-generated files (Gemini) #550, now also split correctly across several text blocks. A citation with type: "file" (fileId, filename, optional index, which defaults to the end of the text) becomes a file_citation annotation {type, file_id, filename, index} on output_text, streamed through response.output_text.annotation.added. Chat Completions, Anthropic and Gemini have no wire shape for a file citation, so they skip it.

files (#548)

• OpenAI Responses: each file becomes an image_generation_call output item {type, id, status, result}, with result holding the base64 bytes. The items come after the reasoning and web-search items and before the message and tool-call items, on text, tool-only, content+toolCalls and blocks turns. When streaming, each item gets output_item.added (result: null), then response.image_generation_call.in_progress, .generating and .completed, then output_item.done. Only image/* files can be carried this way; any other MIME type answers a 500 with aimock_unsupported_generated_file.
• Gemini: files are served as inlineData parts, as feat: scriptable citations (Responses) and model-generated files (Gemini) #550 added.
• Chat Completions and Anthropic Messages: these wires have no generated-file shape, so a fixture that sets files is rejected before any byte is written. The response is a 500 with code: "aimock_unsupported_generated_file" in the provider's error envelope. It uses the same journaling and error path as UnsupportedToolCallError for customToolCalls. Before this PR the files were dropped silently with a 200.

Validation and error codes

Fixture citations and files are checked on every request that serves them on Chat Completions, Anthropic Messages, Gemini and the OpenAI Responses API. A malformed value answers a 500 before any content is written:

• aimock_invalid_fixture_citation: citations is not an array, an entry is not an object, url (or fileId / filename for a file citation) is missing or not a non-empty string, an offset is not an integer, offsets fall outside the text (0 <= startIndex <= endIndex <= text length, 0 <= index <= text length), an offset splits a UTF-16 surrogate pair, an entry mixes URL and file citation fields, or the turn has no text to cite (tool calls only, blocks with no text block, or empty content).
• aimock_invalid_fixture_file: files is not an array, or an entry lacks a non-empty string mimeType or a string data.
• aimock_unsupported_generated_file: any file on Chat Completions or Anthropic Messages, or a non-image/* file on the OpenAI Responses API.

Each message names the field, says what was wrong, and says how to fix it. A null value or null optional field is treated as absent, so JSON fixtures that write null for an unset key keep working. The new InvalidFixtureOutputError (behind the two aimock_invalid_fixture_* codes) and UnsupportedGeneratedFileError are exported from @copilotkit/aimock. Other wires (Bedrock, Ollama, Cohere, Gemini Interactions, Gemini Live, Realtime) do not read these keys and ignore them, as before.

Behavior under misbehavior faults

The fixture is checked on its original turn before any fault applies, so the result is the same with or without --misbehavior. Only an invalid misbehavior config answers its own 400 or 501 first.

• A fixture the wire rejects (malformed citations or files, generated files on Chat Completions or Anthropic, a non-image/* file on Responses) is never faulted. It takes the normal path, is journaled as a 500 that keeps its fixture, and consumes no fault times budget or seed ordinal. aimock_misbehavior_total does not count it. This holds on HTTP and on the Responses WebSocket.
• A valid fixture under a fault that changes the reply text (refusal, empty response, content filter, reasoning only) is served without its citations, so no citation points into text the client never receives. A fault that keeps the text keeps the citations.
• A non-stream Chat Completions fault is journaled before its timing wait, and a client that hangs up during the timing wait of any non-stream Chat Completions reply is journaled as interrupted, not as served.

Load warnings

Fixture load validation reports malformed citations and files as warnings, with the same problem text the 500 carries, plus which 500 the requests will answer. The fixture still loads, as in the release that added the keys. For example:

[aimock] Fixture 1: citations[0] offsets must satisfy 0 <= startIndex <= endIndex <= text length (6), got startIndex 0, endIndex 99; fix the offsets, or omit them to cite the whole text — requests it serves on Chat Completions, Anthropic Messages, Gemini or the OpenAI Responses API answer a 500 (aimock_invalid_fixture_citation)

Compatibility

• The change is additive. New public API: ChatCompletionAnnotation, annotations? on ChatCompletionMessage and SSEDelta, FixtureUrlCitation, FixtureFileCitation, InvalidFixtureOutputError and UnsupportedGeneratedFileError. The API reports for every entry point are updated because they carry the citation types transitively.
• FixtureCitation changed from an interface to the union FixtureUrlCitation | FixtureFileCitation. It has never been released: feat: scriptable citations (Responses) and model-generated files (Gemini) #550 merged after 1.46.0, and the type is not in the v1.46.0 tag. FixtureUrlCitation has the same fields as the old interface, plus an optional type: "url".
• Fixtures without citations or files produce the same responses as before. I compared 16 responses (Chat Completions, Responses, Anthropic and Gemini, streaming and non-streaming, text and content+toolCalls fixtures) between base 508dd9b9 and this branch. After normalizing ids and timestamps they are byte-identical. The tests also assert that no annotations key, citations key, groundingMetadata or image_generation_call appears when the field is absent.
• Fixtures with malformed citations or files that used to be served with a 200 now answer a 500 on the four wires above. That is the intended fail-loud behavior, and the load warning names each such fixture.

Local red-green proof (real server, real wire)

I started the CLI server (npx tsx src/cli.ts -f fixtures.json) from base 508dd9b9 (RED) and from this branch (GREEN), and sent the same requests to each.

Citations and files

RED (base): no citations on Chat Completions, Anthropic or Gemini, no image_generation_call, and files ignored with a 200.

===== chat non-stream (cite)
{"id":"chatcmpl-ABR70VXJpPbtwn9o","object":"chat.completion","created":1792794119,"model":"gpt-4o","choices":[{"index":0,"message":{"role":"assistant","content":"Paris is the capital of France.","refusal":null},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":8,"total_tokens":9}}
HTTP 200
===== anthropic non-stream (cite)
{"id":"msg_2Oqvf9l5-9Vmv84Q","type":"message","role":"assistant","content":[{"type":"text","text":"Paris is the capital of France."}],"model":"claude-sonnet-4-5","stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0}}
HTTP 200
===== gemini non-stream (cite)
{"candidates":[{"content":{"role":"model","parts":[{"text":"Paris is the capital of France."}]},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":0,"candidatesTokenCount":0,"totalTokenCount":0}}
HTTP 200
===== responses non-stream (file)
{"id":"resp-0iBUzUlYjZNai8Ey","object":"response","created_at":1792794119,"model":"gpt-4o","status":"completed","output":[{"type":"message","id":"msg-7D1aKAwUVCFjfhYO","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Here is your chart.","annotations":[]}]}],"usage":{"input_tokens":0,"output_tokens":0,"total_tokens":0}}
HTTP 200
===== chat non-stream (file)
{"id":"chatcmpl-xGemf9eHOvlINMFB","object":"chat.completion","created":1792794119,"model":"gpt-4o","choices":[{"index":0,"message":{"role":"assistant","content":"Here is your chart.","refusal":null},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":5,"total_tokens":6}}
HTTP 200
===== anthropic non-stream (file)
{"id":"msg_AZdaedU5PiA5IeHJ","type":"message","role":"assistant","content":[{"type":"text","text":"Here is your chart."}],"model":"claude-sonnet-4-5","stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0}}
HTTP 200

GREEN (this branch), same requests:

===== chat non-stream (cite)
{"id":"chatcmpl-WXkfcPuLdYFBJub6","object":"chat.completion","created":1792794122,"model":"gpt-4o","choices":[{"index":0,"message":{"role":"assistant","content":"Paris is the capital of France.","refusal":null,"annotations":[{"type":"url_citation","url_citation":{"url":"https://example.com/paris","title":"Paris","start_index":0,"end_index":5}},{"type":"url_citation","url_citation":{"url":"https://example.com/france","title":"https://example.com/france","start_index":0,"end_index":31}}]},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":8,"total_tokens":9}}
HTTP 200
===== anthropic non-stream (cite)
{"id":"msg_glKfFbKTh0KO7CGA","type":"message","role":"assistant","content":[{"type":"text","text":"Paris is the capital of France.","citations":[{"type":"web_search_result_location","cited_text":"Paris","url":"https://example.com/paris","title":"Paris","encrypted_index":"YWltb2NrLWNpdGF0aW9uOjA6aHR0cHM6Ly9leGFtcGxlLmNvbS9wYXJpcw=="},{"type":"web_search_result_location","cited_text":"Paris is the capital of France.","url":"https://example.com/france","title":"https://example.com/france","encrypted_index":"YWltb2NrLWNpdGF0aW9uOjE6aHR0cHM6Ly9leGFtcGxlLmNvbS9mcmFuY2U="}]}],"model":"claude-sonnet-4-5","stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":0,"output_tokens":0}}
HTTP 200
===== gemini non-stream (cite)
{"candidates":[{"content":{"role":"model","parts":[{"text":"Paris is the capital of France."}]},"finishReason":"STOP","groundingMetadata":{"groundingChunks":[{"web":{"uri":"https://example.com/paris","title":"Paris"}},{"web":{"uri":"https://example.com/france","title":"https://example.com/france"}}],"groundingSupports":[{"segment":{"partIndex":0,"startIndex":0,"endIndex":5,"text":"Paris"},"groundingChunkIndices":[0]},{"segment":{"partIndex":0,"startIndex":0,"endIndex":31,"text":"Paris is the capital of France."},"groundingChunkIndices":[1]}]},"index":0}],"usageMetadata":{"promptTokenCount":0,"candidatesTokenCount":0,"totalTokenCount":0}}
HTTP 200
===== responses non-stream (file)
{"id":"resp-SdNRLMU3Ar8AIbNP","object":"response","created_at":1792794122,"model":"gpt-4o","status":"completed","output":[{"type":"image_generation_call","id":"ig-0TcEztbhS3UKMcgT","status":"completed","result":"iVBORw0KGgo="},{"type":"message","id":"msg-yQ0AN_JYl54A_5fW","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Here is your chart.","annotations":[]}]}],"usage":{"input_tokens":0,"output_tokens":0,"total_tokens":0}}
HTTP 200
===== chat non-stream (file)
{"error":{"message":"aimock: fixture \"files\" (model-generated files) are only supported on Gemini and the OpenAI Responses API; this request arrived on OpenAI Chat Completions","type":"server_error","code":"aimock_unsupported_generated_file"}}
HTTP 500
===== anthropic non-stream (file)
{"type":"error","error":{"type":"api_error","code":"aimock_unsupported_generated_file","message":"aimock: fixture \"files\" (model-generated files) are only supported on Gemini and the OpenAI Responses API; this request arrived on Anthropic Messages"}}
HTTP 500

File citations, validation and faults

RED (base): a file citation is served as a broken url_citation with no URL, malformed citations and files are served with a 200, a PDF on Responses is dropped silently, and a malformed fixture under a refusal fault is faulted.

===== responses non-stream (file citation)
HTTP 200
... "content":[{"type":"output_text","text":"See the attached report.","annotations":[{"type":"url_citation","start_index":0,"end_index":24}]}] ...
===== chat non-stream (out-of-range citation offsets)
HTTP 200
... "message":{"role":"assistant","content":"Short.","refusal":null} ...
===== anthropic non-stream (file missing data)
HTTP 200
... "content":[{"type":"text","text":"Chart."}] ...
===== responses non-stream (non-image file)
HTTP 200
... "output":[{"type":"message", ... "text":"Doc.","annotations":[]}]}] ...
===== chat non-stream (malformed citation under refusal fault)
HTTP 200
... "message":{"role":"assistant","content":null,"refusal":"I can't help with that."} ...

GREEN (this branch), same requests:

===== responses non-stream (file citation)
HTTP 200
... "content":[{"type":"output_text","text":"See the attached report.","annotations":[{"type":"file_citation","file_id":"file-abc123","filename":"report.pdf","index":24}]}] ...
===== responses stream (file citation)
event: response.output_text.annotation.added
data: {"type":"response.output_text.annotation.added", ... "annotation_index":0,"annotation":{"type":"file_citation","file_id":"file-abc123","filename":"report.pdf","index":24}}
===== chat non-stream (out-of-range citation offsets)
HTTP 500
{"error":{"message":"aimock: invalid fixture \"citations\": citations[0] offsets must satisfy 0 <= startIndex <= endIndex <= text length (6), got startIndex 0, endIndex 99; fix the offsets, or omit them to cite the whole text","type":"server_error","code":"aimock_invalid_fixture_citation"}}
===== gemini non-stream (out-of-range citation offsets)
HTTP 500
{"error":{"code":500,"message":"aimock: invalid fixture \"citations\": citations[0] offsets must satisfy ...","status":"INTERNAL","details":[{"@type":"type.googleapis.com/google.rpc.ErrorInfo","reason":"AIMOCK_INVALID_FIXTURE_CITATION","domain":"aimock","metadata":{"code":"aimock_invalid_fixture_citation"}}]}}
===== anthropic non-stream (file missing data)
HTTP 500
{"type":"error","error":{"type":"api_error","code":"aimock_invalid_fixture_file","message":"aimock: invalid fixture \"files\": files[0].data must be a string, got undefined; set the base64-encoded file bytes"}}
===== responses non-stream (non-image file)
HTTP 500
{"error":{"message":"aimock: fixture \"files\" on the OpenAI Responses API carry only image/* files (as image_generation_call items); got mimeType \"application/pdf\"","type":"server_error","code":"aimock_unsupported_generated_file"}}
===== chat non-stream (malformed citation under refusal fault)
HTTP 500
{"error":{"message":"aimock: invalid fixture \"citations\": citations[0].url must be a non-empty string, got undefined; set the cited source URL","type":"server_error","code":"aimock_invalid_fixture_citation"}}
===== chat non-stream (valid citation under refusal fault)
HTTP 200
... "message":{"role":"assistant","content":null,"refusal":"I can't help with that."} ...   (no annotations: the refusal replaced the cited text)

With --validate-on-load, the base server prints no warning for these fixtures. This branch prints one warning per malformed fixture (fixtures 1, 2 and 4), as shown in the load warnings section above.

Through the official SDK clients

Earlier on this branch I also ran the citation and file fixtures through openai 6.49.0 and @anthropic-ai/sdk 0.78.0 against the running server. Non-streaming and accumulated streaming results both carried the annotations and citations shown above. The Responses client reported output types image_generation_call,message with result: iVBORw0KGgo=. The Chat Completions and Anthropic clients raised a 500 with aimock_unsupported_generated_file for the files fixture.

Tests and quality gates

• New tests cover each provider (server.test.ts, messages.test.ts, gemini.test.ts, responses.test.ts), the builders' cited source text, validation and load warnings (fixture-citations-files-validation.test.ts), the fault matrix and fault journaling (misbehavior-citations.test.ts, misbehavior-fixture-output-validation.test.ts, misbehavior-fault-journal-order.test.ts, ws-responses-fault-journal-order.test.ts), and the InvalidFixtureOutputError export.
• pnpm format:check, pnpm lint, pnpm typecheck, pnpm build and pnpm api:report are clean. pnpm test: 18539 tests pass.

Deferred

These are out of scope for this PR and are tracked in our follow-ups list:

• The other wires (Bedrock, Ollama, Cohere, Gemini Interactions, Gemini Live, Realtime) still ignore citations and files. Each needs a decision between serving them and rejecting them like Chat Completions and Anthropic.
• File citations are skipped on Chat Completions, Anthropic and Gemini without a warning, because those wires have no shape for them.
• files[].data is not checked for empty or non-base64 content.
• The misbehavior planner's check, and the load validator's text lookup for malformed blocks, treat any thrown error as "cannot serve" without logging it. The fixture still answers its coded 500 on the normal path, and validateBlocks reports malformed blocks as a load error.
• On non-stream Chat Completions, a citation error waits out the configured latency before the 500.
• Type tightening (for example, marking file-only fields as never on FixtureUrlCitation) and some additional test cases.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@578

commit: 9eaa9f1