Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"source": {
"source": "npm",
"package": "@copilotkit/aimock",
"version": "^1.46.0"
"version": "^1.47.0"
},
"description": "Fixture authoring skill for @copilotkit/aimock — LLM, multimedia (image/TTS/transcription/video), MCP, A2A, AG-UI, vector, embeddings, structured output, sequential responses, streaming physics, record/replay, agent loop patterns, and debugging"
}
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "aimock",
"version": "1.46.0",
"version": "1.47.0",
"description": "Fixture authoring guidance for @copilotkit/aimock — LLM, multimedia, MCP, A2A, AG-UI, vector, and service mocking",
"author": {
"name": "CopilotKit"
Expand Down
23 changes: 6 additions & 17 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,17 @@
# @copilotkit/aimock

## [Unreleased]
## [1.47.0] - 2026-10-26

### Added

- Fixture `citations` serve OpenAI Responses `url_citation` annotations; fixture `files` serve Gemini `inlineData` parts (#550 by @abhiaiyer91)
- Fixture `citations` also serve Chat Completions `url_citation` annotations (`message.annotations`, one streamed `delta.annotations` chunk), Anthropic `web_search_result_location` citations on text blocks (`citations_delta` when streaming), and Gemini `groundingMetadata` (`groundingChunks` + `groundingSupports`, byte offsets). A citation with `type: "file"` (`fileId`, `filename`, optional `index`, which defaults to the end of the text) serves an OpenAI Responses `file_citation` annotation on `output_text`, streamed through `response.output_text.annotation.added`; Chat Completions, Anthropic and Gemini skip it. Closes #547.
- Fixture `files` serve OpenAI Responses `image_generation_call` output items (`result` holds the base64 bytes), streamed with the `response.image_generation_call.*` events. Chat Completions and Anthropic Messages reject a fixture with `files` with a 500 carrying `aimock_unsupported_generated_file` (new `UnsupportedGeneratedFileError`). Closes #548.
- Malformed fixture `citations` or `files` (not an array, a missing or mistyped field, a non-integer offset, an offset outside the text, or citations on a turn with no text: tool calls only, `blocks` with no text block, or empty content) answer a 500 before any content is written: `aimock_invalid_fixture_citation` or `aimock_invalid_fixture_file`. A non-`image/*` file on the OpenAI Responses API answers a 500 with `aimock_unsupported_generated_file`. This is the same on every wire that reads these keys, with or without a misbehavior fault: the fixture is checked on its original turn before any fault applies (only an invalid misbehavior config answers its own 400 or 501 first). A valid fixture under a fault that changes the reply text is served without its `citations`. Only Chat Completions, Anthropic Messages, Gemini and OpenAI Responses read these keys; other wires ignore them.
- Fixture load validation warns about malformed `citations` and `files`, with the same problem description the 500 carries (the 500 message adds an `aimock: invalid fixture` prefix; the warning adds which 500 the requests answer). The fixture still loads.
- `InvalidFixtureOutputError` (the error behind `aimock_invalid_fixture_citation` and `aimock_invalid_fixture_file`) is exported from `@copilotkit/aimock`.
- Opt-in fixture `nonStreamingTimingAllProviders: true` delays non-streaming replies on OpenAI chat, Anthropic, Responses, Gemini/Vertex, Gemini Interactions, Cohere, Bedrock invoke/converse and Ollama; `nonStreamingTiming` is unchanged (OpenAI chat only); error replies are never delayed. On the OpenAI Responses API the wait also counts the stream frames that fixture `citations` and generated `files` add
- Raw passthrough fixtures: a `raw` response (`wire`, `events`, `json`, optional `status`, `headers`, and `done` on `openai-chat`) sends provider events or JSON exactly as written, with only the named wire's framing, on the HTTP LLM wires (OpenAI Responses and Chat Completions, Anthropic, Gemini/Vertex, Gemini Interactions, Cohere, Ollama `/api/chat` as `ollama` and `/api/generate` as `ollama-generate`, Bedrock invoke and converse). A raw fixture matches only its own wire. A request whose mode has no body answers that wire's 500 with `aimock_raw_mode_unavailable`; an invalid raw fixture added in code answers a 500 with `aimock_raw_fixture_invalid`, and `--validate-on-load` reports it at load. Streaming timing, truncation, disconnects and chaos apply; misbehavior faults skip raw fixtures. A non-streaming raw reply follows `nonStreamingTiming` like a typed fixture: `nonStreamingTiming: true` delays it on OpenAI Chat Completions only, `nonStreamingTimingAllProviders: true` on every provider; a raw reply with status 400 or above is never delayed, and with a `json` it answers JSON even to a streaming request. WebSocket Responses answers a raw fixture with an error event with the code `aimock_raw_mode_unavailable`. New exported types `RawResponse`, `RawWireId` and `WireId` ([#549](https://github.com/CopilotKit/aimock/issues/549))
- Fixture `citations` serve sources on Responses, Chat Completions, Claude and Gemini (#550, #578)
- Fixture `files` serve generated images on Gemini and OpenAI Responses (#550, #578)
- Raw passthrough fixtures send provider SSE events or JSON exactly as written (#580)
- Opt-in `nonStreamingTimingAllProviders` delays non-streaming replies on every provider (#579)

### Fixed

- Misbehavior faults on Chat Completions, Anthropic Messages, Gemini and the OpenAI Responses API (HTTP and WebSocket): a fixture the wire rejects (generated `files` on Chat Completions or Anthropic, a non-`image/*` file on Responses, malformed `citations` or `files`) is never faulted. It takes the normal path, is journaled as a 500 that keeps its fixture, and consumes no fault `times` budget or seed ordinal; `aimock_misbehavior_total` does not count it. A non-stream Chat Completions fault is journaled before its timing wait.
- Journal accuracy: a client that hangs up during the timing wait of any non-stream Chat Completions reply (faulted or not) is journaled as interrupted, not as served.
- Client disconnects stop delivery (no more chunks or waits) on OpenAI chat, Anthropic, Responses, Gemini/Vertex, Gemini Interactions, Cohere, Bedrock, Ollama, transcription streams and the Responses, Realtime and Gemini Live WebSockets
- Client disconnects on those surfaces are journaled as `interrupted` with `interruptReason: "client aborted"` (HTTP `status: 0` when nothing was sent); aimock-caused teardowns record `"disconnect"`. Journal snapshots of hung-up requests now carry these fields
- WebSocket: a client TCP FIN without a close frame now closes the connection at once (1006) and ends the turn
- Daily drift: the OpenAI Responses K9 reasoning-exhaustion probes (HTTP and WebSocket) request `gpt-5.4-mini`. OpenAI retired `o4-mini` on 2026-10-23 (its `/v1/models` `shutdown_date`), and it has answered `404 model_not_found` since. The probe model is one constant, `OPENAI_K9_PROBE_MODEL`. The usage check accepts `output_tokens` equal to `output_tokens_details.reasoning_tokens` (gpt-5.4-mini spends the 32-token cap on reasoning; o4-mini reported 0). The event-order check is unchanged, and the retained o4-mini replay fixtures are still graded against o4-mini. Drift harness only; aimock behavior is unchanged (#584).
- Daily drift: a preflight checks every model the live drift probes pin (`src/__tests__/drift/probe-models.ts`) against the provider's own retirement data before the run. A retired model fails the run with `probe model X retired on DATE — update <constant>` instead of looking like API drift; a shutdown date within 30 days is a warning in the job summary and in Slack. The Gemini embeddings canary now probes `gemini-embedding-001`; `text-embedding-004` answers 404. Drift harness only; aimock behavior is unchanged.
- Docker image: the base image is pinned to `node:22.23.4-alpine` by digest. Node 22.23.4 fixes CVE-2026-111497 (Brotli decompression memory bug); the 1.46.0 image shipped Node 22.23.3.
- Client disconnects stop and journal every LLM reply; Docker image on Node 22.23.4 (#579, #576)

## [1.46.0] - 2026-10-20

Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,13 +92,16 @@ Run them all on one port with `npx @copilotkit/aimock --config aimock.json`, or
- **[Native model catalogs and token counts](https://aimock.copilotkit.dev/claude-messages#model-catalog)** — Anthropic `GET /v1/models` (selected by `anthropic-version`, or `x-api-key` without `Authorization`; the list route needs `policy.modelCatalog: "native"` and otherwise stays the OpenAI list) and Gemini `GET /v1beta/models` list and detail routes with provider paging, built from fixture model IDs with synthetic metadata; `POST /v1/messages/count_tokens` and Gemini `:countTokens` return a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1. These routes never proxy or record
- **Context-Based Fixture Routing** — `X-AIMock-Context` header scopes fixtures per integration; fixtures with `match.context` only match requests carrying that context, fixtures without it remain shared
- **[Drift Detection](https://aimock.copilotkit.dev/drift-detection)** — Daily CI validation against real APIs
- **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter`
- **[Streaming Physics](https://aimock.copilotkit.dev/streaming-physics)** — Configurable `ttft`, `tps`, and `jitter`; opt in with `nonStreamingTimingAllProviders: true` to give non-streaming replies the same timing on every LLM provider
- **[Client Disconnects](https://aimock.copilotkit.dev/error-injection#client-disconnects)** — When the client hangs up, every LLM provider (HTTP and WebSocket) stops sending and journals the request as `interrupted`, so you can test cancel and abort paths
- **[WebSocket APIs](https://aimock.copilotkit.dev/websocket)** — OpenAI Realtime (GA protocol with models: gpt-realtime, gpt-realtime-2, gpt-realtime-1.5, gpt-realtime-mini; transcription/translation via gpt-4o-transcribe, gpt-4o-mini-transcribe, whisper-1; image input; commentary phase), Responses WS, Gemini Live
- **[GPT-Live](https://aimock.copilotkit.dev/websocket#openai-live)** — Mock, record, and replay primary WebSocket sessions at `/v1/live/sessions` in client and managed modes, with fixture-backed audio and backend tool continuation. Saved fixtures replay offline; aimock does not synthesize speech. This voice surface is separate from OpenAI text-model family support; WebRTC/SDP and sideband connections are outside this scope.
- **[Prometheus Metrics](https://aimock.copilotkit.dev/metrics)** — Request counts, latencies, fixture match rates
- **[Docker + Helm](https://aimock.copilotkit.dev/docker)** — Container image and Helm chart for CI/CD
- **[Vitest & Jest Plugins](https://aimock.copilotkit.dev/test-plugins)** — Zero-config `useAimock()` with auto lifecycle and env patching
- **[Response Overrides](https://aimock.copilotkit.dev/fixtures)** — Control `id`, `model`, `usage`, `finishReason` in fixture responses
- **[Citations and Generated Files](https://aimock.copilotkit.dev/fixtures#citations-and-files)** — Fixture `citations` serve source citations on OpenAI Responses (URL and file), Chat Completions, Claude and Gemini; fixture `files` serve generated images on the Responses API and Gemini. Each uses the provider's own format, streaming and non-streaming
- **[Raw Passthrough](https://aimock.copilotkit.dev/raw-passthrough)** — A `raw` fixture sends provider SSE events or JSON exactly as written on any HTTP LLM wire, for fields aimock does not model yet; timing, chaos and disconnects still apply
- **[Ordered Blocks](https://aimock.copilotkit.dev/fixtures#ordered-blocks)** — A `blocks` array streams text and tool calls in any order (tool-first or interleaved); blocks-only fixtures are first-class, and the recorder captures order from genuinely tool-first/interleaved streams
- **[Namespaced and Custom Tool Calls](https://aimock.copilotkit.dev/responses-api#namespaced-and-custom-tool-calls)** — Responses API fixtures can return namespaced function calls and custom tool calls (`custom_tool_call`) over HTTP streaming, non-streaming and WebSocket, and can match on `toolNamespace`. This lets you script OpenAI Codex CLI's namespaced MCP and sub-agent tools and its freeform `apply_patch` tool (with the `responsesTools: "extended"` server option)
- **[Streaming Usage Chunks](https://aimock.copilotkit.dev/streaming-physics)** — `stream_options.include_usage` support emits a final chunk with token counts, matching OpenAI's streaming usage protocol
Expand Down
2 changes: 1 addition & 1 deletion charts/aimock/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ name: aimock
description: Mock infrastructure for AI application testing (OpenAI, Anthropic, Gemini, MCP, A2A, vector)
type: application
version: 0.1.0
appVersion: "1.46.0"
appVersion: "1.47.0"
5 changes: 5 additions & 0 deletions docs/chat-completions/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ <h1>OpenAI Chat Completions</h1>
<a href="/raw-passthrough">raw passthrough fixture</a> with
<code>wire: "openai-chat"</code>.
</p>
<p>
Fixture <code>citations</code> serve <code>url_citation</code> annotations. A fixture with
generated <code>files</code> answers a 500, because Chat Completions has no place for
them. See <a href="/fixtures#citations-and-files">Citations and generated files</a>.
</p>

<h2>Endpoint</h2>
<table class="endpoint-table">
Expand Down
6 changes: 6 additions & 0 deletions docs/claude-messages/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ <h1>Anthropic Claude Messages API</h1>
<a href="/raw-passthrough">raw passthrough fixture</a> with
<code>wire: "anthropic"</code>.
</p>
<p>
Fixture <code>citations</code> serve <code>web_search_result_location</code> citations on
text blocks. A fixture with generated <code>files</code> answers a 500, because the
Messages API has no place for them. See
<a href="/fixtures#citations-and-files">Citations and generated files</a>.
</p>

<h2>Endpoint</h2>
<table class="endpoint-table">
Expand Down
62 changes: 31 additions & 31 deletions docs/docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -262,26 +262,47 @@ <h2>What's New</h2>

<div class="highlight-grid">
<div class="highlight-card">
<span class="highlight-card-title">Model Catalogs &amp; Token Counts</span>
<span class="version-note">New in 1.46.0</span>
<span class="highlight-card-title">Citations &amp; Generated Images</span>
<span class="version-note">New in 1.47.0</span>
<p class="highlight-card-desc">
List and look up Gemini and Anthropic models in their native formats (the Anthropic
list is opt-in), and estimate prompt size with <code>count_tokens</code> /
<code>countTokens</code>
Script source citations on OpenAI, Claude and Gemini replies, and generated images on
the Responses API and Gemini, in each provider's own format
</p>
<a href="/claude-messages#model-catalog" class="highlight-card-cta"
<a href="/fixtures#citations-and-files" class="highlight-card-cta"
>Docs <span>&rarr;</span></a
>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Raw Passthrough</span>
<span class="version-note">New in 1.47.0</span>
<p class="highlight-card-desc">
Send provider events or JSON exactly as written, for fields aimock does not model yet,
with timing, chaos and disconnects still applied
</p>
<a href="/raw-passthrough" class="highlight-card-cta">Docs <span>&rarr;</span></a>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Stream Realism</span>
<span class="version-note">New in 1.47.0</span>
<p class="highlight-card-desc">
Every LLM provider stops sending and journals the request when the client hangs up,
and non-streaming replies on every provider can opt in to realistic timing
</p>
<a href="/error-injection#client-disconnects" class="highlight-card-cta"
>Docs <span>&rarr;</span></a
>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Native Auth Errors</span>
<span class="highlight-card-title">Native Catalogs &amp; Auth Errors</span>
<span class="version-note">New in 1.46.0</span>
<p class="highlight-card-desc">
Opt in to provider-native 401 bodies and required-header checks, so you can test how
your app handles a bad key or a missing <code>anthropic-version</code>
Gemini and Anthropic model lists, token counts, and opt-in provider-native 401s, so
you can test model pickers and bad-key handling
</p>
<a href="/claude-messages#authentication-errors" class="highlight-card-cta"
<a href="/claude-messages#model-catalog" class="highlight-card-cta"
>Docs <span>&rarr;</span></a
>
</div>
Expand All @@ -306,18 +327,6 @@ <h2>What's New</h2>
>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Stream Realism</span>
<span class="version-note">New in 1.46.0</span>
<p class="highlight-card-desc">
Anthropic streams stop and journal when the client disconnects, and non-streaming
OpenAI replies can opt in to streaming-profile timing
</p>
<a href="/streaming-physics#non-streaming" class="highlight-card-cta"
>Docs <span>&rarr;</span></a
>
</div>

<div class="highlight-card">
<span class="highlight-card-title">Responses API Tools</span>
<p class="highlight-card-desc">
Expand All @@ -336,15 +345,6 @@ <h2>What's New</h2>
</p>
<a href="/vector-stores" class="highlight-card-cta">Docs <span>&rarr;</span></a>
</div>

<div class="highlight-card">
<span class="highlight-card-title">GPT-Live Sessions</span>
<p class="highlight-card-desc">
Mock, record, and replay OpenAI GPT-Live sessions, with message, audio, and lifetime
limits
</p>
<a href="/websocket#openai-live" class="highlight-card-cta">Docs <span>&rarr;</span></a>
</div>
</div>

<!-- ─── The Suite (compact table) ───────────────────────────── -->
Expand Down
Loading
Loading