Skip to content

feat: stop delivery on client disconnect across providers, and opt-in non-streaming timing for all providers - #579

Merged
jpr5 merged 7 commits into
mainfrom
feat/cross-provider-disconnect-timing
Oct 24, 2026
Merged

feat: stop delivery on client disconnect across providers, and opt-in non-streaming timing for all providers#579
jpr5 merged 7 commits into
mainfrom
feat/cross-provider-disconnect-timing

Conversation

@jpr5

@jpr5 jpr5 commented Oct 24, 2026

Copy link
Copy Markdown
Contributor

Summary

When a client hangs up, aimock now stops delivering the reply on every provider and every transport, and the journal records it. Before this change, only some paths noticed the hang-up. The others kept writing into a dead socket, or kept waiting out a delay, and journaled the turn as a normal 200.

  • Client hang-ups are journaled with interrupted: true and the reason "client aborted". When the client left before any byte of the reply reached its socket, the journaled status is 0, because the client never got a status line.
  • When aimock itself tears the connection down (for example a chaos disconnect), the reason is "disconnect", so you can tell your client's behaviour apart from aimock's.
  • New opt-in fixture field nonStreamingTimingAllProviders. It applies the non-streaming reply delay on every HTTP provider: OpenAI Chat, Responses, Anthropic, Gemini, Vertex, Gemini Interactions, Bedrock Invoke and Converse, Ollama chat and generate, and Cohere. The existing nonStreamingTiming field is unchanged and still applies only where it did before.
  • A client that hangs up during the non-streaming wait now ends the wait at once. The reply is not sent and the turn is journaled as a client abort.
  • WebSocket connections now detect a peer FIN that arrives without a close frame. Before, a half-open socket stayed open until a later write failed.
  • Error replies are never delayed. A 4xx or 5xx is sent at once, with or without the new field.

Compatibility

Additive only.

  • The defaults are byte-identical to 1.46.0. Committed byte goldens and delay goldens for every provider codec, captured on the pre-change build, pin this.
  • The exported writers keep their exact 1.46.0 contract. A pinned matrix of 62 cases, recorded on the pre-change build, checks this.
  • The API report diff adds one field, nonStreamingTimingAllProviders?: boolean, on Fixture and FixtureFileEntry. No other public surface changes.

Red-green evidence

Each row was run against the real aimock CLI with curl or a raw WebSocket client, then the journal was read. RED is the pre-change build. GREEN is this branch.

  • Client abort: the client hangs up mid-stream.
  • Abort in wait: the client hangs up during the non-streaming wait.
  • Opt-in delay: a non-streaming reply with nonStreamingTimingAllProviders: true. The figure is the time to the reply.
  • Proxy abort: the client hangs up while aimock relays an upstream stream.
Provider Client abort (RED → GREEN) Abort in wait (RED → GREEN) Opt-in delay (RED → GREEN) Proxy abort (RED → GREEN)
OpenAI Chat not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 5 ms → 308 ms client aborted → client aborted
Anthropic client aborted → client aborted full reply sent, 200 → stops at once, client aborted, status 0 5 ms → 309 ms not journaled → client aborted
Responses not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 6 ms → 317 ms not journaled → client aborted
Gemini not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 7 ms → 313 ms not journaled → client aborted
Vertex AI not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 6 ms → 319 ms not journaled → client aborted
Gemini Interactions not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 8 ms → 311 ms not journaled → client aborted
Bedrock Invoke not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 7 ms → 313 ms not journaled → client aborted
Bedrock Converse not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 7 ms → 315 ms not journaled → client aborted
Ollama chat not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 5 ms → 309 ms not journaled → client aborted
Ollama generate not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 5 ms → 309 ms not journaled → client aborted
Cohere not journaled → client aborted full reply sent, 200 → stops at once, client aborted, status 0 6 ms → 313 ms not journaled → client aborted
Transcription not journaled → client aborted not journaled → client aborted, status 0 not applicable (stays instant by design) not applicable
WS Responses interrupted with no reason, about 9 s later → client aborted not applicable not applicable not applicable
WS Realtime not journaled → client aborted (close during the delay or after a delta) not applicable not applicable not applicable
Gemini Live not journaled → client aborted (text, text plus tool calls, close frame) not applicable not applicable not applicable

WebSocket FIN without a close frame (WS Responses): RED, the server kept the socket open for the full 30 s probe and kept reading. GREEN, the server closed it within 0.1 ms of the FIN.

In every GREEN run, the healthy stream, the default non-streaming reply, nonStreamingTiming: true, and the 429 error reply have the same bytes (same body hash) and the same timing as RED. The 429 is sent in about 5 ms in both. The streamed bytes before each abort are also identical between RED and GREEN.

Not in this PR

  • A WebSocket turn that aimock itself closes during a long wait can still be labelled as a client abort.
  • The status-line check for a pipelined HTTP request on a keep-alive socket falls back to headersSent.
  • A proxied response that is torn down at the exact moment it finishes can be journaled as complete (writableFinished edge).

Spec: cross-provider disconnect and timing (Notion)

@jpr5
feat: add nonStreamingTimingAllProviders fixture opt-in
4d9ceba 
Add the nonStreamingTimingAllProviders field to Fixture and
FixtureFileEntry, pass it through the fixture loader, and warn when
either non-streaming timing opt-in is not a boolean.
@jpr5 jpr5 added the public-api-addition Intended public API report changes reviewed for compatibility; freshness remains required label Oct 24, 2026
@pkg-pr-new

pkg-pr-new Bot commented Oct 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@579

commit: 6a35aff

jpr5 added 6 commits October 23, 2026 17:35
@jpr5
test: pin writer bytes, delay order and the exported writer contract
08d4db0 
Add golden byte and delay snapshots for every streaming codec and
handler, the exported-writer base-contract matrix recorded at 508dd9b,
and the shared delivery-lifecycle harness that the provider tests use.
@jpr5
feat(response): shared delivery lifecycle, frame loop, timing and proxy
883ada6 
Add src/response (lifecycle, frame-loop, timing, deliver, proxy) as the
one place that decides how a reply is paced, when a client disconnect
stops it, and how teardown is journaled. Move the SSE, NDJSON and AWS
event-stream codecs and writers onto it while keeping the exported
writers on the 508dd9b contract, detect a proxy client hang-up on an
actual close, and remove the delay() abort listener when its timer
fires.
@jpr5
fix: stop OpenAI chat and Anthropic delivery on client disconnect
64d2140 
Serve OpenAI Chat Completions and Anthropic Messages through the shared
delivery foundation, so a client disconnect stops the reply and is
journaled as client aborted, and honor nonStreamingTimingAllProviders
on their non-streaming replies.
@jpr5
fix: stop HTTP provider delivery on client disconnect
2f86eb2 
Move Responses, Gemini and Vertex, Gemini Interactions, Cohere, Bedrock
InvokeModel and Converse, Ollama chat and generate, and audio
transcription onto the shared delivery foundation. A client disconnect
now stops each reply and is journaled, record-mode transcription relays
journal a client hang-up, and nonStreamingTimingAllProviders delays the
non-streaming replies of every provider except transcription.
@jpr5
fix: journal WebSocket client disconnects through the shared lifecycle
365f7ae 
Treat a client FIN without a close frame as a WebSocket disconnect, and
route Responses, Realtime and Gemini Live turns through wsLifecycle
with one closed-socket rule, so a client close mid-turn is journaled as
client aborted.
@jpr5
docs: document client disconnects and nonStreamingTimingAllProviders
6a35aff 
Note client disconnect handling and non-streaming timing on each
provider page, the streaming-physics and error-injection pages, the
fixtures reference and the write-fixtures skill. Add the CHANGELOG
entry and regenerate the API report for the new Fixture field.
@jpr5
jpr5 force-pushed the feat/cross-provider-disconnect-timing branch from 8d34bfc to 6a35aff Compare October 24, 2026 00:35
@jpr5
jpr5 merged commit 42902c2 into main Oct 24, 2026
30 checks passed
@jpr5
jpr5 deleted the feat/cross-provider-disconnect-timing branch October 24, 2026 01:29
@jpr5 jpr5 mentioned this pull request Oct 24, 2026
Merged
jpr5 added a commit that referenced this pull request Oct 24, 2026
@jpr5
feat: raw SSE/JSON passthrough fixtures for LLM endpoints (#580)
0ad4e97 
Closes #549

## Summary

This PR adds raw passthrough fixtures. A fixture response of the form `{
raw: { wire, events?, json?, status?, headers?, done? } }` sends
provider events or JSON exactly as written. aimock adds only the framing
of the named wire. No overrides, reasoning gating, usage synthesis,
provider shaping or drift checks apply to a raw fixture.

**Per-wire framing.** Each wire uses the same bytes that the typed
handlers write for that wire:

- Named SSE (`event: <type>` then `data: <json>`): `openai-responses`,
`anthropic`, `cohere`.
- Data SSE (`data: <json>`): `openai-chat`, `gemini`,
`gemini-interactions`. On `openai-chat`, the stream ends with `data:
[DONE]` unless `raw.done` is `false`.
- NDJSON: `ollama` (`/api/chat`) and `ollama-generate`
(`/api/generate`). These are two wires because the frames of the two
routes have different shapes.
- AWS event stream: `bedrock-invoke` wraps each event as a `chunk` frame
with base64 `bytes`. `bedrock-converse` uses the single key of each
event as the frame's event type and its value as the payload.

That gives 10 wires: `openai-responses`, `openai-chat` (including the
OpenRouter and Azure routes), `anthropic`, `gemini` (including Vertex),
`gemini-interactions`, `cohere`, `ollama`, `ollama-generate`,
`bedrock-invoke` and `bedrock-converse`. The router offers a raw fixture
only to the handler for its `raw.wire`. The handler supplies the wire,
never the request body. When a raw fixture matches a request in every
way except its wire, and no other fixture matches, the server log names
both wires instead of returning a silent 404.

**Stream vs JSON.** A streaming request gets `events` and a
non-streaming request gets `json`. If the body that the request needs is
missing, aimock answers with the native 500 error of that wire and the
code `aimock_raw_mode_unavailable`.

**Status and headers.** `status` defaults to 200 and must be 200-599,
except 204 and 304. `headers` replace the defaults case-insensitively.
If `headers` has no Content-Type, a stream gets the wire's own type and
a non-streaming reply gets `application/json`. aimock rejects headers
that it must set itself, because it frames the body.

**Error codes.** An invalid raw fixture gets a 500 in the wire's own
error shape with the code `aimock_raw_fixture_invalid`. Examples are a
factory that returns `status: 99`, or a factory that returns a raw
fixture for another wire. On Gemini, Ollama and OpenRouter the code is
prefixed to the message.

**Validation.** `--validate-on-load`, `addFixturesFromJSON` and `POST
/__aimock/fixtures` all check raw fixtures with one shared checker:

- the wire is known, and `events` or `json` is present;
- the events are plain JSON objects;
- the per-wire type and single-key rules pass;
- `status` and the header names and values are valid;
- `done` is a boolean, and only on `openai-chat`;
- `customToolCalls` and `responsesBlocks` are not next to `raw`;
- the fixture does not set its own misbehavior.

Unknown keys inside `raw` and an empty `events` are warnings.
`chunkSize` and `openRouterProcessing` on a raw fixture are warnings,
because they are ignored. The same check runs again when a fixture is
served, so fixtures added with `on()`, `addFixture()` or factories are
checked too.

**Misbehavior skip.** Model misbehavior faults skip raw fixtures. aimock
skips a server-level fault, serves the raw reply, and records the fault
as skipped in the journal. An `X-AIMock-Misbehavior` header returns 501
`aimock_misbehavior_not_applicable`. A raw fixture that sets its own
misbehavior is an invalid fixture: load validation reports it, and when
it is served it gets `aimock_raw_fixture_invalid`.

**Timing rules.** `latency`, `streamingProfile`, `recordedTimings` and
`replaySpeed` pace streamed raw events in the same way as typed frames.
`truncateAfterChunks`, `disconnectAfterMs` and chaos also apply, and
each raw event counts as one chunk. A non-streaming raw reply follows
`nonStreamingTiming` like a typed fixture. `nonStreamingTiming: true`
delays the reply on OpenAI Chat Completions only.
`nonStreamingTimingAllProviders: true` delays it on every raw wire.
aimock sizes the wait from the served JSON.

**Error replies.** A raw reply with a status of 400 or more is never
delayed, streaming or not. If the reply has a `json`, a streaming
request gets that JSON instead of events, because provider APIs answer
errors that way. Error replies are recorded in the journal.

**WS Responses.** `/v1/responses` over WebSocket answers a raw fixture
with an error event with the code `aimock_raw_mode_unavailable` and the
message "raw fixtures are not yet supported on the WebSocket transport".
Before this change, it answered with the generic unknown-type error.

**OpenRouter failover.** In an OpenRouter `models[]` fallback chain, an
`openai-chat` raw fixture with a status of 400 or more fails over to the
next model. A typed error fixture with the same status does the same.

## Compatibility

- **Additive only.** The feature is opt-in. A response is raw only if
`raw` is a non-null, non-array object and no other response type's
marker key (`content`, `toolCalls`, `error`, `json`, and so on) is
present. A typed fixture with a stray `raw` key is still served, and
validated, exactly as before.
- **Typed fixtures unchanged.** Existing typed fixtures produce output
byte-identical to main: an offline probe compared 321 cases (status,
headers except `date`, and body) on the pre-review build, and the
no-regression snapshot plus the #579 byte goldens pass at the final
head. The no-regression suite (`raw-no-regression.test.ts`, with a
snapshot) is green on main and on this branch.
- **API report.** The API report changes are additions only:
- new exported types `RawResponse`, `RawWireId` and `WireId` (`WireId`
was already declared, and is now exported);
- `RawResponse` is added to the `FixtureResponse` and
`FixtureFileResponse` unions;
  - a new `@internal` `wire` field on `MatchOptions`.

  `isRawResponse` is internal and not exported.

## Red-green evidence

Each wire was reproduced against a real aimock server, with and without
`--validate-on-load`, and with the real provider SDK tests. Before the
change, every raw fixture request got a 500 with the body `Fixture
response did not match any known type`. After the change, the authored
events or JSON are served with the wire's framing.

| Wire | Before (curl, stream / non-stream) | After (curl, stream /
non-stream) | SDK tests before | SDK tests after |
| --- | --- | --- | --- | --- |
| openai-responses | 500 / 500, "did not match any known type" | 200
`text/event-stream` / 200 `application/json` | 16 failed, 1 passed (17)
| 17 passed (17) |
| openai-chat | 500 / 500, "did not match any known type" | 200
`text/event-stream`, ends `data: [DONE]` / 200 `application/json` | 14
failed (14) | 14 passed (14) |
| anthropic | 500 / 500, "did not match any known type" | 200
`text/event-stream` / 200 | 9 failed (9) | 9 passed (9) |
| gemini, gemini-interactions | 500 / 500 on both, "did not match any
known type" | 200 / 200 on both | 16 failed (16) | 16 passed (16) |
| cohere | 500 / 500, "did not match any known type" | 200
`text/event-stream` / 200 | 21 failed (21), cohere and ollama together |
21 passed (21), cohere and ollama together |
| ollama, ollama-generate | `/api/chat` 500, `/api/generate` 500, "did
not match any known type" | `/api/chat` 200, `/api/generate` 200,
`application/x-ndjson` | (see cohere row) | (see cohere row) |
| bedrock-invoke, bedrock-converse | 500 / 500 on both, "did not match
any known type" | 200 `application/vnd.amazon.eventstream` / 200 on both
| 13 failed (13), the SDK reported UnknownError | 13 passed (13) |

The split between `ollama` and `ollama-generate` has its own red-green.
Before the split, the test "/api/generate gets the ollama-generate
fixture in generate shape" failed (1 failed, 1 passed). After the split,
it passes (2 passed).

The full suite, typecheck, lint, format check, build, goldens/contract
tests and the API report check all pass on this branch.

## Not in this PR

- Raw fixtures over WebSocket Responses, and `record.raw` (recording raw
fixtures). Both are in spec PR3.
- Bedrock exception framing for raw error replies.

Spec: raw SSE/JSON passthrough fixtures (Notion)
@jpr5 jpr5 mentioned this pull request Oct 27, 2026
Merged
jpr5 added a commit that referenced this pull request Oct 27, 2026
@jpr5
chore: release v1.47.0 (#588)
61a5002 
Release v1.47.0. Merging this PR publishes through
`publish-release.yml`: npm, the git tag, the GitHub Release and Slack.

## Changelog

### Added

- Fixture `citations` serve sources on Responses, Chat Completions,
Claude and Gemini (#550, #578)
- Fixture `files` serve generated images on Gemini and OpenAI Responses
(#550, #578)
- Raw passthrough fixtures send provider SSE events or JSON exactly as
written (#580)
- Opt-in `nonStreamingTimingAllProviders` delays non-streaming replies
on every provider (#579)

### Fixed

- Client disconnects stop and journal every LLM reply; Docker image on
Node 22.23.4 (#579, #576)

The detailed notes from `[Unreleased]` are condensed into these bullets.
The full detail is in the linked PRs. The drift-harness and CI changes
(#583–#587) do not change what users see, so they are left out.

## Docs (separate commit)

- The fixtures page gets a new **Citations and generated files**
section: the fixture shape, the wire shape for each provider, the error
codes, and override-table and provider-matrix rows. The Responses, Chat
Completions, Claude and Gemini pages link to it.
- What's New has three 1.47.0 cards: citations and generated images, raw
passthrough, and stream realism (disconnects and timing on every
provider). The two 1.46.0 cards are merged into one. The GPT-Live card
is removed because it shipped in 1.43.0, more than a month ago.
- The README has new feature bullets for citations and files, raw
passthrough and client disconnects. The Streaming Physics bullet now
mentions `nonStreamingTimingAllProviders`.
- The competitive matrix has three new hand-reviewed rows:
provider-native citations, model-generated images, and stopping on
client disconnect. Each competitor cell was checked against a shallow
clone of that competitor's source on 2026-10-26, and an HTML comment on
each row records the evidence. The rows are added to `HAND_ROWS`, so the
weekly scan never rewrites them. The homepage ND count in the test is
now 56.

## Version carriers (`git grep -F 1.46.0`)

`package.json`, `charts/aimock/Chart.yaml` appVersion,
`.claude-plugin/plugin.json`, `.claude-plugin/marketplace.json`
(`^1.47.0`), the `docs/index.html` banner, the
`packages/aimock-pytest/README.md` default, and `_version.py`
`AIMOCK_VERSION`. These are not bumped: the CHANGELOG history, the
`version-note` labels for 1.46.0 features, and `pyproject.toml`.

## Local verification

- format:check, lint, typecheck, build, api:report with no `etc/` diff,
test:exports, the docs and matrix tests, and the full `pnpm test`.
- The citations and files wire shapes were exercised against the built
server for Chat, Responses, Claude and Gemini, including the 500 for
`files` on Chat Completions.
- Screenshots of What's New, the matrix and the fixtures section at 375,
430 and 1440 px.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

public-api-addition Intended public API report changes reviewed for compatibility; freshness remains required

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jpr5