From bb79406507a6f2fc75a532823d2cd809c8e3902a Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Thu, 6 Aug 2026 17:21:57 -0700 Subject: [PATCH 1/2] Add tool-calling scoping note to the CI-stub section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to review feedback on #597: defineBackend defaults the tools capability to false, and generate() calls that declare tools auto-require it — so the simple reply stub fails a tool-calling application up front, before its generate() runs. Call this out as a third scoping note: tool tests need tools: true plus a generate() that returns the expected tool-call sequence; the plain stub stays flagless. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/models/local-development.md b/reference/models/local-development.md index 5ee106e2..6f37a479 100644 --- a/reference/models/local-development.md +++ b/reference/models/local-development.md @@ -124,6 +124,6 @@ models.registerBackend( The stub still exercises routing, accounting, and [analytics](./analytics) — only the inference itself is faked. -Two scoping notes. `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. And a backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. +Three scoping notes. `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. A backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. And tool calling: `defineBackend` defaults the `tools` capability to `false`, and a `generate()` call that declares tools [automatically requires that capability](./routing#capability-routing) — so a tool-calling application fails against this stub up front, before its `generate()` ever runs. Tool-calling tests need a stub defined with `tools: true` whose `generate()` returns the tool-call sequence the test expects; leave the flag off a simple reply stub like this one, where the up-front failure is the honest signal. As with any registered backend, register the stub during component initialization (for example, in `handleApplication`) rather than at a test file's top level: each worker thread keeps its own registry, so registration must run in every thread that serves requests — see [`registerBackend()`](./backends#registerbackend). From 1e9a4d503c539d283120385a1148a60792a26e39 Mon Sep 17 00:00:00 2001 From: Nathan Heskew Date: Thu, 6 Aug 2026 17:25:50 -0700 Subject: [PATCH 2/2] Split the scoping notes into bold-led sub-paragraphs Per review: the three-notes paragraph had grown too dense to scan. Reuse the bold-led pattern from the parity-caveats section rather than a bulleted list, keeping the page's prose style. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Vq8zum5E6fbjzmLsdDDkJh --- reference/models/local-development.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/reference/models/local-development.md b/reference/models/local-development.md index 6f37a479..b6ae07c7 100644 --- a/reference/models/local-development.md +++ b/reference/models/local-development.md @@ -124,6 +124,12 @@ models.registerBackend( The stub still exercises routing, accounting, and [analytics](./analytics) — only the inference itself is faked. -Three scoping notes. `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. A backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. And tool calling: `defineBackend` defaults the `tools` capability to `false`, and a `generate()` call that declares tools [automatically requires that capability](./routing#capability-routing) — so a tool-calling application fails against this stub up front, before its `generate()` ever runs. Tool-calling tests need a stub defined with `tools: true` whose `generate()` returns the tool-call sequence the test expects; leave the flag off a simple reply stub like this one, where the up-front failure is the honest signal. +Three scoping notes: + +**Streaming.** `defineBackend` derives `generate` from a supplied `generateStream` (by draining the stream), but not the reverse — if the suite calls `generateStream()`, give the stub a `generateStream` implementation. + +**Embedding.** A backend registered under `'generative'` serves only generation: `embed()` resolves the `'embedding'` registry, so embedding tests need their own stub registered with `models.registerBackend('embedding', 'default', …)`. + +**Tool calling.** `defineBackend` defaults the `tools` capability to `false`, and a `generate()` call that declares tools [automatically requires that capability](./routing#capability-routing) — so a tool-calling application fails against this stub up front, before its `generate()` ever runs. Tool-calling tests need a stub defined with `tools: true` whose `generate()` returns the tool-call sequence the test expects; leave the flag off a simple reply stub like this one, where the up-front failure is the honest signal. As with any registered backend, register the stub during component initialization (for example, in `handleApplication`) rather than at a test file's top level: each worker thread keeps its own registry, so registration must run in every thread that serves requests — see [`registerBackend()`](./backends#registerbackend).