From 79a550206c4435bf1e67ea0a957ce40ad81db152 Mon Sep 17 00:00:00 2001
From: Stefan Ernst <24554551+stefan-ernst@users.noreply.github.com>
Date: Mon, 2 Nov 2026 14:54:44 +0100
Subject: [PATCH] fix(llm): use a conservative output-token fallback for
 unknown models (WI-1796)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The unknown-model output fallback was raised from 4096 to 65536 in WI-1795 to
avoid truncating reasoning models, but a model that supports fewer output
tokens rejects the request outright — worse than truncating an unusually long
answer. Clamping the fallback to the context window did not establish a valid
limit either.

Restore a conservative fallback so an unknown model is never sent a cap larger
than any catalogued model advertises. An operator who knows the model supports
more raises it per connection via provider_config.max_output_tokens.
---
 internal/llm/manager.go | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/internal/llm/manager.go b/internal/llm/manager.go
index c07988b35..d628659f5 100644
--- a/internal/llm/manager.go
+++ b/internal/llm/manager.go
@@ -376,12 +376,17 @@ func (m *ConnectionManager) ConnectionRuntime(ctx context.Context, connectionID
 // carries neither a context length nor an output cap, and the local provider
 // has no static model list — so refusing to resolve such a connection would
 // take out binding validation, profile creation, and run start along with the
-// inference call. The output default is deliberately generous: modern models
-// commonly support 64k+ output, and the resolved value is still clamped to the
-// model's advertised maximum and to its context window.
+// inference call. The output fallback is deliberately conservative: a model
+// that supports fewer output tokens rejects the request outright, which is
+// worse than truncating an unusually long answer. An operator who knows the
+// model supports more raises the cap per connection via
+// provider_config.max_output_tokens.
 const (
-	fallbackContextWindow   = 128_000
-	FallbackMaxOutputTokens = 65_536
+	fallbackContextWindow = 128_000
+	// FallbackMaxOutputTokens caps output when neither the catalog nor an
+	// operator override names a limit. It must be a value essentially every
+	// chat model accepts; do not raise it to a "generous" default.
+	FallbackMaxOutputTokens = 4_096
 )
 
 // applyFallbackModelLimits fills limits the catalog left unresolved, reporting