feat(providers): explicit single-model completion test

docs-site/src/content/docs/guides/web-dashboard.md

@@ -480,3 +480,30 @@ them. Both switches apply immediately; **Save** never changes them. When interce
 Claude/Anthropic accounts are managed only on **Providers > Anthropic > Accounts**: login and the browser option, the account roster with switch, pause, remove, and reauthentication, the paste-code field, account pool settings, and quota. Claude no longer has an Account sub-tab. Provider-level controls such as the default provider, removal, and the enabled switch stay on Providers.
 
 The bookmarks `#claude` and `#claude/code` open this page, and the former `#claude/settings` redirects to `#claude/code`; `#claude/desktop` selects the separate Claude Desktop tab. The old `#claude/account` bookmark redirects to `#providers?provider=anthropic&tab=accounts`. Legacy `#integrations/claude` and `#integrations/claude/desktop` bookmarks still redirect to `#claude/code` and `#claude/desktop`, respectively. Arrow keys move between tabs; Home and End select the first and last tab.
+
+
+## Test one provider model
+
+In **Providers**, **Test connection** checks discovery connectivity. Use the
+separate **Test completion** control to check whether a selected model answers.
+Choose a model and accept the confirmation that the request may incur provider
+charges. Cancelling the confirmation sends nothing. Changing the provider or
+model, or leaving the page, cancels the test and discards a stale result.
+
+The completion test sends `Reply exactly OK.` once with a 32-token output cap,
+a 20-second deadline and a 64 KiB stream limit. It reports TTFT at the first
+non-empty text delta, total latency, the upstream-reported model and a match,
+mismatch or unknown comparison against the wire model id. Missing or conflicting
+metadata is unknown; upstream metadata does not verify model identity. The panel
+shows the applicability reason when authentication, adapter or transport settings
+cannot be tested. No generated text is displayed, and the diagnostic does not
+update routing health, request history or the provider catalog.
+
+The authenticated management endpoint is
+`POST /api/providers/test-completion?name=<provider>` with exactly
+`{"model":"<model>","confirm":true}`. Dashboard sessions require their usual CSRF
+token. Invalid input returns 400, unknown providers return 404 and a concurrent
+test for the same provider returns 409. Completed diagnostics and applicability
+results return 200, including a structured error when inference fails. Destination
+policy and provider egress settings apply; redirects, retries and failover are disabled.
+For the CLI workflow, see [Providers and accounts](/reference/cli/providers-accounts/).

docs-site/src/content/docs/reference/cli/providers-accounts.md

@@ -24,7 +24,7 @@ both `--adapter` and `--base-url`.
 | `list` | `--json`, `--jsonl` | List configured providers and the remaining registry entries; `--jsonl` emits one configured provider object per line. |
 | `add <name>` | `--adapter <adapter>`, `--base-url <url>`, `--api-key <key>`, `--responses-path <path>`, `--auth-mode <key\|forward\|oauth\|local>`, `--default-model <model>`, `--set-default`, `--force`, `--json`, `--sync`, `--live` | Save locally by default; `--sync` attempts client sync in either output mode. `--live` writes through the running proxy and cannot combine with `--sync`. `--force` permits overwrite. |
 | `edit <name>` | provider field flags, `--headers <json>`, `--model-context-tier <model=default\|long_context>`, `--upstream-http-version <http1.1\|->`, `--fast <on\|off>`, `--context-window <N\|->`, `--json` | Edit validated live provider fields without replacing key pools. Repeat `--model-context-tier` for multiple Copilot models. `--headers` merges custom request headers; pass `{}` or `-` to clear them. |
-| `test <name>` | `--json` | Probe the real upstream model endpoint. |
+| `test <name>` | `--model <id>`, `--yes`, `--json` | Check discovery; with `--model`, request one confirmed completion test. |
 | `show <name>` | `--json` | Show config with API keys masked. |
 | `remove <name>` | `--json`, `--live`, `--yes` | Local removal refuses the default and last provider. Live removal requires `--yes` and uses server dependency checks, default reassignment and account/custom-model cleanup. |
 | `set-default <name>` | `--json`, `--live` | Select a local default, or update the running proxy with `--live`. |
@@ -120,10 +120,37 @@ notices. Null, absent or `not-requested` refresh can exit 0 without a client syn
 Read back before recovery; do not repeat a persisted write to repair convergence.
 Errors use safe prose on stderr, including in JSON mode, rather than a JSON error envelope.
 
-`provider test` is separate: it checks upstream model discovery connectivity. A failed connection exits 1; success and a static catalog with no applicable discovery endpoint exit 0.
+`provider test <name>` without `--model` checks upstream model discovery connectivity. A failed connection exits 1; success and a static catalog with no applicable discovery endpoint exit 0.
 `applicable: false` is an expected static-catalog result, not successful inference
 or a failed connection.
 
+To send one completion request for a chosen model, use:
+
+```bash
+ocx provider test my-relay --model my-model
+ocx provider test my-relay --model my-model --yes --json
+```
+
+This test may incur provider charges. An interactive terminal asks for confirmation;
+`--yes` accepts the charge warning. Non-interactive use and `--json` require
+`--yes`. Using `--yes` without `--model` is a usage error. Declining confirmation
+sends no request and exits 1. The command uses the running proxy's management API.
+
+The completion test sends the fixed prompt `Reply exactly OK.`, requests a 32-token output cap, stops after 20 seconds or 64 KiB, and never retries or fails over.
+It supports key-auth or local OpenAI Chat and Responses providers, including
+per-model wire overrides. Disabled, OAuth, forwarded-auth and unsupported adapter
+or transport configurations return an applicability reason without sending a completion.
+TLS profiles, HTTP version pins, enabled request pacing and custom executors with
+a provider proxy override are currently unsupported.
+
+Results include time to first text delta (TTFT), total latency, the requested and
+wire model ids, sanitized upstream model metadata and a distinct match, mismatch
+or unknown comparison. Missing or conflicting metadata is unknown. The comparison
+uses the upstream id before masking, against the id actually sent after alias and
+suffix handling. Provider-reported metadata cannot verify model identity:
+`identityVerified` is always `false`. Generated text is not returned. A failed
+completion exits 1; success and a not-applicable result exit 0.
+
 ### Transport settings and request pacing
 
 `provider edit` is already live. Use `--upstream-http-version http1.1` to pin that

gui/src/components/provider-workspace/ProviderCompletionTest.tsx

@@ -0,0 +1,110 @@
+/** An explicitly consented, bounded completion probe; it never runs during discovery. */
+import { useEffect, useRef, useState } from "react";
+import { confirmAction } from "../../action-dialogs";
+import { readJsonOrThrow } from "../../fetch-json";
+import { useI18n, useT } from "../../i18n/shared";
+import { Select } from "../../ui";
+import "./provider-completion-test.css";
+
+type InapplicableReason = "disabled" | "forward_auth" | "oauth_auth" | "unsupported_adapter" | "unsupported_transport";
+type TransportDetail = "tls_profile" | "upstream_http_version" | "request_pacing" | "custom_executor_egress";
+export type CompletionTestResult =
+  | { applicable: false; reason: InapplicableReason; adapter?: string; detail?: TransportDetail }
+  | { applicable: true; ok: boolean; provider: string; requestedModel: string; wireModel: string;
+      wire: "openai-chat" | "openai-responses"; upstreamModel: string | null;
+      modelComparison: "match" | "mismatch" | "unknown";
+      comparisonNote?: "missing_metadata" | "conflicting_metadata"; identityVerified: false;
+      ttftMs: number | null; totalLatencyMs: number; outputTokenCap: 32;
+      termination: "completed" | "failed" | "timeout" | "aborted" | "too_large";
+      error: { code: string; status?: number; message: string } | null };
+
+type Props = { apiBase: string; provider: string; models: string[] };
+
+export default function ProviderCompletionTest({ apiBase, provider, models }: Props) {
+  const t = useT();
+  const [model, setModel] = useState("");
+  const selectedModel = models.includes(model) ? model : "";
+  return (
+    <section className="pws-completion-test" aria-label={t("pws.completion.title")}>
+      <h4 className="pws-section-title">{t("pws.completion.title")}</h4>
+        <Select label={t("pws.completion.model")} value={selectedModel}
+          options={[{ value: "", label: t("pws.completion.choose") }, ...models.map(value => ({ value, label: value }))]}
+          onChange={setModel} />
+        <CompletionRun key={JSON.stringify([apiBase, provider, selectedModel])}
+          apiBase={apiBase} provider={provider} model={selectedModel} />
+    </section>
+  );
+}
+
+function CompletionRun({ apiBase, provider, model }: Omit<Props, "models"> & { model: string }) {
+  const t = useT();
+  const { locale } = useI18n();
+  const [busy, setBusy] = useState(false);
+  const [result, setResult] = useState<CompletionTestResult | null>(null);
+  const [error, setError] = useState<string | null>(null);
+  const flight = useRef<AbortController | null>(null);
+  useEffect(() => () => { flight.current?.abort(); flight.current = null; }, []);
+
+  async function run() {
+    if (!model || flight.current) return;
+    const controller = new AbortController();
+    flight.current = controller;
+    setBusy(true);
+    setResult(null);
+    setError(null);
+    try {
+      if (!await confirmAction({ message: t("pws.completion.confirm", { model, provider }),
+        confirmLabel: t("pws.completion.run"), signal: controller.signal, locale }) || controller.signal.aborted) return;
+      // The dashboard's installed fetch wrapper supplies session and CSRF headers, as for the connection test.
+      const response = await fetch(`${apiBase}/api/providers/test-completion?name=${encodeURIComponent(provider)}`, {
+        method: "POST", headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({ model, confirm: true }), signal: controller.signal,
+      });
+      const value = await readJsonOrThrow<CompletionTestResult>(response, t("pws.completion.failed"));
+      if (!value) throw new Error(t("pws.completion.failed"));
+      if (!controller.signal.aborted) setResult(value);
+    } catch (cause) {
+      if (!controller.signal.aborted) setError(cause instanceof Error ? cause.message : t("pws.completion.failed"));
+    } finally {
+      if (flight.current === controller) {
+        flight.current = null;
+        if (!controller.signal.aborted) setBusy(false);
+      }
+    }
+  }
+
+  return <>
+    <button type="button" className="btn btn-ghost btn-sm" disabled={!model || busy} onClick={() => void run()}>
+      {busy ? t("pws.testing") : t("pws.completion.run")}
+    </button>
+    {error && <p role="alert" className="pws-inline-error pws-completion-result">{error}</p>}
+    {result && <CompletionResult result={result} />}
+  </>;
+}
+
+function CompletionResult({ result }: { result: CompletionTestResult }) {
+  const t = useT();
+  if (!result.applicable) return <div role="status" className="pws-completion-result muted">
+    <p>{t("pws.completion.notApplicable")}: {t(`pws.completion.${result.reason}`)}</p>
+    {result.detail && <p>{t(`pws.completion.${result.detail}`)}</p>}
+    {result.adapter && <p><code>{result.adapter}</code></p>}
+  </div>;
+  return <div role="status" className="pws-completion-result" data-completion-test-state={result.ok ? "ok" : "failed"}>
+    <p className={result.ok ? "pws-status-ok" : "pws-status-warn"}>
+      {t(result.ok ? "pws.completion.ok" : "pws.completion.failed")}
+    </p>
+    <dl className="pws-kv">
+      <div className="pws-kv-row"><dt>{t("pws.completion.ttft")}</dt><dd>{result.ttftMs === null ? "—" : `${result.ttftMs} ms`}</dd></div>
+      <div className="pws-kv-row"><dt>{t("pws.completion.total")}</dt><dd>{result.totalLatencyMs} ms</dd></div>
+    </dl>
+    <p>{result.upstreamModel === null ? t("pws.completion.unreported", { model: result.wireModel })
+      : t("pws.completion.reported", { upstream: result.upstreamModel, model: result.wireModel })}</p>
+    <span className={`badge ${result.modelComparison === "match" ? "badge-green" : result.modelComparison === "mismatch" ? "badge-amber" : "badge-muted"}`}>
+      {t(`pws.completion.${result.modelComparison}`)}
+    </span>
+    {result.comparisonNote && <p className="muted">{t(`pws.completion.${result.comparisonNote}`)}</p>}
+    <p className="muted">{t("pws.completion.identity")}</p>
+    {result.error && <p role="alert" className="pws-inline-error"><code>{result.error.code}</code>
+      {result.error.status !== undefined && <> ({result.error.status})</>}: {result.error.message}</p>}
+  </div>;
+}

gui/src/components/provider-workspace/ProviderDetails.tsx

@@ -288,6 +288,7 @@ export default function ProviderDetails({
             preset={preset}
             apiBase={apiBase}
             connectionIdentity={connectionIdentity}
+            availableModels={availableModels}
             usageTotals={usageTotals}
             quotaReport={quotaReport}
             currentQuotaReading={currentQuotaReading}

gui/src/components/provider-workspace/ProviderOverview.tsx

@@ -15,6 +15,7 @@ import type { ProviderUpdatePatch, ProviderUpdateResult } from "./types";
 import ProviderCurrentQuota from "./ProviderCurrentQuota";
 import type { CatalogPreset } from "../provider-catalog/provider-presets";
 import ProviderSponsor from "./ProviderSponsor";
+import ProviderCompletionTest from "./ProviderCompletionTest";
 
 type ConnectionTestResult = {
   applicable?: boolean;
@@ -33,7 +34,7 @@ type ConnectionTestState = {
 
 export default function ProviderOverview({
   item, preset, usageTotals, quotaReport, currentQuotaReading, onRefreshQuota, oauthEmail, oauth,
-  apiBase, connectionIdentity,
+  apiBase, connectionIdentity, availableModels = [],
   onEditSettings, onViewUsage, onUpdateProvider,
   onCreateJevAuto,
   onReauthenticate, onCancelLogin, reauthBusy = false,
@@ -48,6 +49,7 @@ export default function ProviderOverview({
   /** Login state for OAuth summaries that carry no email (e.g. Cursor/Kimi). */
   oauth?: { loggedIn?: boolean };
   apiBase?: string;
+  availableModels?: string[];
   /** Opaque active credential identity used only to invalidate stale probe results. */
   connectionIdentity?: string;
   onEditSettings?: () => void;
@@ -195,6 +197,8 @@ export default function ProviderOverview({
             )}
           </div>
         )}
+        {apiBase && <ProviderCompletionTest key={connectionProbeKey} apiBase={apiBase} provider={item.name}
+          models={[...new Set([...(item.models ?? []), ...availableModels, ...(item.defaultModel ? [item.defaultModel] : [])])]} />}
         {onEditSettings && (
           <button type="button" className="link-btn pws-edit-settings-link" onClick={onEditSettings}>
             {t("pws.editSettings")}

gui/src/components/provider-workspace/provider-completion-test.css

@@ -0,0 +1,22 @@
+.pws-completion-test {
+  display: grid;
+  grid-template-columns: minmax(0, 1fr) auto;
+  align-items: start;
+  gap: var(--space-2);
+  margin-top: var(--space-4);
+}
+.pws-completion-test > h4,
+.pws-completion-result {
+  grid-column: 1 / -1;
+}
+.pws-completion-test > .custom-select {
+  min-width: 0;
+}
+.pws-completion-result {
+  overflow-wrap: anywhere;
+}
+@media (max-width: 640px) {
+  .pws-completion-test {
+    grid-template-columns: minmax(0, 1fr);
+  }
+}

gui/src/i18n/de.ts

@@ -6,6 +6,34 @@ import type { TKey } from "./en";
  * German i18n catalog, generated from en.ts. Must match the `TKey` set (compile-checked).
  */
 export const de: Record<TKey, string> = {
+  "pws.completion.title": "Modellantwort testen",
+  "pws.completion.model": "Zu testendes Modell",
+  "pws.completion.choose": "Modell auswählen",
+  "pws.completion.run": "Modell testen",
+  "pws.completion.confirm": "{model} bei {provider} testen? Dabei wird eine echte Anfrage mit maximal 32 Ausgabetokens gesendet. Dies verbraucht Kontingent und kann Kosten verursachen.",
+  "pws.completion.failed": "Modelltest fehlgeschlagen",
+  "pws.completion.ok": "Modelltest erfolgreich",
+  "pws.completion.notApplicable": "Test nicht anwendbar",
+  "pws.completion.ttft": "Zeit bis zum ersten Token",
+  "pws.completion.total": "Gesamtlatenz",
+  "pws.completion.reported": "Upstream meldete {upstream} für {model}",
+  "pws.completion.unreported": "Upstream meldete kein Modell für {model}",
+  "pws.completion.identity": "Die Modellmetadaten stammen vom Upstream-Anbieter und bestätigen nicht die Identität des tatsächlichen Modells.",
+  "pws.completion.match": "Übereinstimmung",
+  "pws.completion.mismatch": "Abweichung",
+  "pws.completion.unknown": "Unbekannt",
+  "pws.completion.missing_metadata": "Die Upstream-Antwort enthält keine Modellmetadaten.",
+  "pws.completion.conflicting_metadata": "Die Upstream-Antwort enthält widersprüchliche Modellmetadaten.",
+  "pws.completion.disabled": "Dieser Anbieter ist deaktiviert.",
+  "pws.completion.forward_auth": "Weitergeleitete Authentifizierung wird nicht unterstützt.",
+  "pws.completion.oauth_auth": "OAuth-Authentifizierung wird nicht unterstützt.",
+  "pws.completion.unsupported_adapter": "Dieser Adapter wird nicht unterstützt.",
+  "pws.completion.unsupported_transport": "Diese Transportkonfiguration wird nicht unterstützt.",
+  "pws.completion.tls_profile": "Ein eigenes TLS-Profil ist konfiguriert.",
+  "pws.completion.upstream_http_version": "Eine Upstream-HTTP-Version ist konfiguriert.",
+  "pws.completion.request_pacing": "Die Anfragedrosselung ist aktiviert.",
+  "pws.completion.custom_executor_egress": "Ein eigener Fetch-Executor mit Anbieter-Proxy kann nicht beibehalten werden.",
+
   "reasoningRetention.advanced": "Erweiterte Einstellungen",
   "reasoningRetention.title": "Aufbewahrung von Denktext",
   "reasoningRetention.enable": "Aufbewahrung von Denktext aktivieren",

gui/src/i18n/en.ts

@@ -7,6 +7,34 @@ import { desktopCompatibilityCopy } from "./desktop-compatibility-copy";
  * `{var}` are plain interpolations.
  */
 export const en = {
+  "pws.completion.title": "Model completion test",
+  "pws.completion.model": "Model to test",
+  "pws.completion.choose": "Choose a model",
+  "pws.completion.run": "Test model",
+  "pws.completion.confirm": "Test {model} on {provider}? This sends a real completion request with an output cap of 32 tokens. It consumes quota and may incur costs.",
+  "pws.completion.failed": "Model test failed",
+  "pws.completion.ok": "Model test succeeded",
+  "pws.completion.notApplicable": "Test not applicable",
+  "pws.completion.ttft": "Time to first token",
+  "pws.completion.total": "Total latency",
+  "pws.completion.reported": "Upstream reported {upstream} for {model}",
+  "pws.completion.unreported": "Upstream did not report a model for {model}",
+  "pws.completion.identity": "Model metadata is reported by the upstream provider and does not verify the actual model's identity.",
+  "pws.completion.match": "Match",
+  "pws.completion.mismatch": "Mismatch",
+  "pws.completion.unknown": "Unknown",
+  "pws.completion.missing_metadata": "The upstream response contains no model metadata.",
+  "pws.completion.conflicting_metadata": "The upstream response reports conflicting model metadata.",
+  "pws.completion.disabled": "This provider is disabled.",
+  "pws.completion.forward_auth": "Forwarded authentication is not supported.",
+  "pws.completion.oauth_auth": "OAuth authentication is not supported.",
+  "pws.completion.unsupported_adapter": "This adapter is not supported.",
+  "pws.completion.unsupported_transport": "This transport configuration is not supported.",
+  "pws.completion.tls_profile": "A custom TLS profile is configured.",
+  "pws.completion.upstream_http_version": "An upstream HTTP version is configured.",
+  "pws.completion.request_pacing": "Request pacing is enabled.",
+  "pws.completion.custom_executor_egress": "A custom fetch executor with a provider proxy cannot be preserved.",
+
   "reasoningRetention.advanced": "Advanced settings",
   "reasoningRetention.title": "Reasoning retention",
   "reasoningRetention.enable": "Enable reasoning retention",