Skip to content

fix(health): stop judging header-auth servers by a stale OAuth token record - #1262

Merged
Dumbris merged 3 commits into
mainfrom
fix/1172-stale-oauth-token-health
Sep 12, 2026
Merged

fix(health): stop judging header-auth servers by a stale OAuth token record#1262
Dumbris merged 3 commits into
mainfrom
fix/1172-stale-oauth-token-health

Conversation

@Dumbris

@Dumbris Dumbris commented Sep 12, 2026

Copy link
Copy Markdown
Member

Problem

After switching an upstream from OAuth to a static Authorization header ("oauth": null in the config), the server keeps working — connected, tools served — but /api/v1/servers reports it as unhealthy / "Token expired" / action: login indefinitely. After POST /api/v1/servers/<name>/logout + restart it flipped to "Refresh token expired" instead. Reporter confirmed on v0.63.0 and v0.64.0 with a natural A/B across ten upstreams of the same provider: the eight with a leftover oauth_tokens record were red, the two without were healthy.

Root cause

Two records outlive the config change and both were treated as evidence:

  1. The stored token. Runtime.GetAllServers looks up the oauth_tokens bucket for every URL-addressed server (so that autodiscovery OAuth servers, which have no oauth block, still show their token state). Nothing ever removed the reporter's records: the config-load path only clears OAuth state when an explicit oauth block changes (config.OAuthConfigChanged), and an autodiscovery server goes nil → nil. The dead record synthesised an oauth object with token_expires_at in the past, which set oauth_status: "expired", and the health calculator's OAuth branch reported "Token expired".
  2. The refresh schedule. RefreshManager.Start builds an in-memory schedule from every stored token; a fully-expired one is parked in RefreshStateFailed. Once the token stopped driving the OAuth branch (or after logout removed it), that schedule surfaced as "Refresh token expired". TriggerOAuthLogout cleared the token but never dropped the schedule (server removal already did).

Fix

The principle: a token record is only evidence about a server that can actually be using it. OAuth and a static Authorization header are mutually exclusive on the wire — the OAuth transport populates that very header, and the headers-auth strategy runs before OAuth is ever attempted — so a server with no oauth block and a static Authorization header has stopped using OAuth. The live connection has the final say: if the configured header were rejected and the OAuth strategy rescued the connection with the stored token, the token is in play and is reported as for any OAuth server.

  • config.ServerConfig.HasStaticAuthorizationHeader() — case-insensitive name, non-blank value. Only Authorization counts; X-API-Key-style headers do not contradict OAuth, so they are deliberately not treated as evidence.
  • core.Client.AuthStrategy() — the two identical HTTP/SSE strategy loops are folded into runAuthStrategies, which records the winner ("headers", "no-auth", AuthStrategyOAuth); reset at the start of each connect attempt and on disconnect. Read atomically because Connect holds c.mu for the whole attempt (an OAuth flow can take minutes) and the projection must not block on it. managed.Client.ConnectedWithOAuth() exposes it.
  • Runtime.StoredOAuthTokenInPlay(name, cfg) — the single decision: cfg == nil || cfg.OAuth != nil || !HasStaticAuthorizationHeader() → true; otherwise only if the live connection was made with OAuth.
  • Runtime.GetAllServers skips the stored-token lookup when the token is not in play; the later IsOAuthError(LastError) branch is untouched, so a header-auth server that genuinely gets a 401 still receives the login CTA.
  • Runtime.HealthRefreshState(name, cfg) wraps RefreshManager.GetRefreshState with the same decision, and all three CalculateHealth call sites (REST via Runtime.GetAllServers, the Go tray via Server.GetAllServers, the MCP upstream_servers list) read refresh state through it. The calculator itself is unchanged.
  • TriggerOAuthLogout now calls refreshManager.OnTokenCleared, as server removal already did.

Deletion decision — the stale record is ignored, not deleted. Explicit-OAuth → null already clears state on config load (OAuthConfigChanged, unchanged). For the reporter's autodiscovery case there is no "OAuth removed" transition to hook — the only new signal is the header — and deleting a credential the user obtained through an interactive login on the strength of a config edit (possibly transient, possibly tool-written) is a destructive step the projection does not need. Once ignored the record is inert: logout still removes it, and the existing startup orphan cleanup removes it when the server is deleted. If a user later drops the header, the token is consulted again exactly as before.

Testing

TDD: every test below was written first and observed failing (three of the runtime tests failed with "Token expired" / "unhealthy" / a surviving schedule; the scope-control test passed pre-fix, as intended).

  • internal/runtime/stale_oauth_token_health_test.go
    • TestGetAllServers_StaticAuthHeaderIgnoresStaleOAuthToken — the reporter's case: stale expired record + Authorization header + connected → healthy, action: none, oauth: null, authenticated: false, no oauth_status/token_expires_at.
    • TestGetAllServers_StaticAuthHeaderIgnoresStaleRefreshState — same, with RefreshManager.Start having parked the record in Failed → not "Refresh token expired".
    • TestGetAllServers_AutodiscoveryOAuthStillReportsExpiredToken — scope control: no Authorization header (a non-credential header only) → still "Token expired" / login.
    • TestHealthRefreshState_SharedSeam — the seam all three surfaces use: withheld for header-auth, reported for autodiscovery, explicit oauth block wins, nil config keeps history.
    • TestTriggerOAuthLogout_ClearsRefreshSchedule — logout removes the token and the schedule.
  • internal/upstream/core/auth_strategy_test.gorunAuthStrategies records the winner (headers / OAuth-rescue / no-auth), leaves it empty on exhaustion, and still returns a transport error unwrapped without reaching later strategies.
  • internal/config/static_authorization_header_test.go — predicate table incl. case-insensitivity, blank value, X-API-Key, Proxy-Authorization, nil receiver.
  • go test -race on internal/runtime/... internal/health/... internal/config/... internal/upstream/... and internal/server (CI skip regex) — green. go build ./... green. golangci-lint (v2, .github/.golangci.yml) — 0 issues on touched packages.
  • opencode cross-review (github-copilot/gpt-6-astra), 3 rounds: round 1 found the OAuth-rescue edge (fixed via live strategy); round 2 found that a calculator-level guard hid genuine autodiscovery refresh failures on the tray/MCP surfaces (fixed by moving the decision into the runtime seam and routing all three call sites through it); round 3 VERDICT: CLEAN.

Closes #1172

🤖 Generated with Claude Code

Dumbris and others added 3 commits September 12, 2026 08:04
…record

A server that once logged in with OAuth (autodiscovery, no `oauth` block)
and was then switched to a static `Authorization` header keeps its old
record in the oauth_tokens bucket: nothing removes it because the config
never carried an OAuth block whose removal could have cleared it. The
server-list projection built an `oauth` object from that dead record and,
once its expiry passed, health reported the connected, tool-serving server
as unhealthy / "Token expired" / login forever. After logout the in-memory
refresh schedule outlived the token and the same server flipped to
"Refresh token expired".

- config: add ServerConfig.HasStaticAuthorizationHeader (case-insensitive
  name, non-blank value). OAuth populates that very header and the
  headers-auth strategy runs first, so the two are mutually exclusive.
- runtime: a server with no `oauth` block and a static Authorization
  header no longer consults the stored token for its projection. The
  record is left alone (logout still removes it); it is simply not
  evidence for such a server.
- health: refresh state only applies when OAuthRequired — it is a
  property of the token being refreshed.
- runtime: TriggerOAuthLogout drops the refresh schedule alongside the
  token, as server removal already did.

Closes #1172

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review round 1 (opencode/astra): a rejected static Authorization header
can fall through to the OAuth strategy, which then authenticates the
connection with the stored token — and the header-based gate would have
hidden that token's state. Record the strategy that won each HTTP/SSE
connect attempt (core.Client.AuthStrategy, via a shared runAuthStrategies
loop replacing the two identical ones), expose it as
managed.Client.ConnectedWithOAuth, and let the runtime consult the stored
token whenever the live connection was made with OAuth.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ot the calculator

Review round 2 (opencode/astra): guarding the calculator's refresh-state
branch on OAuthRequired hid genuine refresh failures on the two surfaces
(Server.GetAllServers for the tray, the MCP upstream_servers list) that
only learn OAuthRequired from an explicit `oauth` block — an autodiscovery
OAuth server with a failed refresh would have read healthy there.

Revert the calculator change and move the decision to where the evidence
is: Runtime.StoredOAuthTokenInPlay(name, cfg) answers "is the oauth_tokens
record (and the RefreshManager schedule built from it) evidence about
this server", and Runtime.HealthRefreshState wraps GetRefreshState with
it. All three CalculateHealth call sites now read refresh state through
that seam, so a header-authenticated server's stale schedule is withheld
everywhere and an autodiscovery server's genuine failure is kept.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 18998ac
Status: ✅  Deploy successful!
Preview URL: https://76cdb5e5.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-1172-stale-oauth-token-h.mcpproxy-docs.pages.dev

View logs

@github-actions

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/1172-stale-oauth-token-health

Available Artifacts

  • archive-darwin-amd64 (29 MB)
  • archive-darwin-arm64 (27 MB)
  • archive-linux-amd64 (17 MB)
  • archive-linux-arm64 (16 MB)
  • archive-windows-amd64 (29 MB)
  • archive-windows-arm64 (26 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (24 MB)
  • installer-dmg-darwin-arm64 (21 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 34675616136 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 65.51724% with 20 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/runtime/runtime.go 68.18% 5 Missing and 2 partials ⚠️
internal/server/mcp.go 0.00% 4 Missing and 1 partial ⚠️
internal/server/server.go 0.00% 5 Missing ⚠️
internal/upstream/managed/client.go 0.00% 2 Missing ⚠️
internal/upstream/core/connection_http.go 93.33% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@Dumbris
Dumbris merged commit 527097b into main Sep 12, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Server keeps showing "Token expired" after switching it from OAuth to a header token

2 participants