Skip to content

fix: maintain typed auth clients as remote source headers change - #8181

Merged
danielkov merged 9 commits into
mainfrom
fix/remote-header-typed-maintenance
Oct 28, 2026
Merged

danielkov merged 9 commits into
mainfrom
fix/remote-header-typed-maintenance

Conversation

@danielkov

@danielkov danielkov commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A credential source's typed auth clients, server links and copied credential headers now stay in line as the source and its servers are written, inside the same transaction as the write. Before this, only the organization-wide authclientbackfill run maintained them.

Covered writes: remoteMcp.createServerHeader, updateServerHeader, deleteServerHeader, remoteMcp.deleteServer, tunneledMcp.deleteServer, and the creation, backend switch and deletion of an MCP server.

Motivation

Two gaps, both leaving records that no longer match the configuration:

  • A remote source's Authorization header got no typed client, no header copy and no server link until a run came along, and deleting the header left its client behind with no tombstone.
  • The credential headers copied onto the auth clients of upstream OAuth and EMA clients were only maintained by the run. The serving path loads those copies and sends them with the upstream request, so a deleted header could still be sent, and a rotated one sent with its old value, until the next run.

Tunneled sources had no typed reconciliation at all: deleting one left its derived clients, links and copies live.

Impact

Correctness and freshness of the records the upstream serving path reads. No change to how a request is authorized or to which client serves it. Administrator-created clients, administrator detachments, credential choices and auth grants are untouched, and every path is scoped to the writing organization and project. A write whose evidence changes under it rolls back rather than committing records that do not match.

Technical details

The sync reuses the backfill's planner rather than restating it

The per-client copy decisions were extracted from copyOntoOAuthClients and copyOntoEMAClients into copyOntoOAuthClient and copyOntoEMAClient, which the run and the sync both call. ListOAuthClientSources gained an optional client-id filter so the sync reads one client's sources instead of the organization's. A test asserts that a backfill run straight after a sync reports no pending clients, links, copies, retirements or gaps.

A client is decided from all of its servers' sources

The rule that refuses a client shared across sources with credential headers reads every source its servers sit on, so the sync reads them all for the clients it re-decides — scoped to those clients, never the organization. A client left with no live server on any source carries no copies, so its own are retired.

A server that leaves every credential source

Switching a server to a toolset backend, or deleting a server that had none, leaves no source to re-decide its clients from. SyncServerDerivedCopies re-decides them from the server itself for those cases. It takes no lock: every copy write repeats the ownership rule it was decided from, so a concurrent change loses the write rather than overwriting it.

Tunneled sources

SyncTunneledSource mirrors SyncRemoteSource. No management API writes a tunneled source's headers and its deletion leaves them on the tombstoned row, so the source's liveness is what says they are gone. No tunneled header API was added.

Lock order is source-then-server everywhere

The sync writes to a source's mcp_servers rows and to their links, while a source's own writers lock the source first. The shared MCP server creation command, updateMcpServer and deleteMcpServer take the source lock up front and reject a concurrent backend switch as a retryable conflict.

The creation command also settles each source's liveness and project ownership under that lock, for both kinds. It was previously checked before any lock, so a deletion committing while the create waited could leave a server in front of a tombstoned source. LockRemoteSource and LockTunneledSource still accept a tombstoned source, because the update and delete paths have to lock one to retire its clients.

Association writes

Associating an upstream OAuth client with a server on a second source makes the client shared, and the ownership rule then says it carries neither source's credential headers — while the first source's copies are still on it. ListAuthClientHeaders loads those copies by auth client id and organization alone and revalidates no source ownership, and withoutDeniedSources only strips forwarded headers from refused request headers, so the first source's secret went to the second source's upstream until the next backfill run. Detaching the server whose source lent the headers leaves the mirror state.

All three writers of an association now re-decide the copies in their own transaction: the identity commit, the catalog identity-provider attachment, and the dashboard detach. The mechanism predates this PR, but this PR records copies at header-write time rather than only at backfill time, so the stale state was reachable in more windows than before.

Lock behaviour of the copies pass

It locks only the header copy rows it writes — no source, server, link or upstream client row — and every caller runs it after its own writes, so the copy rows are the last locks the transaction takes. Two orders needed correcting for that to hold: a source sync marks servers before writing copies, and MCP server deletion runs its syncs after the issuer cascade, which row-locks the remote_session_clients an identity commit writes before reaching the same copies.

Consistency under a concurrent source or header write comes from the writes rather than a lock: an added or updated copy repeats the ownership rule it was decided from, so an invalidating change makes the write affect no row; a retired copy is unconditional, so the decision to stop sending a credential always lands. A copy wrongly retired by a racing writer is restored by the next source write or backfill run.

@danielkov
danielkov requested a review from a team as a code owner October 27, 2026 22:23
@changeset-bot

changeset-bot Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 675f7ca

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
server Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@blacksmith-sh

This comment has been minimized.

@blacksmith-sh

This comment has been minimized.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

danielkov and others added 6 commits October 28, 2026 00:08
A remote MCP source's credential headers only got their typed upstream auth
client, header copies and server links from the organization-wide backfill,
so a source configured between runs had none. Keep them in line inside the
transaction of the write that changes them: the source's header create,
update and delete, the source's own deletion, and the creation, backend
switch and deletion of an MCP server in front of it.

The sync reuses the backfill's derivation for a single source, so it records
exactly what a backfill run would. It leaves administrator-created clients,
administrator detachments, credential choices and grants alone, and leaves a
source without an Authorization header to the backfill, whose decision needs
the organization's upstream OAuth and EMA clients.

Every path now locks the source before any of its MCP servers, so the sync's
writes to a source's servers never wait on a server lock in the opposite
order to the source's own writers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A source without an Authorization header keeps the client the backfill
recorded for it, but kept every backfilled link to that client, including
one from a server that had since switched to another backend. That server
would have gone on sending the old source's credential headers. Keep only
the links of servers still backed by the source, and retire the client with
them once no server is left on it, as the backfill decides.

Also lock the remote source before inserting the server row in the shared
creation command, so no path reaches the sync holding a server lock first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s server

The shared MCP server creation command checked the source's liveness without
holding its row lock, so a source a concurrent deletion tombstoned while the
check waited could still get a server committed onto it. Take the live,
project-scoped source lock there instead, which also gives the lock the auth
client sync needs. The two callers that took that lock themselves now leave
it to the command, so the invariant has one home.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ange

The serving path loads an auth client's header copies and sends them with the
upstream request, so a copy the configuration no longer yields is a deleted
header still going out, or a rotated one going out with its old value. Only
the organization-wide backfill maintained the copies on upstream OAuth and
EMA clients, so a source written between runs kept the stale ones.

Each write that changes them now re-decides them in its own transaction,
through the backfill's own planner so a sync and a run record the same
copies. A client is decided from every source its servers sit on, not just
the one being written, because the rule refusing a client shared across
sources reads them all; a client left with no live server on any source
carries none.

Tunneled sources get the same treatment as remote ones. No management API
writes a tunneled source's headers, so its own lifecycle is what has to
retire the records: its deletion, and servers arriving on or leaving it.

A server that leaves every credential source is reached only through itself,
so server updates and deletions re-decide their own clients' copies too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removed as unused on main while this branch was open. The ema copy test
needs it: a client whose last live server is gone must stop carrying its
source's credentials, and tombstoning the server is how the test gets there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@danielkov
danielkov force-pushed the fix/remote-header-typed-maintenance branch from ba043fb to 321b569 Compare October 28, 2026 00:15

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

danielkov and others added 2 commits October 28, 2026 00:31
…its server

The shared creation command settled a remote source's liveness under its row
lock but took the tunneled lock without one, and that lock accepts a
tombstoned source so the retirement paths can use it. A deletion committing
while the create waited therefore let a server be committed onto a deleted
tunneled source. Both kinds now verify liveness and project ownership under
the lock, through one helper.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… them

Associating an upstream OAuth client with a server on a second source makes
the client shared, and the ownership rule then says it carries neither
source's credential headers. The copies the first source left stayed on it,
and the serving path loads them by auth client id and organization alone and
revalidates no source ownership, so the first source's secret went to the
second source's upstream until the next backfill run. Detaching the server
whose source lent the headers leaves the mirror state.

All three writers of an association now re-decide the copies in their own
transaction: the identity commit, the catalog identity-provider attachment
and the dashboard detach.

The copies pass locks only the header copy rows it writes, so each caller
runs it after its own writes and the copy rows are the last locks the
transaction takes. Two orders needed correcting for that to hold: a source
sync now marks servers before writing copies, and MCP server deletion runs
its syncs after the issuer cascade, which row-locks the remote session
clients an identity commit writes before reaching the same copies.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 9 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

The test configured the arriving server without the client ever reaching the
header-bearing source, so the copies were retired because the client's only
source carried no headers, not because it became shared. Associate the sole
server first and assert it keeps its association, so the commit really does
make the client span two sources.
@danielkov danielkov added the review:bypass Merge without human review approval. Required status checks still apply. label Oct 28, 2026
@danielkov
danielkov enabled auto-merge October 28, 2026 11:04

@gram-bot gram-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved: review:bypass label applied by @danielkov. Required status checks still gate this merge.

@danielkov
danielkov added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit 7164f6c Oct 28, 2026
55 checks passed
@danielkov
danielkov deleted the fix/remote-header-typed-maintenance branch October 28, 2026 11:23
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

review:bypass Merge without human review approval. Required status checks still apply.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant