fix(link): recover preflight deadlines with bounded diagnostics

docs-site/src/content/docs/guides/remote-link.md

@@ -99,6 +99,16 @@ When a step fails, the dashboard shows the reason and, when SSH reported one, a
 - **The remote computer did not report an OpenCodex version**: `ocx --version` on that computer printed something else, for example the usage text of an unsupported Windows install.
 - **Requests fail with `home_unreachable` while the Home is down**: a Child never serves requests from its own providers. When the tunnel is down before a request is forwarded, the Child answers a retryable 503 and sends nothing to the Home. If the tunnel drops after forwarding started, the answer is also `home_unreachable`, but the Home may already have received the request. A failed link relay authentication answers `link_relay_auth_failed` instead: upgrade the Home and Child or re-link. Whether GOAT may move that device's work to its own sign-ins is the per-device local fallback permission on the Home's **Remote Link** page, or `ocx fallback local on --device <alias>`; see [Fallback off](/guides/fallback-off/).
 
+### Retryable 503 diagnostics
+
+A relay-generated `503` includes a server-generated `x-opencodex-link-request-id`. Use it to correlate a failure with local evidence under the Child's OpenCodex configuration home: `link/diagnostics.jsonl` and one rotated backup, `link/diagnostics.jsonl.1`. Each file is capped at 256 KiB, with private permissions (directory 0700 and files 0600 on POSIX; private ACLs on Windows). Records contain only fixed event/outcome/state categories, generated correlation IDs, numeric timing/generation/attempt metadata and timestamps, never secrets, request bodies, headers, URLs, account identifiers or raw errors. Each event/outcome pair is limited to one record per second; the next record includes a count of suppressed events, so an absent record does not prove an absent failure. A backward wall-clock adjustment starts a new rate-limit window, including when the writer queue is full, and preserves the suppression count. An asynchronous writer keeps at most 56 pending records, counts overflow as suppression, and yields while Windows ACL work runs. Logging is best effort and cannot change whether a request is admitted. Graceful signal shutdown and standalone recycling wait up to one second for queued diagnostics before exiting; stalled storage, forced termination and power loss can still lose records.
+
+`owner_lookup` records refused ownership checks; routine successful checks are silent. `owner_lookup.elapsedMs` measures the owner lookup itself. For `auth_timeout`, `relay_retry` and `relay_failure` report the actual transport-attempt duration in `elapsedMs` (normally about five seconds). A `relay_recovered` event with outcome `recovered` is emitted only after a retry receives an upstream response with `upstream.ok` (HTTP 2xx); it does not prove the streamed body completed. Rate limiting makes request-ID correlation sampled evidence, not a guaranteed trace of every failed request.
+
+Before sending data, OpenCodex allows one shared preflight retry for either an owner-lookup deadline or a specifically identified five-second authentication deadline. The retry obtains fresh owner proof where required and a fresh physical socket, then authenticates that socket before sending the link credential or body. It does not reuse cached owner approval, bypass security checks, restart SSH or OpenCodex, or replay a dispatched request body. The two-second owner-lookup and five-second authentication deadlines are unchanged for each attempt; other failures do not qualify for this preflight retry.
+
+This can recover these transient deadline paths and helps diagnose future failures. It does not establish the historical cause of [#7297](https://github.com/lidge-jun/opencodex/issues/7297).
+
 ## Security
 
 The Child uses the Home computer's providers and provider credentials through the link. The Home creates a separate link key for each Child; removing the link revokes that key. On the Child, the key stays inside OpenCodex: credentials that Codex or Claude Code send there are not forwarded to the Home, including Bearer, Azure `api-key`, Anthropic-compatible `x-api-key`, and Google `x-goog-api-key` forms. Any program on the Child that reaches `127.0.0.1:<port>` uses the Home without a key, the same local trust a standalone install gives. Web pages from other sites are refused. Compare the host fingerprint before confirmation so a wrong machine or changed host key is not accepted by mistake. Dashboard sessions issued from a Tailscale identity cannot manage machine links.

scripts/test-layout/layout-clients.json

@@ -1,4 +1,5 @@
 {
+  "link-diagnostics.test.ts": "clients",
   "gentle-shell-path.test.ts": "clients",
   "gentle-shell-client.test.ts": "clients",
   "droid-managed-reasoning-defaults.test.ts": "clients",

src/client/link-relay-transport.ts

@@ -27,6 +27,14 @@ export class LinkRelayAuthenticationError extends Error {
   }
 }
 
+/** Only the credential-free preflight deadline expired; no data request has begun. */
+export class LinkRelayAuthTimeoutError extends Error {
+  constructor() {
+    super("link relay authentication timed out");
+    this.name = "LinkRelayAuthTimeoutError";
+  }
+}
+
 /** One physical connection, ever. A closed keep-alive socket cannot trigger a silent reconnect. */
 class SingleConnectionAgent extends Agent {
   socket: Socket | undefined;
@@ -153,8 +161,17 @@ async function responseOnConnection(incoming: IncomingMessage, agent: SingleConn
 
 let peerReport: Record<string, string> | undefined;
 
-/** Authenticate and stream one request on the SAME socket; never fall back to an ordinary fetch. */
-export async function fetchBoundLinkRelay(target: BoundLinkRelayTarget, destination: string, init: RequestInit): Promise<Response> {
+/**
+ * Authenticate and stream one request on the SAME socket; never fall back to an ordinary fetch.
+ * LinkRelayAuthTimeoutError certifies that only the credential-free preflight expired before
+ * data-request construction. Caller aborts and failures after that boundary keep their own errors.
+ */
+export async function fetchBoundLinkRelay(
+  target: BoundLinkRelayTarget, destination: string, init: RequestInit,
+  options: { authTimeoutMs?: number } = {},
+): Promise<Response> {
+  const authTimeoutMs = options.authTimeoutMs ?? LINK_RELAY_AUTH_TIMEOUT_MS;
+  if (!Number.isFinite(authTimeoutMs) || authTimeoutMs <= 0) throw new RangeError("invalid link relay auth timeout");
   if (!target.linkId || !target.apiKeyId) throw new LinkRelayAuthenticationError();
   const url = new URL(destination);
   if (url.origin !== `http://127.0.0.1:${target.tunnelPort}`) throw new LinkRelayAuthenticationError();
@@ -163,7 +180,8 @@ export async function fetchBoundLinkRelay(target: BoundLinkRelayTarget, destinat
   probe.search = new URLSearchParams({ version: LINK_RELAY_AUTH_VERSION, key: target.apiKeyId,
     link: target.linkId, nonce: challenge.nonce, proof: challenge.caller }).toString();
   const agent = new SingleConnectionAgent({ keepAlive: true, maxSockets: 1, maxFreeSockets: 1 });
-  const deadline = clearableDeadline(LINK_RELAY_AUTH_TIMEOUT_MS, init.signal ?? undefined);
+  const deadline = clearableDeadline(authTimeoutMs, init.signal ?? undefined);
+  let dataRequestStarted = false;
   try {
     // The challenge also carries this Child's version and features for the Home's link status.
     peerReport ??= linkPeerReportHeaders(packageVersion("0.0.0"), buildPeerFeatures());
@@ -177,15 +195,20 @@ export async function fetchBoundLinkRelay(target: BoundLinkRelayTarget, destinat
     for await (const chunk of response) {
       if (chunk.length) throw new LinkRelayAuthenticationError();
     }
+    deadline.signal.throwIfAborted();
     deadline.clear();
     init.signal?.throwIfAborted();
     if (!socket || socket.destroyed || socket !== agent.socket) throw new LinkRelayAuthenticationError();
     const headers = new Headers(init.headers);
     headers.set(LINK_RELAY_SESSION_HEADER, challenge.nonce);
+    dataRequestStarted = true;
     const incoming = await requestOnConnection(url, { ...init, headers }, agent, socket);
     return await responseOnConnection(incoming, agent, init.method ?? "GET");
   } catch (error) {
+    const authDeadlineExpired = !dataRequestStarted && deadline.didExpire()
+      && !(error instanceof LinkRelayAuthenticationError);
     await agent.close();
+    if (authDeadlineExpired && !init.signal?.aborted) throw new LinkRelayAuthTimeoutError();
     throw error;
   } finally { deadline.clear(); }
 }

src/client/link-relay.ts

@@ -1,11 +1,13 @@
-import { fetchBoundLinkRelay, LinkRelayAuthenticationError } from "./link-relay-transport";
+import { randomUUID } from "node:crypto";
+import { fetchBoundLinkRelay, LinkRelayAuthenticationError, LinkRelayAuthTimeoutError } from "./link-relay-transport";
 import { homeUnreachableBody } from "./home-unreachable";
 import {
   boundedRelayResponseStream,
   filterRelayHeaders,
   headersWithinLimit,
   validateHubRelayRequestHeaders,
 } from "./hub-relay";
+import type { LinkDiagnosticRecord, LinkDiagnosticSink } from "../link/diagnostics";
 import { linkRouteAllowed } from "../link/routes";
 import { isLinkPort } from "../link/ports";
 import { resolveInboundBodyLimitBytes } from "../server/request-decompress";
@@ -26,14 +28,23 @@ export interface LinkRelayClock {
   clearTimeout: typeof clearTimeout;
 }
 
+/** A fresh admission verdict plus optional diagnostic evidence; metadata never grants admission. */
+export interface LinkAdmissionResult {
+  kind: "match" | "mismatch" | "lookup_error" | "deadline" | "stale" | "identity_unknown" | "identity_mismatch" | "unavailable";
+  elapsedMs?: number;
+  generation?: number;
+  state?: string;
+  checkId?: string;
+}
+
 /**
  * The Child's tunnel as the relay sees it (the client link supervisor implements it). A request
  * waits on it only while the tunnel is being (re)established. Every relayed fetch waits for a
  * fresh bounded asynchronous ownership proof, including after a reconnect or refused retry.
  */
 export interface LinkTunnelGate {
-  /** True only while a keyed probe has established this live tunnel. */
-  connected(): boolean | Promise<boolean>;
+  /** Admission and its evidence are returned together; only match permits a fetch. */
+  connected(context?: { requestId?: string }): LinkAdmissionResult | Promise<LinkAdmissionResult>;
   /** True while the tunnel is connecting or reconnecting. */
   pending(): boolean;
   /**
@@ -51,12 +62,13 @@ export interface LinkTunnelGate {
  * supervisor, and a relay with no gate at all still refuses every request.
  */
 export const HOME_INITIATED_LINK_TUNNEL: LinkTunnelGate = {
-  connected: () => true,
+  connected: () => ({ kind: "match" }),
   pending: () => false,
   waitForConnected: async () => false,
 };
 
 export interface LinkRelayDeps {
+  diagnostic?: LinkDiagnosticSink;
   /** Trusted complete-send test seam. Production always uses the connection-bound transport. */
   fetchImpl?: typeof fetch;
   clock?: LinkRelayClock;
@@ -136,6 +148,7 @@ function relayAuthenticationFailed(error: string): Response {
   });
 }
 
+/** Preserve path/query on a fixed IPv4 loopback origin; throw when the tunnel port is invalid. */
 export function linkRelayDestination(url: URL, target: Pick<LinkRelayTarget, "tunnelPort">): string {
   if (!isLinkPort(target.tunnelPort)) {
     throw new RangeError("invalid link tunnel port");
@@ -180,6 +193,7 @@ function linkRequestHeaders(
   return headers;
 }
 
+/** Drop hop-by-hop and stale length/encoding headers before exposing the decoded response. */
 export function sanitizeLinkResponseHeaders(source: Headers): Headers {
   const connectionNamed = new Set((source.get("connection") ?? "")
     .split(",").map(value => value.trim().toLowerCase()).filter(Boolean));
@@ -328,6 +342,11 @@ function idleBoundedStream(
   });
 }
 
+/**
+ * Admit an allowlisted route and stream it through the authenticated tunnel with bounded bodies.
+ * Owner/auth preflight deadlines share one recovery attempt before data dispatch; an ambiguous
+ * post-dispatch failure never earns that recovery. Diagnostic failures cannot change admission.
+ */
 export async function relayLinkDataRequest(
   req: Request,
   target: LinkRelayTarget,
@@ -352,6 +371,16 @@ export async function relayLinkDataRequest(
     return jsonError(431, "link relay request headers too large");
   }
 
+  const requestId = randomUUID();
+  let admission: LinkAdmissionResult = { kind: "unavailable" };
+  let recoveryAttempts = 0;
+  const diagnostic = (event: LinkDiagnosticRecord["event"], outcome: LinkDiagnosticRecord["outcome"], elapsedMs?: number) => {
+    try { deps.diagnostic?.({ event, outcome, requestId,
+      checkId: admission.checkId, generation: admission.generation,
+      state: admission.state, elapsedMs: admission.elapsedMs,
+      ...(elapsedMs !== undefined ? { elapsedMs } : {}), attempt: recoveryAttempts }); }
+    catch { /* diagnostics never alter admission */ }
+  };
   const relayAbort = new AbortController();
   const clock = deps.clock ?? defaultClock;
   let headerTimer: ReturnType<typeof setTimeout> | undefined = clock.setTimeout(() => {
@@ -390,20 +419,45 @@ export async function relayLinkDataRequest(
     const remaining = holdUntil - now;
     return remaining > 0 && await gate.waitForConnected(remaining, relayAbort.signal);
   };
-  const tunnelUnavailable = async (): Promise<Response> => {
+  const tunnelUnavailable = async (
+    reason: LinkDiagnosticRecord["outcome"] = admission.kind === "match" ? "unavailable" : admission.kind,
+    message = "link tunnel unavailable",
+    elapsedMs?: number,
+  ): Promise<Response> => {
+    diagnostic("relay_failure", relayAbort.signal.aborted
+      ? req.signal.aborted ? "client_abort" : "header_timeout" : reason, elapsedMs);
     cleanup();
     try { await body?.cancel(); } catch { /* best effort */ }
-    return homeUnreachable("link tunnel unavailable");
+    const response = reason === "auth_failed" ? relayAuthenticationFailed(message) : homeUnreachable(message);
+    response.headers.set("x-opencodex-link-request-id", requestId);
+    return response;
+  };
+  const retryPreflight = (reason: "deadline" | "auth_timeout", elapsedMs?: number): boolean => {
+    if (recoveryAttempts >= 1 || relayAbort.signal.aborted || bodyOverflow || bodyCancelled || body?.locked) return false;
+    recoveryAttempts += 1;
+    diagnostic("relay_retry", reason, elapsedMs);
+    return true;
+  };
+  const checkConnected = async (): Promise<boolean> => {
+    admission = tunnel ? await tunnel.connected({ requestId }) : { kind: "unavailable" };
+    return admission.kind === "match";
   };
   const readyForFetch = async (): Promise<boolean> => {
-    if (!tunnel) return false;
-    if (await tunnel.connected()) return true;
-    return tunnel.pending() && await holdForTunnel(tunnel) && await tunnel.connected();
+    if (!tunnel || relayAbort.signal.aborted) return false;
+    if (await checkConnected()) return !relayAbort.signal.aborted;
+    // No fetch or body read has occurred. A deadline proves neither a foreign owner nor a
+    // broken SSH process: give this request one fresh proof, without caching or restarting it.
+    if (admission.kind === "deadline" && retryPreflight("deadline")) {
+      if (await checkConnected()) return !relayAbort.signal.aborted;
+    }
+    return !relayAbort.signal.aborted && tunnel.pending() && await holdForTunnel(tunnel)
+      && await checkConnected() && !relayAbort.signal.aborted;
   };
 
   let upstream: Response | undefined;
   while (!upstream) {
     if (!await readyForFetch()) return await tunnelUnavailable();
+    const fetchStartedAt = performance.now();
     try {
       const init: RequestInit & { duplex?: "half" } = {
         method: req.method,
@@ -415,24 +469,28 @@ export async function relayLinkDataRequest(
       upstream = await (deps.fetchImpl ? deps.fetchImpl(destination, init) : fetchBoundLinkRelay(target, destination, init));
     } catch (error) {
       if (error instanceof LinkRelayAuthenticationError) {
-        cleanup();
-        try { await body?.cancel(); } catch { /* an upload might already be drained */ }
-        return relayAuthenticationFailed(error.message);
+        return await tunnelUnavailable("auth_failed", error.message);
       }
       if (bodyOverflow) {
         cleanup();
         return jsonError(413, "link relay request body too large");
       }
+      if (error instanceof LinkRelayAuthTimeoutError) {
+        // The transport certifies that only its credential-free challenge was sent. A new
+        // iteration obtains a fresh ownership proof and a new authenticated physical socket.
+        const elapsedMs = performance.now() - fetchStartedAt;
+        if (retryPreflight("auth_timeout", elapsedMs)) continue;
+        return await tunnelUnavailable("auth_timeout", "link tunnel unavailable", elapsedMs);
+      }
       // A refused connection sent nothing, and an untouched body can go again once the tunnel
       // is back inside the same hold window. Anything else is never replayed.
       const resendable = connectionRefused(error) && (body === null || (!body.locked && !bodyCancelled));
-      if (!tunnel || !resendable || !tunnel.pending() || !await holdForTunnel(tunnel)) return await tunnelUnavailable();
+      if (!tunnel || !resendable || !tunnel.pending() || !await holdForTunnel(tunnel)) return await tunnelUnavailable("transport_error");
     }
   }
   if (relayAbort.signal.aborted) {
-    cleanup();
     try { await upstream.body?.cancel(); } catch { /* best effort */ }
-    return homeUnreachable("link tunnel unavailable");
+    return await tunnelUnavailable();
   }
 
   const sse = isSse(upstream.headers);
@@ -449,6 +507,7 @@ export async function relayLinkDataRequest(
     try { await upstream.body?.cancel(); } catch { /* best effort */ }
     return jsonError(502, "link relay response body too large");
   }
+  if (recoveryAttempts > 0 && upstream.ok) diagnostic("relay_recovered", "recovered");
   if (req.method === "HEAD" || !upstream.body) {
     cleanup();
     try { await upstream.body?.cancel(); } catch { /* best effort */ }