diff --git a/docs-site/src/content/docs/fr/guides/remote-link.md b/docs-site/src/content/docs/fr/guides/remote-link.md index 9503b02ad37..82d715db467 100644 --- a/docs-site/src/content/docs/fr/guides/remote-link.md +++ b/docs-site/src/content/docs/fr/guides/remote-link.md @@ -55,7 +55,7 @@ Le rôle **Child** exige également qu’OpenCodex fonctionne en mode autonome s - **Connected** signifie que le tunnel SSH est prêt et que Child peut utiliser la liaison Home. - **Reconnecting** signifie que le tunnel est réessayé. Les requêtes peuvent temporairement renvoyer `503` avec `Retry-After`. Sur un Child connecté depuis son propre tableau de bord, une requête attend d’abord jusqu’à 15 secondes le retour du tunnel. -- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Le côté qui ouvre le tunnel SSH réessaie de lui-même après une mise en veille, une panne ou un redémarrage (Home pour un Child qu’il a ajouté, le Child pour une liaison qu’il a rejointe depuis son propre tableau de bord) : environ une fois par minute après un délai dépassé ou une erreur de redirection, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. Pour réessayer tout de suite, choisissez **Retry** dans le tableau de bord du Home ou exécutez `ocx link reconnect --link-id ` sur le Home. Le tunnel de cette liaison redémarre sans redémarrer OpenCodex. +- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Le côté qui ouvre le tunnel SSH réessaie de lui-même après une mise en veille, une panne ou un redémarrage (Home pour un Child qu’il a ajouté, le Child pour une liaison qu’il a rejointe depuis son propre tableau de bord) : une minute après un délai dépassé ou une erreur de redirection, puis avec une attente qui double après chaque nouvel échec, jusqu’à cinq minutes, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. Pour réessayer tout de suite, choisissez **Retry** dans le tableau de bord du Home ou exécutez `ocx link reconnect --link-id ` sur le Home. Le tunnel de cette liaison redémarre sans redémarrer OpenCodex. Une liaison en échec ne bascule pas silencieusement vers un fournisseur local. diff --git a/docs-site/src/content/docs/guides/remote-link.md b/docs-site/src/content/docs/guides/remote-link.md index 3add1b64d6e..6f534ae9f18 100644 --- a/docs-site/src/content/docs/guides/remote-link.md +++ b/docs-site/src/content/docs/guides/remote-link.md @@ -59,12 +59,14 @@ The **Child** role also requires a standalone OpenCodex runtime running on its c - **Connected** means the SSH tunnel is ready and the Child can use the Home link. - **Reconnecting** means the tunnel is being retried. Requests can temporarily return `503` with `Retry-After` while the retry is in progress. On a Child that connected from its own dashboard, a request first waits up to 15 seconds for the tunnel to come back. -- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. The side that runs the SSH tunnel keeps retrying by itself, after sleep, an outage or a restart: Home for a Child it added, and the Child for a link it joined from its own dashboard. It retries about once a minute after a timeout or forwarding error, and every five minutes after an authentication error. A changed host key is never retried. To try again at once, select **Retry** in the Home's dashboard or run `ocx link reconnect --link-id ` on the Home. It restarts that link's tunnel without restarting OpenCodex. +- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. The side that runs the SSH tunnel keeps retrying by itself, after sleep, an outage or a restart: Home for a Child it added, and the Child for a link it joined from its own dashboard. After a timeout or forwarding error it retries after one minute and doubles the wait after each retry that fails again, up to five minutes; after an authentication error it retries every five minutes. A changed host key is never retried. To try again at once, select **Retry** in the Home's dashboard or run `ocx link reconnect --link-id ` on the Home. It restarts that link's tunnel without restarting OpenCodex. A failed link does not silently switch to a local provider. The Child returns a retryable `503` without forwarding the link key or request when its tunnel is failed, stopped, or not supervised. It also keeps the key local when another process owns the tunnel port; free that port so the Child's SSH tunnel can bind it. +On a Child, `GET /healthz` only says that OpenCodex is running. `GET /readyz` (and `ocx ready`) also says whether the link can carry a request right now: while it cannot, it answers `503` with `Retry-After: 1`, `"status": "pending"` and a `link` object such as `{ "state": "unavailable", "reason": "home_unreachable" }`, and while it can, `200` with `"status": "ready"` and `{ "state": "available" }`. The reasons are `tunnel_connecting`, `tunnel_failed`, `tunnel_idle`, `tunnel_unavailable`, `sidecar_invalid`, `home_unreachable`, `home_unauthorized`, `link_auth_failed`, `link_credential_unavailable` and `link_unverified`. A Child that joined from its own dashboard answers from its tunnel. A Child that its Home added has no tunnel of its own to look at, so it asks the Home through the tunnel with the same credential-free check every relayed request starts with; it repeats that check at most once every five seconds, and answers `link_unverified` when the Home has not replied within half a second. + To confirm which Home is serving a Child, compare the `x-opencodex-served-by` header on the Child's `GET /v1/models` with the `instanceId` in the Home's `GET /healthz`. The Child passes the Home's header through unchanged, so the two match while the link works. The value is an opaque id, never a hostname; see [Home identity and the served-by header](/reference/management-api/#home-identity-and-the-served-by-header). ## What a Child shows and manages @@ -98,6 +100,19 @@ When a step fails, the dashboard shows the reason and, when SSH reported one, a - **OpenCodex on the remote computer is too old**: run `ocx update` on that computer. Remote Link needs 2.66.0 or later. - **The remote computer did not report an OpenCodex version**: `ocx --version` on that computer printed something else, for example the usage text of an unsupported Windows install. - **Requests fail with `home_unreachable` while the Home is down**: a Child never serves requests from its own providers. When the tunnel is down before a request is forwarded, the Child answers a retryable 503 and sends nothing to the Home. If the tunnel drops after forwarding started, the answer is also `home_unreachable`, but the Home may already have received the request. A failed link relay authentication answers `link_relay_auth_failed` instead: upgrade the Home and Child or re-link. Whether GOAT may move that device's work to its own sign-ins is the per-device local fallback permission on the Home's **Remote Link** page, or `ocx fallback local on --device `; see [Fallback off](/guides/fallback-off/). +- **A Home link stays Failed with a forwarding error after the Home slept or changed networks**: run `ocx link status --json` on the Home. Its `tunnel` field shows the attempts since the link was last connected, `nextRetryAt`, and the last failure. A `lastFailure.cause` of `remote_forward_conflict` means the Child's SSH server still holds the tunnel port for the session that died with the Home's old connection, so every new tunnel is refused until it lets go. OpenCodex keeps retrying and connects on its own once the port is free; it never stops processes on the Child. The Child's SSH server notices a dead session only when it probes it. Without `ClientAliveInterval` in the Child's `sshd_config`, that can take as long as the operating system's TCP keepalive (two hours by default on macOS). Setting `ClientAliveInterval 15` and `ClientAliveCountMax 3` there makes it drop such a session within about a minute. + +### Retryable 503 diagnostics + +A relay-generated `503` includes a server-generated `x-opencodex-link-request-id`. Use it to correlate a failure with local evidence under the Child's OpenCodex configuration home: `link/diagnostics.jsonl` and one rotated backup, `link/diagnostics.jsonl.1`. Each file is capped at 256 KiB, with private permissions (directory 0700 and files 0600 on POSIX; private ACLs on Windows). Records contain only fixed event/outcome/state categories, generated correlation IDs, numeric timing/generation/attempt metadata and timestamps, never secrets, request bodies, headers, URLs, account identifiers or raw errors. Each event/outcome pair is limited to one record per second; the next record includes a count of suppressed events, so an absent record does not prove an absent failure. A backward wall-clock adjustment starts a new rate-limit window, including when the writer queue is full, and preserves the suppression count. An asynchronous writer keeps at most 56 pending records, counts overflow as suppression, and yields while Windows ACL work runs. Logging is best effort and cannot change whether a request is admitted. Graceful signal shutdown and standalone recycling wait up to one second for queued diagnostics before exiting; stalled storage, forced termination and power loss can still lose records. + +`owner_lookup` records refused ownership checks; routine successful checks are silent. `owner_lookup.elapsedMs` measures the owner lookup itself. For `auth_timeout`, `relay_retry` and `relay_failure` report the actual transport-attempt duration in `elapsedMs` (normally about five seconds). A `relay_recovered` event with outcome `recovered` is emitted only after a retry receives an upstream response with `upstream.ok` (HTTP 2xx); it does not prove the streamed body completed. Rate limiting makes request-ID correlation sampled evidence, not a guaranteed trace of every failed request. + +The Home writes the same kind of file under its own configuration home for the tunnels it runs. Each change of a Home-initiated tunnel's state is a `tunnel_transition` record with `previousState`, `state`, the cause as `outcome` (`auth`, `hostkey`, `forward`, `remote_forward_conflict`, `timeout`, `network`, `unknown`, `spawn_failed` or `listener_unavailable`; `connected`, `spawned` or `stopped` when nothing failed), the `attempt` count since the link was last connected, the link's `tunnelPort` and, when a retry is scheduled, `retryAfterMs`. A retry of a failed link that fails again is recorded too, so an outage leaves one record per attempt. The same rate limit applies, across all links. + +Before sending data, OpenCodex allows one shared preflight retry for either an owner-lookup deadline or a specifically identified five-second authentication deadline. The retry obtains fresh owner proof where required and a fresh physical socket, then authenticates that socket before sending the link credential or body. It does not reuse cached owner approval, bypass security checks, restart SSH or OpenCodex, or replay a dispatched request body. The two-second owner-lookup and five-second authentication deadlines are unchanged for each attempt; other failures do not qualify for this preflight retry. + +This can recover these transient deadline paths and helps diagnose future failures. It does not establish the historical cause of [#7297](https://github.com/lidge-jun/opencodex/issues/7297). ## Security diff --git a/docs-site/src/content/docs/ja/guides/remote-link.md b/docs-site/src/content/docs/ja/guides/remote-link.md index 4ae8b634195..905cfb3db49 100644 --- a/docs-site/src/content/docs/ja/guides/remote-link.md +++ b/docs-site/src/content/docs/ja/guides/remote-link.md @@ -55,7 +55,7 @@ Home のプロバイダーを使うコンピューターで次の操作を行い - **Connected** は SSH トンネルが準備でき、Child が Home のリンクを使える状態です。 - **Reconnecting** はトンネルを再試行している状態です。再試行中はリクエストが `Retry-After` 付きの `503` を一時的に返すことがあります。自分のダッシュボードから接続した Child では、リクエストはまずトンネルの復帰を最大 15 秒待ちます。 -- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。SSH トンネルを張る側が、スリープ、障害、再起動の後も自動で再試行します。Home が追加した Child では Home が、自分のダッシュボードから接続した Child では Child が再試行します。タイムアウトや転送エラーの後は約 1 分ごと、認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。すぐに再接続するには、Home のダッシュボードで **Retry** を選ぶか、Home で `ocx link reconnect --link-id ` を実行します。OpenCodex を再起動せずに、そのリンクのトンネルだけを張り直します。 +- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。SSH トンネルを張る側が、スリープ、障害、再起動の後も自動で再試行します。Home が追加した Child では Home が、自分のダッシュボードから接続した Child では Child が再試行します。タイムアウトや転送エラーの後は 1 分後に再試行し、失敗が続くたびに間隔を倍にして最大 5 分ごとになります。認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。すぐに再接続するには、Home のダッシュボードで **Retry** を選ぶか、Home で `ocx link reconnect --link-id ` を実行します。OpenCodex を再起動せずに、そのリンクのトンネルだけを張り直します。 リンクが失敗しても、ローカルプロバイダーへ自動的に切り替わることはありません。 diff --git a/docs-site/src/content/docs/ko/guides/remote-link.md b/docs-site/src/content/docs/ko/guides/remote-link.md index ea3733f92e8..5ad0f0ea302 100644 --- a/docs-site/src/content/docs/ko/guides/remote-link.md +++ b/docs-site/src/content/docs/ko/guides/remote-link.md @@ -55,7 +55,7 @@ Home의 프로바이더를 사용할 컴퓨터에서 다음을 진행합니다. - **Connected**는 SSH 터널이 준비되어 Child가 Home 링크를 사용할 수 있다는 뜻입니다. - **Reconnecting**은 터널을 다시 연결하는 중이라는 뜻입니다. 재시도 중에는 요청이 일시적으로 `Retry-After`와 함께 `503`을 반환할 수 있습니다. 자기 대시보드에서 연결한 Child에서는 요청이 먼저 터널이 돌아오기를 최대 15초 기다립니다. -- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. SSH 터널을 띄운 쪽이 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. Home이 추가한 Child라면 Home이, 자기 대시보드에서 연결한 Child라면 Child가 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 약 1분마다, 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. 바로 다시 연결하려면 Home 대시보드에서 **Retry**를 누르거나 Home에서 `ocx link reconnect --link-id `를 실행하세요. OpenCodex를 재시작하지 않고 그 링크의 터널만 다시 띄웁니다. +- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. SSH 터널을 띄운 쪽이 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. Home이 추가한 Child라면 Home이, 자기 대시보드에서 연결한 Child라면 Child가 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 1분 뒤에 다시 시도하고, 다시 실패할 때마다 간격을 두 배로 늘려 최대 5분마다 시도합니다. 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. 바로 다시 연결하려면 Home 대시보드에서 **Retry**를 누르거나 Home에서 `ocx link reconnect --link-id `를 실행하세요. OpenCodex를 재시작하지 않고 그 링크의 터널만 다시 띄웁니다. 링크가 실패해도 로컬 프로바이더로 조용히 전환하지 않습니다. diff --git a/docs-site/src/content/docs/reference/cli/lifecycle.md b/docs-site/src/content/docs/reference/cli/lifecycle.md index 758cbda5a05..3940a5c7a64 100644 --- a/docs-site/src/content/docs/reference/cli/lifecycle.md +++ b/docs-site/src/content/docs/reference/cli/lifecycle.md @@ -426,6 +426,12 @@ The CLI's own `--json` output is deliberately narrower than the HTTP body: it em `unreachable`. Exit codes are 0 for ready; 1 for not-ready, pending, failed, timeout, or unreachable; and 64 for invalid arguments. +On a Remote Link Child, readiness also covers the link. While the link cannot carry a request, +`/readyz` answers `503` with `status: "pending"` and `link: {state: "unavailable", reason}`, so +`ocx ready --wait` keeps waiting until the link is back; a working link adds +`link: {state: "available"}` to the `200` answer. `/healthz` on a Child keeps meaning only that the +process runs. The reasons are listed in [Remote Link](/guides/remote-link/#link-status). + ### `ocx resolve [--json]` Resolve the runtime facts a shell needs without re-implementing them: the config home, the @@ -1153,6 +1159,14 @@ starts and then once a minute; the Home keeps reports in memory only, so after a fields read `null` until the next report. A Child whose runtime started before `lastApplyAt` is running an older runtime than its last apply. +For a link the Home added, each row also carries `tunnel`: `attempt` (ssh attempts since the link +was last connected), `nextRetryAt` (ISO time of the next scheduled retry, `null` while one runs or +none is due), `lastTransition` (`{from, to, at, cause}`) and `lastFailure` (`{cause, at}`). A cause +is one of `auth`, `hostkey`, `forward`, `remote_forward_conflict`, `timeout`, `network`, +`unknown`, `spawn_failed` and `listener_unavailable`. The Home keeps it in memory, so it starts +empty after a Home restart. `GET /api/link/status` includes the field only when asked with +`?diagnostics=1`, because an older `ocx link status` rejects fields it does not know. + After `ocx sync` applies the Home's settings on a Child, it restarts the Child's client runtime when that runtime started before `lastApplyAt`, using the same restart as `ocx restart`. A runtime that already started after the apply is left alone, nothing is started when no client runtime is diff --git a/docs-site/src/content/docs/ru/guides/remote-link.md b/docs-site/src/content/docs/ru/guides/remote-link.md index 9ade1c07cc8..229918c6f7c 100644 --- a/docs-site/src/content/docs/ru/guides/remote-link.md +++ b/docs-site/src/content/docs/ru/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** означает, что SSH-туннель готов и Child может использовать связь Home. - **Reconnecting** означает, что туннель переподключается. Во время повторных попыток запросы могут временно получать `503` с `Retry-After`. На Child, подключённом из собственной панели, запрос сначала до 15 секунд ждёт восстановления туннеля. -- **Failed** означает, что связь требует действий. Проверьте SSH-аутентификацию, подтверждённый ключ хоста, перенаправление или причину тайм-аута. Сторона, которая держит SSH-туннель, сама повторяет попытки после сна, сбоя или перезапуска (Home — для добавленного им Child, Child — для связи, к которой он подключился из собственной панели): примерно раз в минуту после тайм-аута или ошибки перенаправления и каждые пять минут после ошибки аутентификации. Изменённый ключ хоста никогда не повторяется. Чтобы переподключиться сразу, нажмите **Retry** в панели Home или выполните `ocx link reconnect --link-id ` на Home. Туннель этой связи перезапустится без перезапуска OpenCodex. +- **Failed** означает, что связь требует действий. Проверьте SSH-аутентификацию, подтверждённый ключ хоста, перенаправление или причину тайм-аута. Сторона, которая держит SSH-туннель, сама повторяет попытки после сна, сбоя или перезапуска (Home — для добавленного им Child, Child — для связи, к которой он подключился из собственной панели): после тайм-аута или ошибки перенаправления через минуту, а затем с интервалом, который удваивается после каждой новой неудачи, но не превышает пяти минут; после ошибки аутентификации — каждые пять минут. Изменённый ключ хоста никогда не повторяется. Чтобы переподключиться сразу, нажмите **Retry** в панели Home или выполните `ocx link reconnect --link-id ` на Home. Туннель этой связи перезапустится без перезапуска OpenCodex. При сбое связи система молча не переключается на локального провайдера. diff --git a/docs-site/src/content/docs/tr/guides/remote-link.md b/docs-site/src/content/docs/tr/guides/remote-link.md index da56c5c72fa..ef4af2ee5ec 100644 --- a/docs-site/src/content/docs/tr/guides/remote-link.md +++ b/docs-site/src/content/docs/tr/guides/remote-link.md @@ -55,7 +55,7 @@ Bağlanmak bu bilgisayardaki OpenCodex'i yeniden başlatır. Zaten çalışan Co - **Connected**, SSH tünelinin hazır ve Child'ın Home bağlantısını kullanabilir olduğu anlamına gelir. - **Reconnecting**, tünelin yeniden denendiği anlamına gelir. Yeniden deneme sırasında istekler geçici olarak `Retry-After` ile birlikte `503` döndürebilir. Kendi panosundan bağlanan bir Child üzerinde istek önce tünelin geri gelmesi için en fazla 15 saniye bekler. -- **Failed**, bağlantının ilgilenilmesi gerektiği anlamına gelir. SSH kimlik doğrulamasını, onaylanan ana bilgisayar anahtarını, yönlendirmeyi veya zaman aşımı nedenini kontrol edin. SSH tünelini açan taraf (eklediği Child için Home, kendi panosundan katıldığı bağlantı için Child); uyku, kesinti veya yeniden başlatmadan sonra kendiliğinden yeniden dener: zaman aşımı veya yönlendirme hatasından sonra yaklaşık dakikada bir, kimlik doğrulama hatasından sonra beş dakikada bir. Değişmiş bir ana bilgisayar anahtarı asla yeniden denenmez. Hemen yeniden denemek için Home panosunda **Retry** düğmesini seçin veya Home üzerinde `ocx link reconnect --link-id ` komutunu çalıştırın. Bu, OpenCodex'i yeniden başlatmadan yalnızca o bağlantının tünelini yeniden başlatır. +- **Failed**, bağlantının ilgilenilmesi gerektiği anlamına gelir. SSH kimlik doğrulamasını, onaylanan ana bilgisayar anahtarını, yönlendirmeyi veya zaman aşımı nedenini kontrol edin. SSH tünelini açan taraf (eklediği Child için Home, kendi panosundan katıldığı bağlantı için Child); uyku, kesinti veya yeniden başlatmadan sonra kendiliğinden yeniden dener: zaman aşımı veya yönlendirme hatasından sonra bir dakika sonra, ardından her yeni başarısızlıkta bekleme ikiye katlanarak en fazla beş dakikada bir; kimlik doğrulama hatasından sonra beş dakikada bir. Değişmiş bir ana bilgisayar anahtarı asla yeniden denenmez. Hemen yeniden denemek için Home panosunda **Retry** düğmesini seçin veya Home üzerinde `ocx link reconnect --link-id ` komutunu çalıştırın. Bu, OpenCodex'i yeniden başlatmadan yalnızca o bağlantının tünelini yeniden başlatır. Bağlantı başarısız olduğunda sistem sessizce yerel bir sağlayıcıya geçmez. diff --git a/docs-site/src/content/docs/zh-cn/guides/remote-link.md b/docs-site/src/content/docs/zh-cn/guides/remote-link.md index dffb61b85e2..37c89eef023 100644 --- a/docs-site/src/content/docs/zh-cn/guides/remote-link.md +++ b/docs-site/src/content/docs/zh-cn/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** 表示 SSH 隧道已就绪，子机可以使用主机链接。 - **Reconnecting** 表示正在重试隧道。重试期间请求可能暂时返回带有 `Retry-After` 的 `503`。在从自己的仪表板连接的 Child 上，请求会先最多等待 15 秒让隧道恢复。 -- **Failed** 表示链接需要处理。请检查 SSH 身份验证、已确认的主机密钥、转发或超时原因。建立 SSH 隧道的一方会在睡眠、故障或重启后自动重试（Home 添加的 Child 由 Home 重试，从自己的仪表板连接的 Child 由 Child 重试）：超时或转发错误后大约每分钟一次，身份验证错误后每五分钟一次。主机密钥变更时不会重试。要立即重连，请在 Home 的仪表板中选择 **Retry**，或在 Home 上运行 `ocx link reconnect --link-id `。这只会重启该链接的隧道，不会重启 OpenCodex。 +- **Failed** 表示链接需要处理。请检查 SSH 身份验证、已确认的主机密钥、转发或超时原因。建立 SSH 隧道的一方会在睡眠、故障或重启后自动重试（Home 添加的 Child 由 Home 重试，从自己的仪表板连接的 Child 由 Child 重试）：超时或转发错误后先在一分钟后重试，之后每次重试失败等待时间翻倍，最长五分钟一次；身份验证错误后每五分钟一次。主机密钥变更时不会重试。要立即重连，请在 Home 的仪表板中选择 **Retry**，或在 Home 上运行 `ocx link reconnect --link-id `。这只会重启该链接的隧道，不会重启 OpenCodex。 链接失败时不会静默切换到本地提供商。 diff --git a/docs-site/src/content/docs/zh-tw/guides/remote-link.md b/docs-site/src/content/docs/zh-tw/guides/remote-link.md index cde622a3fed..d91b65acb3a 100644 --- a/docs-site/src/content/docs/zh-tw/guides/remote-link.md +++ b/docs-site/src/content/docs/zh-tw/guides/remote-link.md @@ -55,7 +55,7 @@ Host devbox - **Connected** 表示 SSH 通道已準備好，Child 可以使用 Home 連結。 - **Reconnecting** 表示正在重試通道。重試期間請求可能暫時回傳帶有 `Retry-After` 的 `503`。在從自己的儀表板連線的 Child 上，請求會先最多等待 15 秒讓通道恢復。 -- **Failed** 表示連結需要處理。請檢查 SSH 驗證、已確認的主機金鑰、轉送或逾時原因。建立 SSH 通道的一方會在睡眠、故障或重新啟動後自動重試（Home 新增的 Child 由 Home 重試，從自己的儀表板連線的 Child 由 Child 重試）：逾時或轉送錯誤後大約每分鐘一次，驗證錯誤後每五分鐘一次。主機金鑰變更時不會重試。要立即重新連線，請在 Home 的儀表板中選擇 **Retry**，或在 Home 上執行 `ocx link reconnect --link-id `。這只會重新啟動該連結的通道，不會重新啟動 OpenCodex。 +- **Failed** 表示連結需要處理。請檢查 SSH 驗證、已確認的主機金鑰、轉送或逾時原因。建立 SSH 通道的一方會在睡眠、故障或重新啟動後自動重試（Home 新增的 Child 由 Home 重試，從自己的儀表板連線的 Child 由 Child 重試）：逾時或轉送錯誤後先在一分鐘後重試，之後每次重試失敗等待時間加倍，最長五分鐘一次；驗證錯誤後每五分鐘一次。主機金鑰變更時不會重試。要立即重新連線，請在 Home 的儀表板中選擇 **Retry**，或在 Home 上執行 `ocx link reconnect --link-id `。這只會重新啟動該連結的通道，不會重新啟動 OpenCodex。 連結失敗時不會靜默切換到本機供應商。 diff --git a/scripts/test-layout/layout-clients.json b/scripts/test-layout/layout-clients.json index 9224a33affb..6c384913a52 100644 --- a/scripts/test-layout/layout-clients.json +++ b/scripts/test-layout/layout-clients.json @@ -1,4 +1,7 @@ { + "link-diagnostics.test.ts": "clients", + "link-tunnel-history.test.ts": "clients", + "link-readiness.test.ts": "clients", "gentle-shell-path.test.ts": "clients", "gentle-shell-client.test.ts": "clients", "droid-managed-reasoning-defaults.test.ts": "clients", diff --git a/src/cli/link.ts b/src/cli/link.ts index 9d7d3b98044..ec0b8ccaebf 100644 --- a/src/cli/link.ts +++ b/src/cli/link.ts @@ -1,7 +1,10 @@ import { clientLinkStatusIdentity } from "../client/link-status"; import { clientLinkTunnelStatus } from "../client/link-tunnel"; import { loadConfig } from "../config"; -import { projectLinkStatus, type ClientLinkIdentity, type LinkStatusConfig, type LinkStatusDto } from "../link/status-projection"; +import { + projectLinkStatus, type ClientLinkIdentity, type LinkStatusConfig, type LinkStatusDto, type LinkTunnelDiagnostics, type LinkWireState, +} from "../link/status-projection"; +import { TUNNEL_CAUSES } from "../link/tunnel-history"; import { isEpochMs, type LinkChildParity } from "../link/child-parity"; import { CODEX_SETTINGS_DIGEST_PATTERN } from "../link/codex-settings"; import type { LinkLocalPeer, LinkPeerStatus } from "../link/compatibility"; @@ -40,30 +43,11 @@ export const LINK_USAGE = `Usage: Forced revoke skips remote cleanup. Run ocx disconnect on the remote client. Cleanup finds Windows Children and explains how to move them to Hub mode; it changes nothing.`; -type LinkState = "connecting" | "connected" | "reconnecting" | "failed" | "idle"; -type ListenerState = "off" | "listening" | "failed"; - -export interface LinkStatusPayload { - role: "standalone" | "home" | "child"; - listener: { state: ListenerState; port: number | null }; - links: Array; - child: null | { - alias: string; - state: LinkState; - since: string; - reason: string | null; - }; - local?: LinkLocalPeer; -} +/** + * What `ocx link status` prints: the runtime's status DTO itself, so a field the server adds (like + * `peer` or the diagnostics `tunnel`) is part of this type the moment it is part of the wire. + */ +export type LinkStatusPayload = LinkStatusDto; export interface LinkIssuePayload { linkId: string; @@ -165,11 +149,11 @@ function validateStatus(value: unknown): LinkStatusPayload { throw new Error("invalid link API response: listener port"); } if (!Array.isArray(value.links)) throw new Error("invalid link API response: links"); - const links = value.links.map((candidate, index) => { + const links = value.links.map((candidate, index): LinkStatusPayload["links"][number] => { if (!isRecord(candidate)) throw new Error(`invalid link API response: link ${index}`); // The OCX-6 parity fields are optional on the wire (an older runtime omits them) and always // present in this output, null when absent. - assertKeys(candidate, ["id", "alias", "direction", "state", "since", "reason", "tunnelPort"], [...PARITY_KEYS, "peer"], `link ${index}`); + assertKeys(candidate, ["id", "alias", "direction", "state", "since", "reason", "tunnelPort"], [...PARITY_KEYS, "peer", "tunnel"], `link ${index}`); const parity = validateParity(candidate, index); if (typeof candidate.id !== "string" || !LINK_ID.test(candidate.id) || !validString(candidate.alias) || (candidate.direction !== "hub-initiated" && candidate.direction !== "client-initiated") @@ -180,14 +164,15 @@ function validateStatus(value: unknown): LinkStatusPayload { return { id: candidate.id, alias: candidate.alias, - direction: candidate.direction, - state: candidate.state, + direction: candidate.direction as LinkDirection, + state: candidate.state as LinkWireState, since: candidate.since, reason: candidate.reason, tunnelPort: candidate.tunnelPort, ...parity, ...(candidate.peer !== undefined ? { peer: validatePeer(candidate.peer, index) } : {}), - } as LinkStatusPayload["links"][number]; + ...(candidate.tunnel !== undefined ? { tunnel: validateTunnel(candidate.tunnel, index) } : {}), + }; }); let child: LinkStatusPayload["child"] = null; if (value.child !== null) { @@ -310,8 +295,9 @@ async function runStatus(args: string[], deps: LinkCliDeps): Promise { } throw error; } - // A proxy older than the compatibility report ignores the query and answers the exact K16 shape. - const result = validateStatus(await linkRequest("/api/link/status?compatibility=1", {}, { ...deps, baseUrl })); + // A proxy older than the compatibility report or the tunnel diagnostics ignores the query and + // answers without those fields. + const result = validateStatus(await linkRequest("/api/link/status?compatibility=1&diagnostics=1", {}, { ...deps, baseUrl })); console.log(JSON.stringify(result)); } @@ -385,6 +371,33 @@ export async function runLinkCommand(rawArgs: string[], deps: LinkCliDeps = {}): } const PARITY_KEYS = ["lastApplyAt", "runtimeStartedAt", "appliedCodexSettingsDigest", "reportedAt"] as const; +const WIRE_STATES = new Set(["connecting", "connected", "reconnecting", "failed", "idle"]); +const CAUSES = new Set(TUNNEL_CAUSES); + +function validIso(value: unknown): value is string { + return typeof value === "string" && value.length <= 64 && Number.isFinite(Date.parse(value)); +} + +function validateTunnel(value: unknown, index: number): LinkTunnelDiagnostics { + const fail = (): never => { throw new Error(`invalid link API response: link ${index} tunnel`); }; + if (!isRecord(value)) return fail(); + assertExactKeys(value, ["attempt", "nextRetryAt", "lastTransition", "lastFailure"], `link ${index} tunnel`); + const { attempt, nextRetryAt, lastTransition, lastFailure } = value; + if (typeof attempt !== "number" || !Number.isSafeInteger(attempt) || attempt < 0) fail(); + if (nextRetryAt !== null && !validIso(nextRetryAt)) fail(); + if (lastTransition !== null) { + if (!isRecord(lastTransition)) return fail(); + assertExactKeys(lastTransition, ["from", "to", "at", "cause"], `link ${index} tunnel transition`); + if (!WIRE_STATES.has(lastTransition.from as string) || !WIRE_STATES.has(lastTransition.to as string) + || !validIso(lastTransition.at) || (lastTransition.cause !== null && !CAUSES.has(lastTransition.cause as string))) fail(); + } + if (lastFailure !== null) { + if (!isRecord(lastFailure)) return fail(); + assertExactKeys(lastFailure, ["cause", "at"], `link ${index} tunnel failure`); + if (!CAUSES.has(lastFailure.cause as string) || !validIso(lastFailure.at)) fail(); + } + return value as unknown as LinkTunnelDiagnostics; +} function validateParity(candidate: Record, index: number): LinkChildParity { const time = (key: "lastApplyAt" | "runtimeStartedAt" | "reportedAt"): number | null => { diff --git a/src/client/link-ingress.ts b/src/client/link-ingress.ts index 85af7ad6002..25281202257 100644 --- a/src/client/link-ingress.ts +++ b/src/client/link-ingress.ts @@ -76,7 +76,7 @@ function liftRequestIdleTimer(req: Request, server: Pick, "timeo * fetch, so a rebinding or cross-site browser page cannot reach the Home through this port. * - Without the committed link key nothing is fetched, and the caller gets * `503 link_credential_unavailable`. - * - `/readyz` stays local: it reports this listener, not the tunnel. + * - `/readyz` stays local and is never relayed; the listener answers it from `link-readiness.ts`. * - A relayed request is exempt from the listener's idle limit, like a standalone data route. */ export function handleLinkIngress( diff --git a/src/client/link-readiness.ts b/src/client/link-readiness.ts new file mode 100644 index 00000000000..4036833f3ce --- /dev/null +++ b/src/client/link-readiness.ts @@ -0,0 +1,97 @@ +/** + * Whether a Remote Link Child can serve relayed requests right now, for its `GET /readyz`. + * + * `/healthz` keeps meaning "this process answers". A Child whose tunnel is down still answers it, + * yet every relayed turn fails with 503, so readiness is where that shows: not ready, with one + * fixed reason word. Nothing here carries ssh output, host names, ports or keys. + * + * - A Child-initiated link reads its own tunnel supervisor; no network traffic. + * - A Home-initiated link has no local tunnel to inspect (the Home runs `ssh -R`), so it runs + * the credential-free relay preflight through the tunnel port. One preflight runs at a time, a + * verdict is reused for `LINK_READINESS_TTL_MS`, and a request waits at most + * `LINK_READINESS_WAIT_MS` for a running preflight before it answers `link_unverified`. An + * unauthenticated caller can therefore cause at most one preflight per TTL. + */ +import { LINK_RELAY_AUTH_TIMEOUT_MS } from "../link/relay-auth"; +import { isLinkPort } from "../link/ports"; +import { probeLinkRelayPreflight, type BoundLinkRelayTarget, type LinkRelayPreflightResult } from "./link-relay-transport"; +import type { ClientLinkSupervisorStatus } from "./link-tunnel"; + +export type LinkReadinessReason = + | "tunnel_idle" | "tunnel_connecting" | "tunnel_failed" | "tunnel_unavailable" | "sidecar_invalid" + | "home_unreachable" | "home_unauthorized" | "link_auth_failed" | "link_credential_unavailable" + | "link_unverified"; + +export type LinkReadiness = { ready: true } | { ready: false; reason: LinkReadinessReason }; + +export const LINK_READINESS_TTL_MS = 5_000; +export const LINK_READINESS_WAIT_MS = 500; + +const READY: LinkReadiness = { ready: true }; +const notReady = (reason: LinkReadinessReason): LinkReadiness => ({ ready: false, reason }); + +/** + * A Child-initiated link from its supervisor. The keyed probe the supervisor runs every 30 seconds + * while connected is display-only for the tunnel, but a Home that stopped answering or refuses the + * key cannot serve a turn, so readiness follows it. A Home whose own startup readiness is pending + * still serves relayed requests (the probe's `home_not_ready`), so that reads ready. + */ +export function clientInitiatedLinkReadiness(status: ClientLinkSupervisorStatus): LinkReadiness { + if (status.kind === "failed") return notReady("sidecar_invalid"); + if (status.kind === "stopped") return notReady("tunnel_idle"); + const { state, probe } = status; + if (state.kind === "connected") { + if (probe === "home_unreachable") return notReady("home_unreachable"); + if (probe === "unauthorized") return notReady("home_unauthorized"); + return READY; + } + if (state.kind === "connecting" || state.kind === "reconnecting") return notReady("tunnel_connecting"); + if (state.kind === "failed") return notReady(state.inFlight ? "tunnel_connecting" : "tunnel_failed"); + return notReady("tunnel_idle"); +} + +function fromPreflight(result: LinkRelayPreflightResult): LinkReadiness { + if (result === "ok") return READY; + return notReady(result === "unreachable" ? "home_unreachable" : "link_auth_failed"); +} + +export interface HomeInitiatedLinkReadinessDeps { + target: Omit; + linkKey: () => string | null; + probe?: (target: BoundLinkRelayTarget, options: { timeoutMs: number }) => Promise; + now?: () => number; + /** Resolves after `ms`; the wait for a running preflight. */ + delay?: (ms: number) => Promise; +} + +/** The readiness of a Home-initiated link, from a shared, rate-limited relay preflight. */ +export function createHomeInitiatedLinkReadiness(deps: HomeInitiatedLinkReadinessDeps): () => Promise { + const probe = deps.probe ?? probeLinkRelayPreflight; + const now = deps.now ?? Date.now; + const delay = deps.delay ?? (ms => new Promise(resolve => { + const timer = setTimeout(resolve, ms); + (timer as { unref?: () => void }).unref?.(); + })); + let last: { at: number; verdict: LinkReadiness } | undefined; + let flight: Promise | undefined; + const run = (admissionKey: string): Promise => { + flight ??= probe({ ...deps.target, admissionKey }, { timeoutMs: LINK_RELAY_AUTH_TIMEOUT_MS }) + .then(fromPreflight, () => notReady("home_unreachable")) + .then(verdict => { + last = { at: now(), verdict }; + return verdict; + }) + .finally(() => { flight = undefined; }); + return flight; + }; + return async () => { + if (!isLinkPort(deps.target.tunnelPort)) return notReady("tunnel_unavailable"); + let admissionKey: string | null; + try { admissionKey = deps.linkKey(); } catch { admissionKey = null; } + if (!admissionKey) return notReady("link_credential_unavailable"); + const at = now(); + if (last && at >= last.at && at - last.at < LINK_READINESS_TTL_MS) return last.verdict; + const unverified = notReady("link_unverified"); + return await Promise.race([run(admissionKey), delay(LINK_READINESS_WAIT_MS).then(() => unverified)]); + }; +} diff --git a/src/client/link-relay-transport.ts b/src/client/link-relay-transport.ts index b43bf4da7b1..3ec9c6ee520 100644 --- a/src/client/link-relay-transport.ts +++ b/src/client/link-relay-transport.ts @@ -9,7 +9,7 @@ import { linkPeerReportHeaders } from "../link/peer-report"; import { buildPeerFeatures } from "../remote/features"; import { LINK_RELAY_AUTH_PATH, LINK_RELAY_AUTH_TIMEOUT_MS, LINK_RELAY_AUTH_VERSION, - LINK_RELAY_SESSION_HEADER, linkRelayChallenge, linkRelayProofMatches, + LINK_RELAY_PROBE_HEADER, LINK_RELAY_SESSION_HEADER, linkRelayChallenge, linkRelayProofMatches, } from "../link/relay-auth"; export interface BoundLinkRelayTarget { @@ -27,6 +27,14 @@ export class LinkRelayAuthenticationError extends Error { } } +/** Only the credential-free preflight deadline expired; no data request has begun. */ +export class LinkRelayAuthTimeoutError extends Error { + constructor() { + super("link relay authentication timed out"); + this.name = "LinkRelayAuthTimeoutError"; + } +} + /** One physical connection, ever. A closed keep-alive socket cannot trigger a silent reconnect. */ class SingleConnectionAgent extends Agent { socket: Socket | undefined; @@ -153,21 +161,76 @@ async function responseOnConnection(incoming: IncomingMessage, agent: SingleConn let peerReport: Record | undefined; -/** Authenticate and stream one request on the SAME socket; never fall back to an ordinary fetch. */ -export async function fetchBoundLinkRelay(target: BoundLinkRelayTarget, destination: string, init: RequestInit): Promise { +/** The credential-free challenge for one connection; the proof binds this nonce to the link key. */ +function relayChallenge(target: BoundLinkRelayTarget, origin: URL) { + const challenge = linkRelayChallenge(serviceApiTokenFingerprint(target.admissionKey), target.apiKeyId!, target.linkId!); + const probe = new URL(LINK_RELAY_AUTH_PATH, origin); + probe.search = new URLSearchParams({ version: LINK_RELAY_AUTH_VERSION, key: target.apiKeyId!, + link: target.linkId!, nonce: challenge.nonce, proof: challenge.caller }).toString(); + // The challenge also carries this Child's version and features for the Home's link status. + peerReport ??= linkPeerReportHeaders(packageVersion("0.0.0"), buildPeerFeatures()); + return { challenge, probe, headers: peerReport }; +} + +/** + * The outcome of the credential-free preflight alone: `ok` when the Home answered with a proof + * only the holder of the link key can make, `unrecognized` for a peer without the route (404), + * `auth_failed` for any other answer, and `unreachable` when nothing answered in time. + */ +export type LinkRelayPreflightResult = "ok" | "unreachable" | "auth_failed" | "unrecognized"; + +/** + * Run only the preflight a relayed request starts with, on its own connection, and close it. It + * sends no link key, no data request and no body; the Child's readiness uses it to learn whether + * the Home answers through the tunnel right now. It is marked probe-only, so the Home reserves no + * session for a data request that never comes. The response body is never read. + */ +export async function probeLinkRelayPreflight( + target: BoundLinkRelayTarget, options: { timeoutMs?: number } = {}, +): Promise { + const timeoutMs = options.timeoutMs ?? LINK_RELAY_AUTH_TIMEOUT_MS; + if (!Number.isFinite(timeoutMs) || timeoutMs <= 0) throw new RangeError("invalid link relay preflight timeout"); + if (!target.linkId || !target.apiKeyId) return "auth_failed"; + const origin = new URL(`http://127.0.0.1:${target.tunnelPort}`); + let request: ReturnType; + try { request = relayChallenge(target, origin); } catch { return "auth_failed"; } + const agent = new SingleConnectionAgent({ keepAlive: false, maxSockets: 1, maxFreeSockets: 0 }); + const deadline = clearableDeadline(timeoutMs); + try { + const response = await requestOnConnection(request.probe, { + method: "GET", signal: deadline.signal, headers: { ...request.headers, [LINK_RELAY_PROBE_HEADER]: "1" }, + }, agent); + const proof = response.headers["x-opencodex-link-proof"]; + if (response.statusCode === 204 && typeof proof === "string" && linkRelayProofMatches(proof, request.challenge.expected)) return "ok"; + return response.statusCode === 404 ? "unrecognized" : "auth_failed"; + } catch { + return "unreachable"; + } finally { + deadline.clear(); + await agent.close(); + } +} + +/** + * Authenticate and stream one request on the SAME socket; never fall back to an ordinary fetch. + * LinkRelayAuthTimeoutError certifies that only the credential-free preflight expired before + * data-request construction. Caller aborts and failures after that boundary keep their own errors. + */ +export async function fetchBoundLinkRelay( + target: BoundLinkRelayTarget, destination: string, init: RequestInit, + options: { authTimeoutMs?: number } = {}, +): Promise { + const authTimeoutMs = options.authTimeoutMs ?? LINK_RELAY_AUTH_TIMEOUT_MS; + if (!Number.isFinite(authTimeoutMs) || authTimeoutMs <= 0) throw new RangeError("invalid link relay auth timeout"); if (!target.linkId || !target.apiKeyId) throw new LinkRelayAuthenticationError(); const url = new URL(destination); if (url.origin !== `http://127.0.0.1:${target.tunnelPort}`) throw new LinkRelayAuthenticationError(); - const challenge = linkRelayChallenge(serviceApiTokenFingerprint(target.admissionKey), target.apiKeyId, target.linkId); - const probe = new URL(LINK_RELAY_AUTH_PATH, url); - probe.search = new URLSearchParams({ version: LINK_RELAY_AUTH_VERSION, key: target.apiKeyId, - link: target.linkId, nonce: challenge.nonce, proof: challenge.caller }).toString(); + const { challenge, probe, headers: preflightHeaders } = relayChallenge(target, url); const agent = new SingleConnectionAgent({ keepAlive: true, maxSockets: 1, maxFreeSockets: 1 }); - const deadline = clearableDeadline(LINK_RELAY_AUTH_TIMEOUT_MS, init.signal ?? undefined); + const deadline = clearableDeadline(authTimeoutMs, init.signal ?? undefined); + let dataRequestStarted = false; try { - // The challenge also carries this Child's version and features for the Home's link status. - peerReport ??= linkPeerReportHeaders(packageVersion("0.0.0"), buildPeerFeatures()); - const response = await requestOnConnection(probe, { method: "GET", signal: deadline.signal, headers: peerReport }, agent); + const response = await requestOnConnection(probe, { method: "GET", signal: deadline.signal, headers: preflightHeaders }, agent); const socket = agent.socket; const proof = response.headers["x-opencodex-link-proof"]; if (response.statusCode !== 204 || typeof proof !== "string" || !linkRelayProofMatches(proof, challenge.expected)) { @@ -177,15 +240,20 @@ export async function fetchBoundLinkRelay(target: BoundLinkRelayTarget, destinat for await (const chunk of response) { if (chunk.length) throw new LinkRelayAuthenticationError(); } + deadline.signal.throwIfAborted(); deadline.clear(); init.signal?.throwIfAborted(); if (!socket || socket.destroyed || socket !== agent.socket) throw new LinkRelayAuthenticationError(); const headers = new Headers(init.headers); headers.set(LINK_RELAY_SESSION_HEADER, challenge.nonce); + dataRequestStarted = true; const incoming = await requestOnConnection(url, { ...init, headers }, agent, socket); return await responseOnConnection(incoming, agent, init.method ?? "GET"); } catch (error) { + const authDeadlineExpired = !dataRequestStarted && deadline.didExpire() + && !(error instanceof LinkRelayAuthenticationError); await agent.close(); + if (authDeadlineExpired && !init.signal?.aborted) throw new LinkRelayAuthTimeoutError(); throw error; } finally { deadline.clear(); } } diff --git a/src/client/link-relay.ts b/src/client/link-relay.ts index d46e406cdca..2ac599c50e1 100644 --- a/src/client/link-relay.ts +++ b/src/client/link-relay.ts @@ -1,4 +1,5 @@ -import { fetchBoundLinkRelay, LinkRelayAuthenticationError } from "./link-relay-transport"; +import { randomUUID } from "node:crypto"; +import { fetchBoundLinkRelay, LinkRelayAuthenticationError, LinkRelayAuthTimeoutError } from "./link-relay-transport"; import { homeUnreachableBody } from "./home-unreachable"; import { boundedRelayResponseStream, @@ -6,6 +7,7 @@ import { headersWithinLimit, validateHubRelayRequestHeaders, } from "./hub-relay"; +import type { LinkDiagnosticRecord, LinkDiagnosticSink } from "../link/diagnostics"; import { linkRouteAllowed } from "../link/routes"; import { isLinkPort } from "../link/ports"; import { resolveInboundBodyLimitBytes } from "../server/request-decompress"; @@ -26,14 +28,23 @@ export interface LinkRelayClock { clearTimeout: typeof clearTimeout; } +/** A fresh admission verdict plus optional diagnostic evidence; metadata never grants admission. */ +export interface LinkAdmissionResult { + kind: "match" | "mismatch" | "lookup_error" | "deadline" | "stale" | "identity_unknown" | "identity_mismatch" | "unavailable"; + elapsedMs?: number; + generation?: number; + state?: string; + checkId?: string; +} + /** * The Child's tunnel as the relay sees it (the client link supervisor implements it). A request * waits on it only while the tunnel is being (re)established. Every relayed fetch waits for a * fresh bounded asynchronous ownership proof, including after a reconnect or refused retry. */ export interface LinkTunnelGate { - /** True only while a keyed probe has established this live tunnel. */ - connected(): boolean | Promise; + /** Admission and its evidence are returned together; only match permits a fetch. */ + connected(context?: { requestId?: string }): LinkAdmissionResult | Promise; /** True while the tunnel is connecting or reconnecting. */ pending(): boolean; /** @@ -51,12 +62,13 @@ export interface LinkTunnelGate { * supervisor, and a relay with no gate at all still refuses every request. */ export const HOME_INITIATED_LINK_TUNNEL: LinkTunnelGate = { - connected: () => true, + connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => false, }; export interface LinkRelayDeps { + diagnostic?: LinkDiagnosticSink; /** Trusted complete-send test seam. Production always uses the connection-bound transport. */ fetchImpl?: typeof fetch; clock?: LinkRelayClock; @@ -136,6 +148,7 @@ function relayAuthenticationFailed(error: string): Response { }); } +/** Preserve path/query on a fixed IPv4 loopback origin; throw when the tunnel port is invalid. */ export function linkRelayDestination(url: URL, target: Pick): string { if (!isLinkPort(target.tunnelPort)) { throw new RangeError("invalid link tunnel port"); @@ -180,6 +193,7 @@ function linkRequestHeaders( return headers; } +/** Drop hop-by-hop and stale length/encoding headers before exposing the decoded response. */ export function sanitizeLinkResponseHeaders(source: Headers): Headers { const connectionNamed = new Set((source.get("connection") ?? "") .split(",").map(value => value.trim().toLowerCase()).filter(Boolean)); @@ -328,6 +342,11 @@ function idleBoundedStream( }); } +/** + * Admit an allowlisted route and stream it through the authenticated tunnel with bounded bodies. + * Owner/auth preflight deadlines share one recovery attempt before data dispatch; an ambiguous + * post-dispatch failure never earns that recovery. Diagnostic failures cannot change admission. + */ export async function relayLinkDataRequest( req: Request, target: LinkRelayTarget, @@ -352,6 +371,16 @@ export async function relayLinkDataRequest( return jsonError(431, "link relay request headers too large"); } + const requestId = randomUUID(); + let admission: LinkAdmissionResult = { kind: "unavailable" }; + let recoveryAttempts = 0; + const diagnostic = (event: LinkDiagnosticRecord["event"], outcome: LinkDiagnosticRecord["outcome"], elapsedMs?: number) => { + try { deps.diagnostic?.({ event, outcome, requestId, + checkId: admission.checkId, generation: admission.generation, + state: admission.state, elapsedMs: admission.elapsedMs, + ...(elapsedMs !== undefined ? { elapsedMs } : {}), attempt: recoveryAttempts }); } + catch { /* diagnostics never alter admission */ } + }; const relayAbort = new AbortController(); const clock = deps.clock ?? defaultClock; let headerTimer: ReturnType | undefined = clock.setTimeout(() => { @@ -390,20 +419,45 @@ export async function relayLinkDataRequest( const remaining = holdUntil - now; return remaining > 0 && await gate.waitForConnected(remaining, relayAbort.signal); }; - const tunnelUnavailable = async (): Promise => { + const tunnelUnavailable = async ( + reason: LinkDiagnosticRecord["outcome"] = admission.kind === "match" ? "unavailable" : admission.kind, + message = "link tunnel unavailable", + elapsedMs?: number, + ): Promise => { + diagnostic("relay_failure", relayAbort.signal.aborted + ? req.signal.aborted ? "client_abort" : "header_timeout" : reason, elapsedMs); cleanup(); try { await body?.cancel(); } catch { /* best effort */ } - return homeUnreachable("link tunnel unavailable"); + const response = reason === "auth_failed" ? relayAuthenticationFailed(message) : homeUnreachable(message); + response.headers.set("x-opencodex-link-request-id", requestId); + return response; + }; + const retryPreflight = (reason: "deadline" | "auth_timeout", elapsedMs?: number): boolean => { + if (recoveryAttempts >= 1 || relayAbort.signal.aborted || bodyOverflow || bodyCancelled || body?.locked) return false; + recoveryAttempts += 1; + diagnostic("relay_retry", reason, elapsedMs); + return true; + }; + const checkConnected = async (): Promise => { + admission = tunnel ? await tunnel.connected({ requestId }) : { kind: "unavailable" }; + return admission.kind === "match"; }; const readyForFetch = async (): Promise => { - if (!tunnel) return false; - if (await tunnel.connected()) return true; - return tunnel.pending() && await holdForTunnel(tunnel) && await tunnel.connected(); + if (!tunnel || relayAbort.signal.aborted) return false; + if (await checkConnected()) return !relayAbort.signal.aborted; + // No fetch or body read has occurred. A deadline proves neither a foreign owner nor a + // broken SSH process: give this request one fresh proof, without caching or restarting it. + if (admission.kind === "deadline" && retryPreflight("deadline")) { + if (await checkConnected()) return !relayAbort.signal.aborted; + } + return !relayAbort.signal.aborted && tunnel.pending() && await holdForTunnel(tunnel) + && await checkConnected() && !relayAbort.signal.aborted; }; let upstream: Response | undefined; while (!upstream) { if (!await readyForFetch()) return await tunnelUnavailable(); + const fetchStartedAt = performance.now(); try { const init: RequestInit & { duplex?: "half" } = { method: req.method, @@ -415,24 +469,28 @@ export async function relayLinkDataRequest( upstream = await (deps.fetchImpl ? deps.fetchImpl(destination, init) : fetchBoundLinkRelay(target, destination, init)); } catch (error) { if (error instanceof LinkRelayAuthenticationError) { - cleanup(); - try { await body?.cancel(); } catch { /* an upload might already be drained */ } - return relayAuthenticationFailed(error.message); + return await tunnelUnavailable("auth_failed", error.message); } if (bodyOverflow) { cleanup(); return jsonError(413, "link relay request body too large"); } + if (error instanceof LinkRelayAuthTimeoutError) { + // The transport certifies that only its credential-free challenge was sent. A new + // iteration obtains a fresh ownership proof and a new authenticated physical socket. + const elapsedMs = performance.now() - fetchStartedAt; + if (retryPreflight("auth_timeout", elapsedMs)) continue; + return await tunnelUnavailable("auth_timeout", "link tunnel unavailable", elapsedMs); + } // A refused connection sent nothing, and an untouched body can go again once the tunnel // is back inside the same hold window. Anything else is never replayed. const resendable = connectionRefused(error) && (body === null || (!body.locked && !bodyCancelled)); - if (!tunnel || !resendable || !tunnel.pending() || !await holdForTunnel(tunnel)) return await tunnelUnavailable(); + if (!tunnel || !resendable || !tunnel.pending() || !await holdForTunnel(tunnel)) return await tunnelUnavailable("transport_error"); } } if (relayAbort.signal.aborted) { - cleanup(); try { await upstream.body?.cancel(); } catch { /* best effort */ } - return homeUnreachable("link tunnel unavailable"); + return await tunnelUnavailable(); } const sse = isSse(upstream.headers); @@ -449,6 +507,7 @@ export async function relayLinkDataRequest( try { await upstream.body?.cancel(); } catch { /* best effort */ } return jsonError(502, "link relay response body too large"); } + if (recoveryAttempts > 0 && upstream.ok) diagnostic("relay_recovered", "recovered"); if (req.method === "HEAD" || !upstream.body) { cleanup(); try { await upstream.body?.cancel(); } catch { /* best effort */ } diff --git a/src/client/link-tunnel.ts b/src/client/link-tunnel.ts index 4174aeb398d..38e984a46d7 100644 --- a/src/client/link-tunnel.ts +++ b/src/client/link-tunnel.ts @@ -1,3 +1,4 @@ +import { randomUUID } from "node:crypto"; import { chmodSync, mkdirSync, readFileSync, statSync, unlinkSync } from "node:fs"; import { execFile } from "node:child_process"; import { readFile as readFileAsync } from "node:fs/promises"; @@ -13,17 +14,18 @@ import { CLIENT_TUNNEL_RETRY_POLICY, classifySshStderr, dueForSpawn, - failedTunnel, + failedSpawnTunnel, IDLE, reduceTunnel, type StderrClass, type TunnelState, } from "../link/tunnel-state"; import { isLinkPort } from "../link/ports"; -import { ownsIpv4LoopbackListener } from "../server/port-reclaim"; +import { inspectIpv4LoopbackListener, type LoopbackOwnerLookupResult } from "../server/port-reclaim"; +import type { LinkDiagnosticSink } from "../link/diagnostics"; import { isLinkConnection, readClientConnectionState } from "./state"; import { clientLinkStatePath, readClientLinkState, type ClientLinkState } from "./link-state"; -import type { LinkTunnelGate } from "./link-relay"; +import type { LinkTunnelGate, LinkAdmissionResult } from "./link-relay"; /** * The client-owned `ssh -N -L 127.0.0.1::127.0.0.1: ` @@ -134,7 +136,8 @@ export interface ClientLinkSupervisorDeps extends ClientLinkTunnelDeps, OrphanRe /** The link key for the keyed readiness probe. The runtime passes its cached key source. */ linkKey?: () => string | null; /** Async local IPv4 LISTEN ownership; an unknown or foreign owner never receives the link key. */ - ownsLoopbackListener?: (port: number, pid: number) => Promise; + inspectLoopbackListener?: (port: number, pid: number) => Promise; + diagnostic?: LinkDiagnosticSink; fetchImpl?: typeof fetch; now?: () => number; random?: () => number; @@ -563,6 +566,10 @@ async function probeTunnel(fetchImpl: typeof fetch, tunnelPort: number, key: str } } +/** + * Own the Child's SSH lifecycle and reconnect holds. Each connected() call checks fresh listener + * ownership and fences results against tunnel replacement or stop; diagnostics cannot admit it. + */ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): ClientLinkSupervisor { const sidecarPath = (): string => clientLinkStatePath(deps.configDir); const readSidecar = deps.readSidecar ?? readWhenFileChanges(sidecarPath, () => readClientLinkState(sidecarPath())); @@ -573,7 +580,7 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): const policy = CLIENT_TUNNEL_RETRY_POLICY; const isAlive = deps.isAlive ?? defaultIsAlive; const linkKey = deps.linkKey ?? (() => null); - const ownerLookup = deps.ownsLoopbackListener ?? ownsIpv4LoopbackListener; + const ownerLookup = deps.inspectLoopbackListener ?? inspectIpv4LoopbackListener; const fetchImpl = deps.fetchImpl ?? fetch; const setSupervisorTimer = deps.setTimer ?? ((callback: () => void, ms: number) => { const interval = setInterval(callback, ms); @@ -620,9 +627,16 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): if (pid === null || pid === undefined || (adopted !== null && !adoptedRecord)) return null; return { pid, generation: ownerGeneration }; }; - const listenerOwnedByTunnel = async (port: number): Promise => { + const listenerOwnedByTunnel = async (port: number, context?: { requestId?: string }): Promise => { const candidate = ownerCandidate(port); - if (!candidate) return false; + if (!candidate) return { kind: "unavailable" }; + const checkId = randomUUID(); + const report = (kind: LinkAdmissionResult["kind"], elapsedMs = 0): LinkAdmissionResult => { + const check = { kind, elapsedMs, checkId, generation: candidate.generation, state: state.kind }; + if (kind !== "match") try { deps.diagnostic?.({ event: "owner_lookup", outcome: kind, requestId: context?.requestId, + checkId, generation: candidate.generation, state: state.kind, elapsedMs }); } catch { /* best effort */ } + return check; + }; if (adopted !== null) { const record = adoptedRecord; const identity = record ? await inspectAdoptedIdentity(record, deps) : "unknown"; @@ -640,16 +654,28 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): invalidateOwnerProof(); setState(IDLE); } - return false; + return report(identity === "mismatch" ? "identity_mismatch" : "identity_unknown"); } } - const owned = await Promise.resolve().then(() => ownerLookup(port, candidate.pid)).catch(() => false); - return owned && !stopping && ownerGeneration === candidate.generation - && ownerCandidate(port)?.pid === candidate.pid; + const result = await Promise.resolve().then(() => ownerLookup(port, candidate.pid)) + .catch((): LoopbackOwnerLookupResult => ({ kind: "lookup_error", elapsedMs: 0 })); + if (stopping || failure !== undefined || ownerGeneration !== candidate.generation || ownerCandidate(port)?.pid !== candidate.pid) { + return report("stale", result.elapsedMs); + } + return report(result.kind, result.elapsedMs); + }; + const connected = async (context?: { requestId?: string }): Promise => { + if (!started || stopping || failure !== undefined || state.kind !== "connected" || linkId === null || tunnelPort === null) { + return { kind: "unavailable" }; + } + const result = await listenerOwnedByTunnel(tunnelPort, context); + // State may change during the asynchronous lookup or an optional diagnostic callback. + if (result.kind === "match" && (!started || stopping || failure !== undefined + || state.kind !== "connected" || result.generation !== ownerGeneration)) { + return { ...result, kind: "stale" }; + } + return result; }; - const connected = async (): Promise => started && !stopping && failure === undefined - && state.kind === "connected" && linkId !== null && tunnelPort !== null - && await listenerOwnedByTunnel(tunnelPort); const settleWaiters = (): void => { if (waiters.size === 0) return; @@ -733,7 +759,8 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): peerListenerPort: sidecar.peerListenerPort, }, deps); } catch { - setState(failedTunnel("forward", timestamp, policy, state.kind === "failed" ? state.since : timestamp)); + // A due retry that cannot start counts as a failed retry, so the backoff keeps its step. + setState(failedSpawnTunnel(state, "forward", timestamp, policy)); deps.warn?.("client link tunnel could not be started"); return; } @@ -800,7 +827,7 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): probeFlight = flight; const wasConnected = state.kind === "connected"; void (async () => { - if (!await listenerOwnedByTunnel(tunnelPort)) return; + if ((await listenerOwnedByTunnel(tunnelPort)).kind !== "match") return; if (probeFlight !== flight || stopping || flight.abort.signal.aborted || child !== flight.child || adopted !== flight.adopted) return; const key = linkKey(); @@ -808,7 +835,7 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): const result = await probeTunnel(fetchImpl, tunnelPort, key, flight.abort.signal); // An aborted probe, or one whose listener changed while it ran, proves nothing. if (probeFlight !== flight || stopping || child !== flight.child || adopted !== flight.adopted - || !await listenerOwnedByTunnel(tunnelPort)) return; + || (await listenerOwnedByTunnel(tunnelPort)).kind !== "match") return; applyProbe(result, wasConnected); })().catch(() => undefined).finally(() => { if (probeFlight === flight) probeFlight = undefined; @@ -971,7 +998,7 @@ export function createClientLinkSupervisor(deps: ClientLinkSupervisorDeps = {}): pending, connected, waitForConnected(timeoutMs: number, signal?: AbortSignal): Promise { - if (state.kind === "connected") return connected(); + if (state.kind === "connected") return connected().then(result => result.kind === "match"); if (!pending() || waiters.size >= CLIENT_LINK_MAX_HOLDS || signal?.aborted || !(timeoutMs > 0)) { return Promise.resolve(false); } diff --git a/src/client/machine-listener.ts b/src/client/machine-listener.ts index 5009507d641..554d0488554 100644 --- a/src/client/machine-listener.ts +++ b/src/client/machine-listener.ts @@ -1,3 +1,4 @@ +import { recordLinkDiagnostic } from "../link/diagnostics"; import type { Server } from "bun"; import { loadConfig } from "../config"; import { browserSecurityHeaders, requestPolicyView } from "../server/auth-cors"; @@ -21,6 +22,7 @@ import type { LinkTunnelGate } from "./link-relay"; import { readClientLinkState, type ClientLinkState } from "./link-state"; import { projectClientLinkChild, type ClientLinkSidecarRead } from "./link-status"; import type { ClientLinkSupervisorStatus } from "./link-tunnel"; +import type { LinkReadiness } from "./link-readiness"; import { packageVersion } from "../lib/package-version"; import { createLocalAttestationProof, LOCAL_ATTESTATION_CHALLENGE_HEADER, LOCAL_ATTESTATION_PROOF_HEADER } from "../lib/local-management-attestation"; import { @@ -65,6 +67,11 @@ export interface MachineListenerDeps { linkTunnel?: LinkTunnelGate; /** Link mode: relay seams (deadline clock and byte cap). */ linkRelay?: LinkIngress["relay"]; + /** + * Link mode: whether relayed requests can be served now. `GET /readyz` answers 503 `pending` + * with `link.reason` while it is not; `/healthz` is unaffected (`link-readiness.ts`). + */ + linkReadiness?: () => LinkReadiness | Promise; /** * The bound restart (OCX-6). With it, `/healthz` proves this process's identity to a challenge, * advertises `restartCapability` and `startedAt`, and `POST /api/system/restart` accepts a @@ -152,7 +159,7 @@ export function startMachineListener( apiKeyId: connection.apiKeyId, policy: requestPolicyView(config, "127.0.0.1"), linkKey: deps.linkKeySource ?? createLinkKeySource(connection.tokenFingerprint, deps.linkKey), - relay: { fetchImpl: deps.fetchImpl, bodyLimitBytes: inboundBodyLimit, tunnel: deps.linkTunnel, ...deps.linkRelay }, + relay: { diagnostic: recordLinkDiagnostic, fetchImpl: deps.fetchImpl, bodyLimitBytes: inboundBodyLimit, tunnel: deps.linkTunnel, ...deps.linkRelay }, } : null; return (deps.serve ?? (options => Bun.serve(options)))({ @@ -186,7 +193,18 @@ export function startMachineListener( return await handleBoundRestart(req, server, deps.restart); } if (url.pathname === "/readyz" && req.method === "GET") { - return Response.json({ service: "opencodex", version: VERSION, role: "client", status: "ready", uptime: process.uptime(), pid: process.pid, port: server.port, protocolVersion: 1 }); + const identity = { service: "opencodex", version: VERSION, role: "client" } as const; + const facts = { uptime: process.uptime(), pid: process.pid, port: server.port, protocolVersion: 1 }; + let link: LinkReadiness | null = null; + if (linkMode && deps.linkReadiness) { + try { link = await deps.linkReadiness(); } catch { link = { ready: false, reason: "link_unverified" }; } + } + if (link && !link.ready) { + // The /readyz contract: not ready is 503 with Retry-After and status pending or failed. + return Response.json({ ...identity, status: "pending", ...facts, link: { state: "unavailable", reason: link.reason } }, + { status: 503, headers: { "Retry-After": "1", "Cache-Control": "no-store" } }); + } + return Response.json({ ...identity, status: "ready", ...facts, ...(link ? { link: { state: "available" } } : {}) }); } if (url.pathname.startsWith("/api/machine/hub-relay/")) { if (!relayEnabled) return json404(req); diff --git a/src/client/runtime.ts b/src/client/runtime.ts index cb12800eb14..631e35c19e7 100644 --- a/src/client/runtime.ts +++ b/src/client/runtime.ts @@ -1,3 +1,4 @@ +import { flushLinkDiagnostics, recordLinkDiagnostic } from "../link/diagnostics"; import { existsSync } from "node:fs"; import type { Server } from "bun"; import { siblingRuntimeField, withSiblingMarker } from "../codex/sibling-start"; @@ -20,6 +21,7 @@ import type { ReplacementStartRequest } from "../server/restart-replacement"; import type { OcxClientConnectionConfig } from "../types"; import { createLinkKeySource } from "./link-ingress"; import { HOME_INITIATED_LINK_TUNNEL } from "./link-relay"; +import { clientInitiatedLinkReadiness, createHomeInitiatedLinkReadiness } from "./link-readiness"; import { createClientLinkSupervisor, type ClientLinkSupervisor } from "./link-tunnel"; import { clientLinkStatePath, isChildInitiatedLink, recordChildInitiatedLink } from "./link-state"; import { startMachineListener, type MachineListenerDeps } from "./machine-listener"; @@ -164,6 +166,7 @@ export async function recycleStandalone( console.warn(`[client] listener stop failed: ${error instanceof Error ? error.message : String(error)}`); } cleanup(); + await flushLinkDiagnostics(); // Recycling back to standalone after `ocx disconnect` must actually bring a standalone // proxy back, under every launch shape. // @@ -227,7 +230,7 @@ export async function bindClientListener( preferred: number; explicitPort: boolean; configuredPort: number; - } & Pick, + } & Pick, io: ClientRuntimeIo = {}, ): Promise<{ server: Server; port: number }> { const { linkMode } = request; @@ -267,6 +270,7 @@ export async function bindClientListener( ...(request.linkStatus ? { linkStatus: request.linkStatus } : {}), ...(request.linkKeySource ? { linkKeySource: request.linkKeySource } : {}), ...(request.linkTunnel ? { linkTunnel: request.linkTunnel } : {}), + ...(request.linkReadiness ? { linkReadiness: request.linkReadiness } : {}), ...(request.restart ? { restart: request.restart } : {}), }); return { server, port: server.port ?? port }; @@ -318,10 +322,19 @@ export async function startClientRuntime( const serve = async (rollback?: { port: number; portWaitMs: number }): Promise => { const supervisor = supervised ? createClientLinkSupervisor({ + diagnostic: recordLinkDiagnostic, onLinkEnded: () => scheduleStandaloneRecycle(state.value.tokenFingerprint), linkKey, }) : null; + // What `/readyz` reports about the link: the supervisor's tunnel, or for a Home-initiated + // link the shared relay preflight. A link with neither refuses every relayed request. + const link = state.value.link; + const linkReadiness: MachineListenerDeps["linkReadiness"] = !linkMode ? undefined + : supervisor ? () => clientInitiatedLinkReadiness(supervisor.status()) + : homeInitiated && link && linkKey + ? createHomeInitiatedLinkReadiness({ target: { tunnelPort: link.tunnelPort, linkId: link.linkId, apiKeyId: state.value.apiKeyId }, linkKey }) + : () => ({ ready: false, reason: "tunnel_unavailable" }); const { server, port: boundPort } = await bindClientListener({ state: state.value, linkMode, @@ -332,6 +345,7 @@ export async function startClientRuntime( restart: { attestationSecret, startedAt: CLIENT_RUNTIME_STARTED_AT, request: () => restartController.request() }, ...(linkMode ? { linkStatus: () => activeSupervisor?.status() ?? { kind: "stopped" as const }, linkKeySource: linkKey } : {}), ...(supervisor ? { linkTunnel: supervisor } : homeInitiated ? { linkTunnel: HOME_INITIATED_LINK_TUNNEL } : {}), + ...(linkReadiness ? { linkReadiness } : {}), }, rollback === undefined ? io : { ...io, portWaitMs: rollback.portWaitMs }); const publish = (): void => { activeServer = server; @@ -417,7 +431,10 @@ export async function startClientRuntime( await recycleStandalone(state.value.tokenFingerprint); } }, - exitProcess: code => { process.exit(code); }, + exitProcess: async code => { + await flushLinkDiagnostics(); + process.exit(code); + }, }); installCrashGuards(); @@ -444,6 +461,7 @@ export async function startClientRuntime( console.warn(`[client] listener stop failed: ${error instanceof Error ? error.message : String(error)}`); } finally { cleanup(); + await flushLinkDiagnostics(); // launchd relaunches only unsuccessful exits; an external signal must still count. process.exit(handledSignalExitCode(signal)); } diff --git a/src/link/diagnostics.ts b/src/link/diagnostics.ts new file mode 100644 index 00000000000..f42f5d681a7 --- /dev/null +++ b/src/link/diagnostics.ts @@ -0,0 +1,193 @@ +import { constants } from "node:fs"; +import { chmod, lstat, mkdir, open, rename, unlink } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { assertNotRealHomeUnderTest } from "../lib/test-home-guard"; +import { hardenSecretDirAsync, hardenSecretPathAsync, reattributeHardenedSecretPath, windowsSecretAclApplies } from "../lib/windows-secret-acl"; +import { linkDir } from "./paths"; +import type { TunnelState } from "./tunnel-state"; + +export type LinkDiagnosticRecord = { + event: "owner_lookup" | "relay_retry" | "relay_failure" | "relay_recovered" | "tunnel_transition"; + outcome: "match" | "mismatch" | "lookup_error" | "deadline" | "stale" | "unavailable" + | "identity_unknown" | "identity_mismatch" | "auth_timeout" | "auth_failed" + | "transport_error" | "client_abort" | "header_timeout" | "recovered" + // tunnel_transition: the Home's -R tunnel changed state, and why (src/link/tunnel-history.ts). + | "connected" | "spawned" | "stopped" | "auth" | "hostkey" | "forward" | "remote_forward_conflict" + | "timeout" | "network" | "unknown" | "spawn_failed" | "listener_unavailable"; + requestId?: string; + checkId?: string; + generation?: number; + state?: string; + previousState?: string; + elapsedMs?: number; + attempt?: number; + tunnelPort?: number; + retryAfterMs?: number; +}; +export type LinkDiagnosticSink = (record: LinkDiagnosticRecord) => void; + +const EVENTS = new Set(["owner_lookup", "relay_retry", "relay_failure", "relay_recovered", "tunnel_transition"]); +const OUTCOMES = new Set([ + "match", "mismatch", "lookup_error", "deadline", "stale", "unavailable", "identity_unknown", + "identity_mismatch", "auth_timeout", "auth_failed", "transport_error", "client_abort", "header_timeout", "recovered", + "connected", "spawned", "stopped", "auth", "hostkey", "forward", "remote_forward_conflict", "timeout", "network", + "unknown", "spawn_failed", "listener_unavailable", +]); +const STATES = new Set(["idle", "connecting", "connected", "reconnecting", "failed"]); +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; +const MAX_BYTES = 256 * 1024; + +/** Copy only bounded enum, UUID and numeric fields; unknown fields never reach disk. */ +function metadata(record: LinkDiagnosticRecord): LinkDiagnosticRecord | null { + if (!record) return null; + const { event, outcome, state, previousState } = record; + if (!EVENTS.has(event) || !OUTCOMES.has(outcome)) return null; + const result: LinkDiagnosticRecord = { event, outcome }; + for (const key of ["requestId", "checkId"] as const) { + const value = record[key]; + if (typeof value === "string" && value.length === 36 && UUID.test(value)) result[key] = value; + } + if (typeof state === "string" && STATES.has(state as TunnelState["kind"])) result.state = state; + if (typeof previousState === "string" && STATES.has(previousState as TunnelState["kind"])) result.previousState = previousState; + for (const key of ["generation", "elapsedMs", "attempt", "retryAfterMs"] as const) { + const value = record[key]; + if (typeof value === "number" && Number.isFinite(value) && value >= 0 && value <= Number.MAX_SAFE_INTEGER + && (key === "elapsedMs" || key === "retryAfterMs" || Number.isSafeInteger(value))) result[key] = value; + } + const { tunnelPort } = record; + if (typeof tunnelPort === "number" && Number.isInteger(tunnelPort) && tunnelPort >= 1 && tunnelPort <= 65_535) result.tunnelPort = tunnelPort; + return result; +} + +/** Append through a private no-follow handle, keeping at most one 256 KiB rotated backup. */ +async function append(dir: string, line: string): Promise { + assertNotRealHomeUnderTest(dirname(dir)); + await mkdir(dir, { recursive: true, mode: 0o700 }); + const directory = await lstat(dir); + if (!directory.isDirectory() || directory.isSymbolicLink()) return; + const windows = windowsSecretAclApplies(); + if (windows) await hardenSecretDirAsync(dir, { required: true }); + else await chmod(dir, 0o700); + const path = join(dir, "diagnostics.jsonl"); + const backup = `${path}.1`; + let file: Awaited> | undefined; + try { + // No read of log contents, and no following a file symlink or writing to a pipe. + const flags = constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW | constants.O_NONBLOCK; + file = await open(path, flags, 0o600); + const current = await file.stat(); + if (!current.isFile() || current.nlink !== 1) return; + if (windows) await hardenSecretPathAsync(path, { required: true }); + else await file.chmod(0o600); + const previous = await lstat(backup).catch(error => { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + return null; + }); + if (previous) { + if (!previous.isFile() || previous.isSymbolicLink() || previous.nlink !== 1) return; + if (previous.size > MAX_BYTES) await unlink(backup); + else if (windows) await hardenSecretPathAsync(backup, { required: true }); + else await chmod(backup, 0o600); + } + if (current.size > MAX_BYTES) await file.truncate(0); + else if (current.size + Buffer.byteLength(line) > MAX_BYTES) { + await file.close(); + file = undefined; + await rename(path, backup); + file = await open(path, flags, 0o600); + const fresh = await file.stat(); + if (!fresh.isFile() || fresh.nlink !== 1 || fresh.size + Buffer.byteLength(line) > MAX_BYTES) return; + if (windows) await hardenSecretPathAsync(path, { required: true }); + else await file.chmod(0o600); + } + await file.write(line); + if (windows) { + const written = await file.stat({ bigint: true }); + const named = await lstat(path, { bigint: true }); + if (!named.isSymbolicLink() && named.dev === written.dev && named.ino === written.ino) { + // Windows publishes content-write ChangeTime on close; retain only the same object's ACL memo. + await file.close(); + file = undefined; + reattributeHardenedSecretPath(path); + } + } + } finally { + await file?.close(); + } +} + +/** + * Create a best-effort metadata sink with one-second event/outcome buckets and a bounded queue. + * File and ACL work starts on the first eligible record and never blocks admission. flush() waits + * at most one second; its timeout does not cancel the writer or guarantee durable storage. + */ +export function createLinkDiagnosticSink( + { configDir, now = Date.now }: { configDir?: string; now?: () => number } = {}, +): LinkDiagnosticSink & { flush: (timeoutMs?: number) => Promise } { + // At most 56 buckets; correlation IDs never become keys or retained memory. + const buckets = new Map(); + const queue: Array<{ dir: string; line: string }> = []; + let flight: Promise | undefined; + const drain = async () => { + while (queue.length) { + const record = queue.shift()!; + try { await append(record.dir, record.line); } catch { /* Best effort; never publish raw errors. */ } + } + }; + const startWriter = () => { + if (flight) return; + flight = drain().finally(() => { + flight = undefined; + if (queue.length) startWriter(); + }); + }; + const sink: LinkDiagnosticSink = record => { + try { + const safe = metadata(record); + if (!safe) return; + const at = now(); + if (!Number.isSafeInteger(at) || at < 0) return; + const key = `${safe.event}:${safe.outcome}`; + const bucket = buckets.get(key); + // Wall-clock rollback starts a new window, even when the writer is backed up. + if (bucket && at >= bucket.at && at - bucket.at < 1_000) { + bucket.suppressed = Math.min(Number.MAX_SAFE_INTEGER, bucket.suppressed + 1); + return; + } + if (queue.length >= 56) { + buckets.set(key, { at: Math.min(bucket?.at ?? at, at), + suppressed: Math.min(Number.MAX_SAFE_INTEGER, (bucket?.suppressed ?? 0) + 1) }); + return; + } + const suppressed = bucket?.suppressed ?? 0; + // Failed writes are rate-limited too; diagnostics cannot hammer an unavailable disk. + buckets.set(key, { at, suppressed: 0 }); + queue.push({ dir: linkDir(configDir), line: `${JSON.stringify({ ...safe, timestamp: at, ...(suppressed ? { suppressed } : {}) })}\n` }); + startWriter(); + } catch { /* Diagnostics must never change request admission or expose raw errors. */ } + }; + return Object.assign(sink, { flush: async (timeoutMs = 1_000) => { + if (!flight) return; + // Keep shutdown bounded even if an asynchronous filesystem or ACL operation stalls. + // Timeout leaves the existing writer alone; it may still finish before process exit. + let timer: ReturnType | undefined; + try { + await Promise.race([ + (async () => { while (flight) await flight; })(), + new Promise(resolve => { timer = setTimeout(resolve, Math.min(1_000, Math.max(0, timeoutMs))); }), + ]); + } catch { /* Best effort; never publish raw errors. */ } + finally { clearTimeout(timer); } + } }); +} + +let defaultSink: ReturnType | undefined; +/** Lazily enqueue sanitized evidence in the default sink without propagating diagnostic failures. */ +export function recordLinkDiagnostic(record: LinkDiagnosticRecord): void { + try { (defaultSink ??= createLinkDiagnosticSink())(record); } catch { /* Best effort. */ } +} + +/** Drain existing diagnostics for at most one second without activating an unused sink. */ +export async function flushLinkDiagnostics(): Promise { + await defaultSink?.flush(); +} diff --git a/src/link/relay-auth.ts b/src/link/relay-auth.ts index e67c7273642..741d321ef63 100644 --- a/src/link/relay-auth.ts +++ b/src/link/relay-auth.ts @@ -2,6 +2,12 @@ import { createHmac, randomBytes, timingSafeEqual } from "node:crypto"; export const LINK_RELAY_AUTH_PATH = "/.well-known/opencodex/link-relay-auth"; export const LINK_RELAY_SESSION_HEADER = "x-opencodex-link-session"; +/** + * Marks a challenge that only asks whether the Home answers (the Child's readiness preflight): the + * Home proves itself without reserving a session, so no data request can follow it and it holds + * neither session capacity nor a drain lease. An older Home ignores it and reserves as before. + */ +export const LINK_RELAY_PROBE_HEADER = "x-opencodex-link-probe"; export const LINK_RELAY_AUTH_VERSION = "2"; export const LINK_RELAY_AUTH_TIMEOUT_MS = 5_000; diff --git a/src/link/status-projection.ts b/src/link/status-projection.ts index c87e33bfec3..9b08dbfbd5e 100644 --- a/src/link/status-projection.ts +++ b/src/link/status-projection.ts @@ -22,6 +22,8 @@ export interface LinkStatusDto { tunnelPort: number; /** Home only: what the link carried from this Child, and the compatibility verdict. */ peer?: LinkPeerStatus; + /** Home-initiated links on request (`?diagnostics=1`): retry progress and the last failure. */ + tunnel?: LinkTunnelDiagnostics; }>; child: null | { alias: string; @@ -33,6 +35,43 @@ export interface LinkStatusDto { local?: LinkLocalPeer; } +/** + * The Home's account of one `-R` tunnel: ssh attempts since it was last connected, when the next + * one is due (null while one runs or none is scheduled), the last state change and the last + * failure. `cause` is a fixed word from `tunnel-history.ts`, never ssh output. + */ +export interface LinkTunnelDiagnostics { + attempt: number; + nextRetryAt: string | null; + lastTransition: { from: LinkWireState; to: LinkWireState; at: string; cause: string | null } | null; + lastFailure: { cause: string; at: string } | null; +} + +function iso(at: number): string { + return new Date(at).toISOString(); +} + +/** Add `tunnel` to each Home-initiated link the supervisor reports. */ +export function attachLinkTunnelDiagnostics(dto: LinkStatusDto, supervisorStates: readonly LinkTunnelStatus[]): LinkStatusDto { + const byId = new Map(supervisorStates.map(status => [status.linkId, status])); + for (const link of dto.links) { + const status = byId.get(link.id); + if (!status || status.direction !== "hub-initiated") continue; + const { state, history } = status; + const due = (state.kind === "failed" || state.kind === "reconnecting") && state.retryAt !== undefined && state.inFlight !== true + ? state.retryAt : null; + link.tunnel = { + attempt: history?.attempt ?? 0, + nextRetryAt: due === null ? null : iso(due), + lastTransition: history?.lastTransition + ? { from: history.lastTransition.from, to: history.lastTransition.to, at: iso(history.lastTransition.at), cause: history.lastTransition.cause } + : null, + lastFailure: history?.lastFailure ? { cause: history.lastFailure.cause, at: iso(history.lastFailure.at) } : null, + }; + } + return dto; +} + /** Add this build's identity and, on a Home, each Child's observed peer status. */ export function attachLinkPeers(dto: LinkStatusDto, local: LinkLocalPeer, observation: (linkId: string) => LinkPeerEntry): LinkStatusDto { if (dto.role === "home") { diff --git a/src/link/supervisor.ts b/src/link/supervisor.ts index 228ed0a24ae..92fb71302d5 100644 --- a/src/link/supervisor.ts +++ b/src/link/supervisor.ts @@ -4,13 +4,16 @@ import { atomicWriteFile } from "../config/atomic-write"; import { linkDir, linkKnownHostsPath, linkStorePath } from "./paths"; import { buildTunnelArgv } from "./ssh-argv"; import { createSshRunner, type SshChild, type SshRunner } from "./ssh-runner"; +import type { LinkDiagnosticSink } from "./diagnostics"; +import { createTunnelHistoryBook, exitCause, type TunnelCause, type TunnelHistory } from "./tunnel-history"; import { classifySshStderr, dueForSpawn, - failedTunnel, + failedSpawnTunnel, HOME_TUNNEL_RETRY_POLICY, IDLE, reduceTunnel, + type TunnelFailure, type TunnelState, } from "./tunnel-state"; import { emptyLinkStore, readLinkStore, type LinkRecord, type LinkStore } from "./store"; @@ -22,6 +25,8 @@ export type LinkTunnelStatus = state: TunnelState; pid: number | null; orphan?: "orphan-unverified"; + /** Attempts, last change and last failure cause of this tunnel (`tunnel-history.ts`). */ + history?: TunnelHistory; } | { linkId: string; @@ -72,6 +77,10 @@ export interface LinkSupervisorDeps { apiKeys?: () => readonly { id: string; name: string }[]; revokeApiKey?: (id: string) => boolean; warn?: (message: string) => void; + /** Where tunnel state changes are recorded; production passes the bounded diagnostics file. */ + diagnostic?: LinkDiagnosticSink; + /** Bounded drain of `diagnostic` at stop. */ + flushDiagnostics?: () => Promise; } const TIMER_MS = 1_000; @@ -166,6 +175,10 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi /** The generation of the child whose exit is still being classified, per link. */ const exiting = new Map(); const recordInstances = new Map(); + const history = createTunnelHistoryBook({ + diagnostic: deps.diagnostic, + tunnelPort: linkId => store.links.find(link => link.id === linkId)?.tunnelPort, + }); const reconcileApiKeys = (): void => { const keys = deps.apiKeys?.() ?? []; @@ -218,12 +231,22 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi } }; - const setEvent = (linkId: string, event: Parameters[1]): TunnelState => { - const next = reduceTunnel(states.get(linkId) ?? IDLE, event, deps.random, HOME_TUNNEL_RETRY_POLICY); + /** Every state change goes through here, so the history sees each transition and its cause. */ + const put = (linkId: string, next: TunnelState, cause: TunnelCause | null = null): TunnelState => { + const before = states.get(linkId) ?? IDLE; states.set(linkId, next); + history.observe(linkId, before, next, cause, now()); return next; }; + const setEvent = (linkId: string, event: Parameters[1], cause: TunnelCause | null = null): TunnelState => + put(linkId, reduceTunnel(states.get(linkId) ?? IDLE, event, deps.random, HOME_TUNNEL_RETRY_POLICY), cause); + + /** Fail a link whose attempt could not start; a due retry that fails this way counts as one. */ + const failAttempt = (linkId: string, prior: TunnelState | undefined, reason: TunnelFailure, cause: TunnelCause): void => { + put(linkId, failedSpawnTunnel(prior, reason, now(), HOME_TUNNEL_RETRY_POLICY), cause); + }; + const spawnFor = (record: LinkRecord): void => { if (stopping || holds.has(record.id) || record.direction !== "hub-initiated" || children.has(record.id)) return; // Only a fresh link or a due attempt spawns. A reload must neither skip the backoff nor start @@ -231,7 +254,8 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi const prior = states.get(record.id); if (prior && prior.kind !== "idle" && !dueForSpawn(prior, now())) return; if (store.listenerPort === null) { - states.set(record.id, { kind: "failed", since: now(), reason: "forward" }); + // Retried like any forward failure; a reload that brings the listener back starts it fresh. + failAttempt(record.id, prior, "forward", "listener_unavailable"); return; } let argv: string[]; @@ -250,6 +274,7 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi exiting.delete(record.id); children.set(record.id, { child, argv, spawnedAt: now() }); orphanUnverified.delete(record.id); + history.spawned(record.id); setEvent(record.id, { type: "spawn", now: now() }); writePidfile(pidfilePath(record.id), { version: 1, linkId: record.id, pid: child.pid, argv }); void child.exited.then(async () => { @@ -264,7 +289,8 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi try { stderr = child.stderr ? await child.stderr : ""; } catch { /* classified as unknown */ } // A reconnect may have started a fresh child meanwhile; its own lifecycle owns the state. if (generations.get(record.id) !== generation) return; - setEvent(record.id, { type: "exit", now: now(), stderrClass: classifySshStderr(stderr) }); + const stderrClass = classifySshStderr(stderr); + setEvent(record.id, { type: "exit", now: now(), stderrClass }, exitCause(stderrClass)); } finally { if (exiting.get(record.id) === generation) exiting.delete(record.id); } @@ -272,11 +298,10 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi if (children.get(record.id)?.child !== child) return; children.delete(record.id); conditionalRemovePidfile(record.id, child.pid); - if (!stopping) setEvent(record.id, { type: "exit", now: now(), stderrClass: "network" }); + if (!stopping) setEvent(record.id, { type: "exit", now: now(), stderrClass: "network" }, "network"); }); } catch { - const at = now(); - states.set(record.id, failedTunnel("forward", at, HOME_TUNNEL_RETRY_POLICY, prior?.kind === "failed" ? prior.since : at)); + failAttempt(record.id, prior, "forward", "spawn_failed"); } }; @@ -300,7 +325,7 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi setEvent(record.id, { type: "ready", now: current }); continue; } - const next = setEvent(record.id, { type: "tick", now: current }); + const next = setEvent(record.id, { type: "tick", now: current }, "timeout"); if (next.kind === "failed" && !next.inFlight && child && retryInFlight(before)) { // A retry started just before the outage limit has not had its grace yet. It carries on as // the failed state's retry instead of being killed with a link that may already be back. @@ -349,7 +374,7 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi const stopChild = async (linkId: string): Promise => { const current = children.get(linkId); if (!current) { - states.set(linkId, IDLE); + put(linkId, IDLE); return; } const generation = generations.get(linkId); @@ -358,7 +383,7 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi await current.child.exited; conditionalRemovePidfile(linkId, current.child.pid); // A stop or removal reservation made while this child exited owns the next state. - if (generations.get(linkId) === generation) states.set(linkId, IDLE); + if (generations.get(linkId) === generation) put(linkId, IDLE); }; const invalidateLink = (linkId: string): void => { @@ -407,6 +432,7 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi if (nextInstances.has(linkId) && !changed.has(linkId)) continue; await stopChild(linkId); states.delete(linkId); + history.forget(linkId); orphanUnverified.delete(linkId); recordInstances.delete(linkId); if (stopping) return; @@ -414,9 +440,13 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi for (const record of nextStore.links) { if (changed.has(record.id)) { states.delete(record.id); + history.forget(record.id); orphanUnverified.delete(record.id); } } + for (const linkId of [...states.keys()]) { + if (!nextInstances.has(linkId)) history.forget(linkId); + } recordInstances.clear(); for (const [linkId, instance] of nextInstances) recordInstances.set(linkId, instance); for (const record of nextStore.links) { @@ -471,12 +501,14 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi return { linkId: record.id, direction: record.direction, state: "client-owned", pid: null }; } const child = children.get(record.id); + const recorded = history.get(record.id); return { linkId: record.id, direction: record.direction, state: states.get(record.id) ?? IDLE, pid: child?.child.pid ?? null, ...(orphanUnverified.has(record.id) ? { orphan: "orphan-unverified" as const } : {}), + ...(recorded ? { history: recorded } : {}), }; }); }, @@ -489,9 +521,10 @@ export function createLinkSupervisor(deps: LinkSupervisorDeps = {}): LinkSupervi await runLifecycle(async () => { await Promise.all([...children.keys()].map(stopChild)); for (const record of store.links) { - if (record.direction === "hub-initiated") states.set(record.id, IDLE); + if (record.direction === "hub-initiated") put(record.id, IDLE); } }); + try { await deps.flushDiagnostics?.(); } catch { /* best effort */ } }, }; } diff --git a/src/link/tunnel-history.ts b/src/link/tunnel-history.ts new file mode 100644 index 00000000000..2a70bfaec45 --- /dev/null +++ b/src/link/tunnel-history.ts @@ -0,0 +1,109 @@ +/** + * What the Home remembers about one `-R` tunnel besides its current state: how many ssh attempts + * the current outage has cost, the last state change and the last failure with its cause. It lives + * in memory next to the supervisor's state, appears in `GET /api/link/status` as `tunnel`, and + * each change it records also goes to the bounded diagnostics file (`diagnostics.ts`), so a link + * that failed for an hour leaves evidence of why after the fact. + * + * Nothing here carries ssh output, argv, host names or keys: a cause is one of a fixed set of + * words, and times are numbers. + */ +import type { LinkDiagnosticSink } from "./diagnostics"; +import type { StderrClass, TunnelState } from "./tunnel-state"; + +export type TunnelCause = + | "auth" | "hostkey" | "forward" | "remote_forward_conflict" | "timeout" | "network" | "unknown" + | "spawn_failed" | "listener_unavailable"; + +export const TUNNEL_CAUSES: readonly TunnelCause[] = [ + "auth", "hostkey", "forward", "remote_forward_conflict", "timeout", "network", "unknown", + "spawn_failed", "listener_unavailable", +]; + +export interface TunnelTransitionRecord { + from: TunnelState["kind"]; + to: TunnelState["kind"]; + at: number; + cause: TunnelCause | null; +} + +export interface TunnelHistory { + /** ssh attempts started since the link was last connected (0 while connected). */ + attempt: number; + lastTransition: TunnelTransitionRecord | null; + lastFailure: { cause: TunnelCause; at: number } | null; +} + +export function exitCause(stderrClass: StderrClass): TunnelCause { + return stderrClass === "remote_forward" ? "remote_forward_conflict" : stderrClass; +} + +/** + * A retry of a failed link ended without connecting: one that ran while the state read failed, or + * one that could not even start (the retry count still grew). + */ +function retryFailedAgain(before: TunnelState, after: TunnelState): boolean { + if (before.kind !== "failed" || after.kind !== "failed" || after.inFlight === true) return false; + return before.inFlight === true || (after.retries ?? 0) > (before.retries ?? 0); +} + +export interface TunnelHistoryBook { + /** Count one ssh attempt for the link. */ + spawned(linkId: string): void; + /** Record a state change; `cause` names why the link left a working state, when known. */ + observe(linkId: string, before: TunnelState, after: TunnelState, cause: TunnelCause | null, at: number): void; + get(linkId: string): TunnelHistory | undefined; + forget(linkId: string): void; +} + +export function createTunnelHistoryBook( + options: { diagnostic?: LinkDiagnosticSink; tunnelPort?: (linkId: string) => number | undefined } = {}, +): TunnelHistoryBook { + const histories = new Map(); + const entry = (linkId: string): TunnelHistory => { + let history = histories.get(linkId); + if (!history) { + history = { attempt: 0, lastTransition: null, lastFailure: null }; + histories.set(linkId, history); + } + return history; + }; + return { + spawned(linkId) { + entry(linkId).attempt += 1; + }, + observe(linkId, before, after, cause, at) { + const refailed = retryFailedAgain(before, after); + if (before.kind === after.kind && !refailed) return; + const history = entry(linkId); + if (after.kind === "connected" || after.kind === "idle") history.attempt = 0; + if (cause && (after.kind === "failed" || after.kind === "reconnecting")) history.lastFailure = { cause, at }; + history.lastTransition = { from: before.kind, to: after.kind, at, cause }; + const retryAt = after.kind === "failed" || after.kind === "reconnecting" ? after.retryAt : undefined; + const tunnelPort = options.tunnelPort?.(linkId); + try { + options.diagnostic?.({ + event: "tunnel_transition", + outcome: cause ?? (after.kind === "connected" ? "connected" : after.kind === "idle" ? "stopped" : "spawned"), + state: after.kind, + previousState: before.kind, + attempt: history.attempt, + ...(tunnelPort !== undefined ? { tunnelPort } : {}), + ...(retryAt !== undefined ? { retryAfterMs: Math.max(0, retryAt - at) } : {}), + }); + } catch { /* Diagnostics never change the tunnel lifecycle. */ } + }, + get(linkId) { + const history = histories.get(linkId); + if (!history) return undefined; + return { + attempt: history.attempt, + lastTransition: history.lastTransition && { ...history.lastTransition }, + lastFailure: history.lastFailure && { ...history.lastFailure }, + }; + }, + forget(linkId) { + histories.delete(linkId); + }, + }; +} diff --git a/src/link/tunnel-state.ts b/src/link/tunnel-state.ts index 804881307d7..683b9c1abed 100644 --- a/src/link/tunnel-state.ts +++ b/src/link/tunnel-state.ts @@ -9,18 +9,35 @@ * Without a retry policy it needs the user. With one (the Home's `-R` supervisor and the Child's * own `-L` tunnel both run with one) a reason that has a delay is tried again at `retryAt`, and * the retry attempt runs with `inFlight` while the state still reads failed; a reason without a - * delay stays terminal. + * delay stays terminal. Each retry that fails again doubles the next delay, up to the policy's + * `maxRetryDelayMs`; `retries` counts those failed retries (absent before the first one). A due + * retry whose ssh could not even be started counts too (`failedSpawnTunnel`). */ export type TunnelFailure = "auth" | "hostkey" | "forward" | "timeout"; -export type StderrClass = "auth" | "hostkey" | "forward" | "network" | "unknown"; +/** + * `remote_forward` is the Home's `-R` request refused by the peer's sshd: the port is still held, + * usually by an earlier session the peer has not dropped yet. It fails the link as `forward` and + * keeps the finer cause in `detail`. + */ +export type StderrClass = "auth" | "hostkey" | "forward" | "remote_forward" | "network" | "unknown"; +export type TunnelFailureDetail = "remote_forward_conflict"; export type TunnelState = | { kind: "idle" } | { kind: "connecting"; since: number } | { kind: "connected"; since: number } | { kind: "reconnecting"; since: number; attempt: number; retryAt: number; inFlight: boolean } - | { kind: "failed"; since: number; reason: TunnelFailure; retryAt?: number; inFlight?: boolean }; + | { + kind: "failed"; + since: number; + reason: TunnelFailure; + retryAt?: number; + inFlight?: boolean; + /** Failed retries since the link failed; absent until the first retry fails. */ + retries?: number; + detail?: TunnelFailureDetail; + }; export type TunnelEvent = | { type: "spawn"; now: number } @@ -29,9 +46,13 @@ export type TunnelEvent = | { type: "tick"; now: number } | { type: "stop" }; -/** Opt-in retry of a failed tunnel: the delay before each failure reason is tried again. */ +/** + * Opt-in retry of a failed tunnel: the first delay before each failure reason is tried again, and + * the cap the doubling delay of later retries stops at. + */ export interface TunnelRetryPolicy { retryFailedAfterMs: Partial>; + maxRetryDelayMs?: number; } export const FAILED_AFTER_MS = 5 * 60_000; @@ -39,19 +60,20 @@ export const BASE_DELAY_MS = 1_000; export const MAX_DELAY_MS = 30_000; /** - * The client-owned tunnel's policy. Timeout and forward failures retry about once a minute. Auth - * retries every five minutes, and because each auth failure schedules the next attempt five - * minutes out, no more than 12 attempts reach the Home's sshd in an hour. A changed host key is a - * security signal and is never retried. + * The client-owned tunnel's policy. Timeout and forward failures are first retried after a minute, + * and each retry that fails again doubles the delay up to five minutes (60, 120, 240, then every + * 300 seconds). Auth starts at the five-minute cap, so no more than 12 attempts reach the Home's + * sshd in an hour. A changed host key is a security signal and is never retried. */ export const CLIENT_TUNNEL_RETRY_POLICY: TunnelRetryPolicy = { retryFailedAfterMs: { timeout: 60_000, forward: 60_000, auth: 5 * 60_000 }, + maxRetryDelayMs: 5 * 60_000, }; /** * The Home's `-R` supervisor dials the Child's sshd on the same cadence and limits. A sleep, an - * outage longer than FAILED_AFTER_MS, or a forward the Child's sshd still holds for the session - * that just died heals about a minute after it clears, instead of waiting for a proxy restart. + * outage longer than FAILED_AFTER_MS, or a forward the Child's sshd still holds for a session that + * died heals at most five minutes after it clears, instead of waiting for a proxy restart. */ export const HOME_TUNNEL_RETRY_POLICY: TunnelRetryPolicy = CLIENT_TUNNEL_RETRY_POLICY; @@ -65,23 +87,59 @@ export function nextDelayMs(attempt: number, random: () => number = Math.random) return Math.round(Math.min(MAX_DELAY_MS, base * jitter)); } +/** The delay before the next retry of `reason` after `retries` failed retries, or undefined. */ +export function failedRetryDelayMs(reason: TunnelFailure, retries: number, policy?: TunnelRetryPolicy): number | undefined { + const base = policy?.retryFailedAfterMs[reason]; + if (base === undefined) return undefined; + const cap = Math.max(base, policy?.maxRetryDelayMs ?? base); + return Math.min(cap, base * 2 ** Math.max(0, Math.min(retries, 16))); +} + /** A failed state; under a policy that retries `reason` it carries the next attempt's time. */ export function failedTunnel( reason: TunnelFailure, now: number, policy?: TunnelRetryPolicy, since: number = now, + options: { retries?: number; detail?: TunnelFailureDetail } = {}, ): TunnelState { - const delay = policy?.retryFailedAfterMs[reason]; + const retries = options.retries ?? 0; + const delay = failedRetryDelayMs(reason, retries, policy); + const extra = { + ...(retries > 0 ? { retries } : {}), + ...(options.detail ? { detail: options.detail } : {}), + }; return delay === undefined - ? { kind: "failed", since, reason } - : { kind: "failed", since, reason, retryAt: now + delay, inFlight: false }; + ? { kind: "failed", since, reason, ...extra } + : { kind: "failed", since, reason, retryAt: now + delay, inFlight: false, ...extra }; +} + +/** + * The state after an attempt whose ssh could not be started (ssh missing, a pidfile that cannot be + * written, no listener port). On a due retry of a failed link it counts as one more failed retry + * from the original moment and keeps the reason it was retrying: an attempt that reached no peer + * says nothing new about the link, so the backoff keeps its step and an auth failure keeps its + * five-minute cadence. Any other state fails fresh with `reason`. + */ +export function failedSpawnTunnel( + prior: TunnelState | undefined, + reason: TunnelFailure, + now: number, + policy?: TunnelRetryPolicy, +): TunnelState { + if (prior?.kind !== "failed") return failedTunnel(reason, now, policy); + return failedTunnel(prior.reason, now, policy, prior.since, { retries: (prior.retries ?? 0) + 1, detail: prior.detail }); } function failureOf(cls: StderrClass): TunnelFailure | null { + if (cls === "remote_forward") return "forward"; return cls === "auth" || cls === "hostkey" || cls === "forward" ? cls : null; } +function detailOf(cls: StderrClass): TunnelFailureDetail | undefined { + return cls === "remote_forward" ? "remote_forward_conflict" : undefined; +} + export function reduceTunnel( state: TunnelState, event: TunnelEvent, @@ -104,11 +162,15 @@ export function reduceTunnel( if (state.kind === "failed") { if (!state.inFlight) return state; // A retry of a failed link failed again. It stays failed from the original moment; a - // transient exit keeps the slow cadence (timeout) instead of restarting fast backoff. - return failedTunnel(failureOf(event.stderrClass) ?? "timeout", event.now, policy, state.since); + // transient exit keeps the slow cadence (timeout) instead of restarting fast backoff, and + // the next delay doubles up to the policy's cap. + return failedTunnel(failureOf(event.stderrClass) ?? "timeout", event.now, policy, state.since, { + retries: (state.retries ?? 0) + 1, + detail: detailOf(event.stderrClass), + }); } const failure = failureOf(event.stderrClass); - if (failure) return failedTunnel(failure, event.now, policy); + if (failure) return failedTunnel(failure, event.now, policy, event.now, { detail: detailOf(event.stderrClass) }); const since = state.kind === "reconnecting" || state.kind === "connecting" ? state.since : event.now; if (event.now - since >= FAILED_AFTER_MS) return failedTunnel("timeout", event.now, policy); const attempt = state.kind === "reconnecting" ? state.attempt + 1 : 1; @@ -137,7 +199,8 @@ export function classifySshStderr(stderr: string): StderrClass { || text.includes("no ecdsa host key is known") || text.includes("no ed25519 host key is known") || text.includes("no rsa host key is known") || text.includes("host key is known for")) return "hostkey"; if (text.includes("permission denied") || text.includes("too many authentication failures")) return "auth"; - if (text.includes("remote port forwarding failed") || text.includes("port forwarding failed") + if (text.includes("remote port forwarding failed")) return "remote_forward"; + if (text.includes("port forwarding failed") || text.includes("address already in use") || text.includes("cannot listen to port") || text.includes("could not request local forwarding")) return "forward"; if (text.includes("connection refused") || text.includes("timed out") || text.includes("could not resolve") diff --git a/src/server/index/link-relay-sessions.ts b/src/server/index/link-relay-sessions.ts index 166558384c0..72daff71431 100644 --- a/src/server/index/link-relay-sessions.ts +++ b/src/server/index/link-relay-sessions.ts @@ -1,7 +1,7 @@ import type { Server } from "bun"; import { LINK_RELAY_AUTH_PATH, LINK_RELAY_AUTH_TIMEOUT_MS, LINK_RELAY_AUTH_VERSION, - LINK_RELAY_SESSION_HEADER, linkRelayProof, linkRelayProofMatches, + LINK_RELAY_PROBE_HEADER, LINK_RELAY_SESSION_HEADER, linkRelayProof, linkRelayProofMatches, } from "../../link/relay-auth"; import { parseLinkPeerReport, type LinkPeerObservations } from "../../link/peer-report"; @@ -81,13 +81,17 @@ export function createLinkRelaySessions(options: LinkRelaySessionOptions) { linkRelayProofMatches(q.get("proof"), linkRelayProof(candidate, "caller", keyId, linkId, nonce))); if (!fingerprint) return refuse(); options.peers?.observe(linkId, LINK_RELAY_AUTH_VERSION, parseLinkPeerReport(req.headers)); - const release = acquire(); - const timer = setTimeout(() => { consume(nonce)?.release(); }, reservationMs); - sessions.set(nonce, { peer, keyId, linkId, fingerprint, release, timer, expiresAt: Date.now() + reservationMs }); - return new Response(null, { status: 204, headers: { + const answer = () => new Response(null, { status: 204, headers: { "x-opencodex-link-proof": linkRelayProof(fingerprint, "listener", keyId, linkId, nonce)!, "cache-control": "no-store", } }); + // A readiness probe closes after this answer and sends no data request, so it reserves + // nothing: no session to consume, no capacity and no drain lease held until expiry. + if (req.headers.get(LINK_RELAY_PROBE_HEADER) === "1") return answer(); + const release = acquire(); + const timer = setTimeout(() => { consume(nonce)?.release(); }, reservationMs); + sessions.set(nonce, { peer, keyId, linkId, fingerprint, release, timer, expiresAt: Date.now() + reservationMs }); + return answer(); } const nonce = req.headers.get(LINK_RELAY_SESSION_HEADER); let release: () => void; diff --git a/src/server/index/optional-listeners.ts b/src/server/index/optional-listeners.ts index 22685617b1b..f055c27f168 100644 --- a/src/server/index/optional-listeners.ts +++ b/src/server/index/optional-listeners.ts @@ -14,6 +14,7 @@ import { type LinkListenerStatus, } from "./link-listener"; import { createLinkSupervisor, type LinkSupervisor } from "../../link/supervisor"; +import { flushLinkDiagnostics, recordLinkDiagnostic } from "../../link/diagnostics"; import { linkManagementRouteAllowed } from "../../link/management-routes"; import { linkChildParityRouteAllowed } from "../../link/child-parity"; import { saveConfigPreservingClaudeCode } from "../../config/live-reconcile"; @@ -56,6 +57,9 @@ export function createOptionalListenerSet(linkDeps: LinkListenerDeps = {}): O let activeConfig: OcxConfig | undefined; let stopJevQuotaActivation: (() => void) | undefined; const supervisor = createLinkSupervisor({ + // Tunnel state changes and their causes reach /link/diagnostics.jsonl (bounded). + diagnostic: recordLinkDiagnostic, + flushDiagnostics: flushLinkDiagnostics, apiKeys: () => activeConfig?.apiKeys ?? [], revokeApiKey: id => { if (!activeConfig) return false; diff --git a/src/server/management/link-routes.ts b/src/server/management/link-routes.ts index 7baa25d2dbe..640e25f294c 100644 --- a/src/server/management/link-routes.ts +++ b/src/server/management/link-routes.ts @@ -6,7 +6,7 @@ import { clearCompensationFailed, compensationPath, readCompensation } from "../ import { loadHostCandidates } from "../../link/ssh-config"; import { newLinkId, type LinkStore } from "../../link/store"; import { sshFailureHint, sshRunnerErrorHint } from "../../link/ssh-runner"; -import { attachLinkPeers, projectLinkStatus, type LinkStatusDto } from "../../link/status-projection"; +import { attachLinkPeers, attachLinkTunnelDiagnostics, projectLinkStatus, type LinkStatusDto } from "../../link/status-projection"; import { homeLinkChildParity } from "../../link/child-parity"; import { localLinkPeer } from "../../link/compatibility"; import { packageVersion } from "../../lib/package-version"; @@ -268,8 +268,9 @@ export async function handleLinkRoutes(ctx: ManagementContext, suppliedState?: L if (denied) return denied; const store = readStoreFor(ctx); const listenerStatus = state.listener.status(); + const supervisorStates = state.supervisor.status(); const dto: LinkStatusDto = projectLinkStatus( - store, state.supervisor.status(), listenerStatus, ctx.config, readCompensation(), + store, supervisorStates, listenerStatus, ctx.config, readCompensation(), ctx.config.runtimeRole === "client" ? clientLinkTunnelStatus() : null, ctx.config.runtimeRole === "client" && ctx.config.client?.link ? clientLinkStatusIdentity(ctx.config.client.link.linkId) : null, linkId => homeLinkChildParity.get(linkId), @@ -288,6 +289,8 @@ export async function handleLinkRoutes(ctx: ManagementContext, suppliedState?: L attachLinkPeers(dto, localLinkPeer(packageVersion("0.0.0"), buildPeerFeatures()), linkId => state.listener.peerObservation?.(linkId) ?? { observed: null, refused: null }); } + // The same opt-in for each Home-initiated tunnel's retry progress and last failure cause. + if (url.searchParams.get("diagnostics") === "1") attachLinkTunnelDiagnostics(dto, supervisorStates); // A dashboard session also learns why it cannot join as a Child. The admin-token answer stays // the exact K16 document that `ocx link status` validates key by key. const denial = ctx.principal === "gui-session" ? joinDenied(ctx) : null; diff --git a/src/server/port-reclaim.ts b/src/server/port-reclaim.ts index 73ddc7caefa..c3dcc0231bf 100644 --- a/src/server/port-reclaim.ts +++ b/src/server/port-reclaim.ts @@ -405,22 +405,37 @@ async function runOwnerLookup(file: string, args: string[], timeoutMs: number): return stdout; } +export type LoopbackOwnerLookupResult = { + kind: "match" | "mismatch" | "lookup_error" | "deadline"; + elapsedMs: number; +}; + export interface LoopbackOwnerLookupIo { platform?: NodeJS.Platform; + nowMs?: () => number; + scheduleDeadline?: (callback: () => void, timeoutMs: number) => () => void; readProc?: (path: string) => Promise; listFds?: (path: string) => Promise; readFdLink?: (path: string) => Promise; run?: (file: string, args: string[], timeoutMs: number) => Promise; } -/** Async, bounded proof that this PID owns the IPv4 loopback LISTEN socket used by the relay. */ -export async function ownsIpv4LoopbackListener( +/** + * Classify fresh IPv4 loopback LISTEN ownership within the bounded lookup deadline. + * Only match proves the expected PID; mismatch, inspection failure and timeout remain distinct. + * elapsedMs uses a monotonic clock by default, and no raw OS output is returned. + */ +export async function inspectIpv4LoopbackListener( port: number, expectedPid: number, io: LoopbackOwnerLookupIo = {}, -): Promise { +): Promise { + const now = io.nowMs ?? (() => performance.now()); + const startedAt = now(); + const result = (kind: LoopbackOwnerLookupResult["kind"]): LoopbackOwnerLookupResult => + ({ kind, elapsedMs: Math.max(0, now() - startedAt) }); if (!Number.isInteger(port) || port < 1 || port > 65535 - || !Number.isSafeInteger(expectedPid) || expectedPid < 1) return false; + || !Number.isSafeInteger(expectedPid) || expectedPid < 1) return result("mismatch"); const platform = io.platform ?? process.platform; const lookup = async (): Promise => { if (platform === "linux") { @@ -458,13 +473,40 @@ export async function ownsIpv4LoopbackListener( } return false; }; - let timer: ReturnType | undefined; - const deadline = new Promise(resolve => { - timer = setTimeout(() => resolve(false), OWNER_LOOKUP_TIMEOUT_MS); + const scheduleDeadline = io.scheduleDeadline ?? ((callback, timeoutMs) => { + const timer = setTimeout(callback, timeoutMs); timer.unref?.(); + return () => clearTimeout(timer); }); - try { return await Promise.race([lookup().catch(() => false), deadline]); } - finally { if (timer) clearTimeout(timer); } + let cancelDeadline: (() => void) | undefined; + const deadline = new Promise(resolve => { + cancelDeadline = scheduleDeadline(() => resolve("deadline"), OWNER_LOOKUP_TIMEOUT_MS); + }); + const inspected = lookup().then( + matches => matches ? "match" as const : "mismatch" as const, + error => { + const failure = error as { + killed?: boolean; signal?: string | null; code?: unknown; stdout?: unknown; stderr?: unknown; + } | null; + // lsof exits 1 with no output when no socket matches its selectors. + if (platform === "darwin" && failure?.code === 1 && !failure.killed + && failure.signal == null && failure.stdout === "" && failure.stderr === "") return "mismatch" as const; + // execFile's timeout kills with SIGTERM; other spawn/exit/buffer errors are lookup failures. + return failure?.killed === true && failure.signal === "SIGTERM" && failure.code === null + ? "deadline" as const : "lookup_error" as const; + }, + ); + try { return result(await Promise.race([inspected, deadline])); } + finally { cancelDeadline?.(); } +} + +/** Boolean compatibility API: only a proven match admits the listener. */ +export async function ownsIpv4LoopbackListener( + port: number, + expectedPid: number, + io: LoopbackOwnerLookupIo = {}, +): Promise { + return (await inspectIpv4LoopbackListener(port, expectedPid, io)).kind === "match"; } /** diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index e2daf09969d..6027d2778ad 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -431,7 +431,7 @@ routes to its Home, serves its own read-only `GET /api/link/status` ([Remote Lin and relays allowlisted management reads to the Home under `/api/machine/hub-relay/api/*` for a Child dashboard session ([Remote Link](remote-link.md#child-management-relay-and-home-codex-settings)). Every other `/api/*` and `/v1/*` path is refused before dispatch with a JSON 404 naming the method and path. There is no second management port on such a machine: -management rides the same listener a standalone or hub install runs, so a connected client has no mutating `/api/*` surface. +management rides the same listener a standalone or hub install runs, so a connected client has no mutating `/api/*` surface. In link mode its `/readyz` also says whether the link can carry a request right now ([Remote Link](remote-link.md)); its `/healthz` does not. The discriminator is `role` on `/healthz` and `/readyz`. The machine listener reports `role: "client"`; the standalone and hub server omit the field. `src/server/proxy-liveness.ts` parses diff --git a/structure/remote-link.md b/structure/remote-link.md index 69daf34d44c..ca7404d3b16 100644 --- a/structure/remote-link.md +++ b/structure/remote-link.md @@ -12,7 +12,9 @@ Every remote `ocx` call goes through `remoteOcxArgv`, a quoted POSIX `sh -c` scr `src/link/ssh-runner.ts` drains tunnel stdout and stderr with a 64 KiB cap per stream. After the SSH process exits, either reader that has not reached EOF gets 250 ms to consume remaining bytes, then is cancelled and its lock released without waiting for the underlying cancellation promise. Already consumed diagnostics are retained for host-key, auth and network classification, so an inherited pipe cannot indefinitely hold the supervisor's exit classification or pause retries and the outage clock. This grace applies to spawned tunnels; bounded one-shot commands retain their command timeout. -`src/link/tunnel-state.ts` is the tunnel lifecycle reducer: connecting, connected, reconnecting with capped jittered backoff, and failed for auth, host-key and forward errors or after five minutes without a connection, whether or not an attempt is in flight. Without a retry policy, failed is terminal. `CLIENT_TUNNEL_RETRY_POLICY` gives failed a `retryAt`: timeout and forward failures are retried after 60 seconds and auth failures after five minutes, so no more than 12 auth attempts reach the peer's sshd in an hour; a host-key failure is never retried. The Child's own tunnel runs with it, and `src/link/supervisor.ts` runs the Home's `-R` tunnels with `HOME_TUNNEL_RETRY_POLICY`, the same policy. A retry runs while the state still reads failed (`inFlight`), a ready event promotes it to connected, and an exit returns it to failed with the original `since` and the next `retryAt`, so a long outage keeps the one-a-minute cadence instead of restarting fast backoff. The Home's supervisor promotes a first attempt to connected when its link key authenticates a catalog request or the child stays alive for five seconds. It promotes a retry (reconnecting, or failed with `inFlight`) on the same catalog request or once the child outlives a 15-second retry grace, longer than the ten seconds ssh's `ConnectTimeout` allows for the TCP connect, handshake and key exchange, because an idle Child may send no catalog request at all. Liveness is a heuristic here, as it is for a first attempt: authentication and the remote forward request come after `ConnectTimeout`, so one that stalls past the grace reads connected until ssh exits, and the exit returns the link to the retry path. Promotion is checked before the outage limit, so a child that has outlived its grace by the time a tick runs, at the limit or late after a sleep, is promoted. At the outage limit it kills the child, except a retry still inside its grace, which carries on as the failed state's retry. A reload spawns a tunnel only for a fresh link or a due attempt, so it neither skips the backoff nor starts a second child while an exited one is still reporting. `reconnect(linkId)` is the deliberate exception: it queues with reload, link stop and shutdown, uses the updated record when a preceding reload finishes, stops that link's child, forgets its backoff or failed state and spawns a fresh attempt. After the child exits it rechecks the persisted record, listener port, generation and reload revision; removal or a changed tunnel cancels that restart. Automatic ticks wait until lifecycle work drains. Overlapping reconnects for one link share one restart, and every spawn bumps a per-link generation, so an exit still being classified or a stop still waiting for its child never overwrites the state of a newer child. `POST /api/link/{id}/reconnect` (the dashboard's **Retry** on a failed or reconnecting Home link, and `ocx link reconnect --link-id `) requires the link listener to be listening (`503 listener_unavailable` otherwise, as for apply), then calls it after `ensureStarted`, answers `{ linkId }`, and refuses a Child-initiated link with `409 link_not_home_initiated` because the Child owns that tunnel. DELETE holds the link against reconnects and automatic spawns throughout remote disconnect, key revocation and store removal. Successful removal reloads the supervisor before releasing the hold; a remote disconnect failure releases it before reloading to restore the retained tunnel. +`src/link/tunnel-state.ts` is the tunnel lifecycle reducer: connecting, connected, reconnecting with capped jittered backoff, and failed for auth, host-key and forward errors or after five minutes without a connection, whether or not an attempt is in flight. Without a retry policy, failed is terminal. `CLIENT_TUNNEL_RETRY_POLICY` gives failed a `retryAt`: timeout and forward failures are first retried after 60 seconds, and each retry that fails again doubles the delay up to `maxRetryDelayMs` (five minutes; the failed state counts those in `retries`). Auth failures start at that five-minute cap, so no more than 12 auth attempts reach the peer's sshd in an hour; a host-key failure is never retried. ssh's `remote port forwarding failed` is its own stderr class, `remote_forward`: it fails the link with reason `forward`, which the dashboard and the K16 `reason` already know, and keeps `detail: "remote_forward_conflict"` on the state. The Child's own tunnel runs with it, and `src/link/supervisor.ts` runs the Home's `-R` tunnels with `HOME_TUNNEL_RETRY_POLICY`, the same policy. A retry runs while the state still reads failed (`inFlight`), a ready event promotes it to connected, and an exit returns it to failed with the original `since` and the next, doubled `retryAt`, so a long outage settles at one attempt every five minutes instead of restarting fast backoff. A due retry whose ssh cannot be started at all (ssh missing, an unwritable pidfile, or on the Home no listener port) goes through `failedSpawnTunnel` on both sides: it counts as a failed retry and keeps the reason and `detail` it was retrying, because it reached no peer, so the delay keeps doubling and an auth failure keeps its five-minute cadence. The Home's supervisor promotes a first attempt to connected when its link key authenticates a catalog request or the child stays alive for five seconds. It promotes a retry (reconnecting, or failed with `inFlight`) on the same catalog request or once the child outlives a 15-second retry grace, longer than the ten seconds ssh's `ConnectTimeout` allows for the TCP connect, handshake and key exchange, because an idle Child may send no catalog request at all. Liveness is a heuristic here, as it is for a first attempt: authentication and the remote forward request come after `ConnectTimeout`, so one that stalls past the grace reads connected until ssh exits, and the exit returns the link to the retry path. Promotion is checked before the outage limit, so a child that has outlived its grace by the time a tick runs, at the limit or late after a sleep, is promoted. At the outage limit it kills the child, except a retry still inside its grace, which carries on as the failed state's retry. A reload spawns a tunnel only for a fresh link or a due attempt, so it neither skips the backoff nor starts a second child while an exited one is still reporting. `reconnect(linkId)` is the deliberate exception: it queues with reload, link stop and shutdown, uses the updated record when a preceding reload finishes, stops that link's child, forgets its backoff or failed state and spawns a fresh attempt. After the child exits it rechecks the persisted record, listener port, generation and reload revision; removal or a changed tunnel cancels that restart. Automatic ticks wait until lifecycle work drains. Overlapping reconnects for one link share one restart, and every spawn bumps a per-link generation, so an exit still being classified or a stop still waiting for its child never overwrites the state of a newer child. `POST /api/link/{id}/reconnect` (the dashboard's **Retry** on a failed or reconnecting Home link, and `ocx link reconnect --link-id `) requires the link listener to be listening (`503 listener_unavailable` otherwise, as for apply), then calls it after `ensureStarted`, answers `{ linkId }`, and refuses a Child-initiated link with `409 link_not_home_initiated` because the Child owns that tunnel. DELETE holds the link against reconnects and automatic spawns throughout remote disconnect, key revocation and store removal. Successful removal reloads the supervisor before releasing the hold; a remote disconnect failure releases it before reloading to restore the retained tunnel. + +`src/link/tunnel-history.ts` is the Home supervisor's memory of each `-R` tunnel beside its state: `attempt` counts ssh attempts since the link was last connected, `lastTransition` is the last change of state kind with its time and cause, and `lastFailure` the last cause that took the link out of a working state. A retry of a failed link that fails again, whether its ssh ran or could not start, counts as a transition. A cause is one fixed word: the stderr class (`auth`, `hostkey`, `forward`, `remote_forward_conflict`, `network`, `unknown`), `timeout` for the outage limit, `spawn_failed` when ssh could not be started and `listener_unavailable` when the store has no listener port. That last case is a forward failure under the retry policy, not a terminal state; a reload that brings the port back changes the record instance and starts a fresh attempt. The supervisor routes every state change through one function so the history sees each one, forgets a link's history when its record changes or disappears, and reports it as `history` in `status()`. `attachLinkTunnelDiagnostics` in `src/link/status-projection.ts` turns it into the link row's `tunnel` field (`attempt`, `nextRetryAt`, `lastTransition`, `lastFailure`, ISO times), which `GET /api/link/status` adds only for `?diagnostics=1` because an older `ocx link status` validates rows key by key; the current CLI asks for it and validates it. Each recorded transition is also a `tunnel_transition` diagnostic (below). Nothing in the history comes from ssh output or argv. `src/link/routes.ts` holds the one route and method table for linked data-plane traffic; the hub-link listener and the client relay both decide admission from it. The read-only management table beside it is described in [Child management relay and Home Codex settings](#child-management-relay-and-home-codex-settings). @@ -60,7 +62,7 @@ Applying a link probes the host key into a temporary file, waits for the operato ## Client link transport -The Child relay requires a positive `connected()` verdict from `src/client/link-tunnel.ts` before every fetch. Before the first keyed readiness probe and after a tunnel restart, the supervisor asynchronously proves the exact `127.0.0.1:` LISTEN socket belongs to its SSH child; an adopted process also needs matching pidfile argv and start time. `src/server/port-reclaim.ts` uses Linux `/proc/net/tcp{,6}` plus the expected PID's `/proc//fd` socket symlinks without external tools, macOS `lsof` and Windows `netstat` with bounded asynchronous execution. Every key-bearing probe and relayed request makes a fresh bounded asynchronous owner lookup; an adopted process also has its current argv and start time rechecked on every admission. An unreadable or timed-out identity denies only that admission and is retried; a confirmed mismatch releases the adopted PID without signalling it and lets the next tick start a fresh tunnel. An unknown socket-owner lookup likewise denies the current admission. A missing supervisor, or a failed or stopped tunnel, returns a retryable 503 without sending the link key or body to the persisted loopback port; its body carries `code: "home_unreachable"` under the [fallback policy](fallback-policy.md#hub-down). A Home-initiated Child is the exception: the Home owns that link's `ssh -R` forward, so the Child has no sidecar, no supervisor and no SSH process whose socket it could prove. A join writes `/link/child-initiated.json` (the link id, mode 0600) beside the sidecar and removes it with the sidecar, so a Child-initiated link that lost its sidecar is recognised (`isChildInitiatedLink` in `src/client/link-state.ts`; an unreadable marker counts as present) and gets no gate. Only a link-mode runtime with neither the sidecar nor a marker for its link id is treated as Home-initiated: `src/client/runtime.ts` passes it `HOME_INITIATED_LINK_TUNNEL` (`src/client/link-relay.ts`), which omits local process ownership proof and reconnect holding but still uses the connection-bound relay authentication below. A refused connection answers 503. A held request rechecks the verdict after its wait and before each retry. +The Child relay requires a `connected()` result with `kind: "match"` from `src/client/link-tunnel.ts` before every fetch. The gate returns one typed `LinkAdmissionResult` containing the verdict and bounded timing/generation/check metadata; diagnostic callbacks do not supply admission or recovery decisions. The supervisor has one typed ownership dependency, `inspectLoopbackListener`. Before the first keyed readiness probe and after a tunnel restart, the supervisor asynchronously proves the exact `127.0.0.1:` LISTEN socket belongs to its SSH child; an adopted process also needs matching pidfile argv and start time. `src/server/port-reclaim.ts` uses Linux `/proc/net/tcp{,6}` plus the expected PID's `/proc//fd` socket symlinks without external tools, macOS `lsof` and Windows `netstat` with bounded asynchronous execution. Every key-bearing probe and relayed request makes a fresh bounded asynchronous owner lookup; an adopted process also has its current argv and start time rechecked on every admission. An unreadable or timed-out identity denies only that admission and is retried; a confirmed mismatch releases the adopted PID without signalling it and lets the next tick start a fresh tunnel. An unknown socket-owner lookup likewise denies the current admission. A missing supervisor, or a failed or stopped tunnel, returns a retryable 503 without sending the link key or body to the persisted loopback port; its body carries `code: "home_unreachable"` under the [fallback policy](fallback-policy.md#hub-down). A Home-initiated Child is the exception: the Home owns that link's `ssh -R` forward, so the Child has no sidecar, no supervisor and no SSH process whose socket it could prove. A join writes `/link/child-initiated.json` (the link id, mode 0600) beside the sidecar and removes it with the sidecar, so a Child-initiated link that lost its sidecar is recognised (`isChildInitiatedLink` in `src/client/link-state.ts`; an unreadable marker counts as present) and gets no gate. Only a link-mode runtime with neither the sidecar nor a marker for its link id is treated as Home-initiated: `src/client/runtime.ts` passes it `HOME_INITIATED_LINK_TUNNEL` (`src/client/link-relay.ts`), which omits local process ownership proof and reconnect holding but still uses the connection-bound relay authentication below. A refused connection answers 503. A held request rechecks the verdict after its wait and before each retry. The listener can change between an ownership check and the TCP connect. The data relay now authenticates that physical connection before sending the key or body (see [Connection-bound relay authentication](#connection-bound-relay-authentication)); this does not change the separate readiness/enrollment probes. A private Unix-socket forward could additionally remove competing TCP listeners from those probe paths. @@ -68,16 +70,32 @@ A client connected with `transport: "link"` reaches its hub through an SSH tunne Codex keeps the standalone loopback routing: `routingTarget` in `src/client/connect.ts` returns the `standaloneCodexRoutingTarget` form for the configured port, root `openai_base_url = "http://127.0.0.1:/v1"` plus the realtime override, with no provider table and no `env_key`. For the same port the join therefore writes the bytes the standalone injection wrote, and a Codex app launched without the shim's environment keeps working; an earlier `env_key` table is rebuilt into this form on the next sync. -`src/client/link-ingress.ts` is the link-mode data plane of the machine listener. A `/v1/responses` WebSocket upgrade answers `426 upgrade_required`, which codex-rs maps to its HTTP fallback, and no upgrade is ever relayed. Every relayed route first passes the standalone loopback Host and Origin gate (`isAllowedRequestOrigin` in `src/server/auth-cors.ts`), so a rebinding or cross-site page gets `403 origin_rejected` and nothing is fetched upstream. `/readyz` is answered locally. The link key is read from the service token file once, when the listener starts, and held in memory; the file must hold the key whose fingerprint the connection committed. While it does not, relayed routes answer `503 link_credential_unavailable` without an upstream fetch, and the file is read again at most once a second; a valid key is never re-read. The key is never logged or returned. +`src/client/link-ingress.ts` is the link-mode data plane of the machine listener. A `/v1/responses` WebSocket upgrade answers `426 upgrade_required`, which codex-rs maps to its HTTP fallback, and no upgrade is ever relayed. Every relayed route first passes the standalone loopback Host and Origin gate (`isAllowedRequestOrigin` in `src/server/auth-cors.ts`), so a rebinding or cross-site page gets `403 origin_rejected` and nothing is fetched upstream. `/readyz` is answered locally and never relayed; in link mode it reports the link through `src/client/link-readiness.ts` (below). The link key is read from the service token file once, when the listener starts, and held in memory; the file must hold the key whose fingerprint the connection committed. While it does not, relayed routes answer `503 link_credential_unavailable` without an upstream fetch, and the file is read again at most once a second; a valid key is never re-read. The key is never logged or returned. `src/client/link-relay.ts` forwards exactly the `linkRouteAllowed` routes from `src/link/routes.ts` through the tunnel. It drops the caller's `Authorization`, Azure `api-key`, Anthropic-compatible `x-api-key`, Google `x-goog-api-key`, `x-opencodex-api-key`, `chatgpt-account-id` and `cookie` and sends the link key as `Authorization: Bearer`, the wire an `env_key` config sent; `GET /v1/usage` takes it as `x-opencodex-api-key`, the only header that route admits. The explicit provider forms matter because the Child and Home are separate credential owners: caller provider credentials never cross the tunnel merely because they are not Bearer tokens. The Home admits the link key and serves the Child with its own accounts. The request body is streamed chunk by chunk with the caller's `Content-Length` and a byte-counting cap at the inbound limit (`resolveInboundBodyLimitBytes`, 256 MiB by default); a larger declared or streamed body answers 413. A lone `Transfer-Encoding: chunked` without `Content-Length` is admitted as a standalone admits it, because the listener has already de-chunked the body; any other Transfer-Encoding, or one next to a `Content-Length`, answers 400. The Home's response headers may take up to 300 seconds, and a caller abort ends the wait sooner. SSE passes through chunk by chunk with caller-abort propagation and a 300-second idle limit, other response bodies stream under the same byte cap, and the relay answers 503 with Retry-After while the tunnel is down. The client supervisor is the relay's tunnel gate (`LinkTunnelGate`): only while the tunnel is connecting or reconnecting (including the start of the client runtime) does a relayed request wait, for at most 15 seconds (`LINK_RELAY_HOLD_MS`) from its first wait and with at most 64 requests waiting, before it is forwarded once; a connected tunnel costs one `pending()` call per request, and a failed one answers 503 at once. A forward whose connection was refused sent nothing, so while the tunnel reconnects it may wait again and be sent again inside the same 15 seconds, provided the streamed body was never read or cancelled; any other failure (a reset, a timeout, a failure after the body started) is never replayed. Both the Child's machine listener and the Home's hub-link listener bind with `idleTimeout: 255`, the public listener's limit, so a held or slow turn is not cut by Bun's 10-second default. Like a standalone data route, a relayed request then lifts its own idle timer (`server.timeout(req, 0)` in `src/client/link-ingress.ts`), so a quiet stretch longer than 255 seconds inside a long generation is not cut either; the relay's header deadline, SSE idle limit and caller abort bound the wait instead. Hub transport keeps the 4 MiB management-relay listener bound and its default idle limit. Link mode waits for the configured port without signalling its holder, then binds there or fails; `src/client/runtime.ts` passes the cached link key, tunnel status and tunnel gate through `bindClientListener` to every bind attempt. Link mode replaces the hub management relay with the read-only link management relay (see [Child management relay and Home Codex settings](#child-management-relay-and-home-codex-settings)) and refuses key rotation and revocation, which belong to the Home. +`src/client/link-readiness.ts` gives a link-mode Child's `GET /readyz` its verdict; `/healthz` keeps meaning only that the process answers. While relayed requests cannot be served, `/readyz` answers `503` with `Retry-After: 1`, `status: "pending"` and `link: { state: "unavailable", reason }`, which the strict readiness contract in `src/server/proxy-liveness.ts` reads as ours and not ready; otherwise `200`, `status: "ready"` and `link: { state: "available" }`. A reason is one fixed word. A Child-initiated link reads its supervisor: connected is ready unless the keyed probe last saw `home_unreachable` or `unauthorized` (`home_not_ready` stays ready, because relayed requests do not depend on the Home's own startup readiness); connecting, reconnecting and a failed state with a retry in flight read `tunnel_connecting`, failed `tunnel_failed`, idle or stopped `tunnel_idle`, an unreadable sidecar `sidecar_invalid`. A Home-initiated link has no local tunnel to inspect, so it runs `probeLinkRelayPreflight` from `src/client/link-relay-transport.ts`: the credential-free challenge every relayed request starts with, on its own connection, without a link key, data request or body, and without reading the response body. It sends `x-opencodex-link-probe: 1` (`LINK_RELAY_PROBE_HEADER`), so the Home verifies and answers without reserving a session (see the reservation book below). A verified proof is ready, no answer within the five-second authentication deadline is `home_unreachable`, any other answer `link_auth_failed`, and a missing link key `link_credential_unavailable` with nothing sent. One preflight runs at a time, its verdict is reused for five seconds, and `/readyz` waits at most 500 ms for a running one before it answers `link_unverified`, so an unauthenticated local caller causes at most one preflight every five seconds. A link with neither a supervisor nor a Home-initiated gate reads `tunnel_unavailable`. Restart handoff and the update job judge a replacement by `/healthz`, so a Child whose Home is down still restarts. + > Decision record: [ADR-6032](decisions/ADR-6032-link-relay-credential-boundary.md) -Regression coverage lives in `tests/clients/link-ssh-argv.test.ts`, `tests/clients/link-ssh-config.test.ts`, `tests/clients/link-tunnel-state.test.ts`, `tests/clients/link-store.test.ts`, `tests/clients/link-boundary.test.ts`, `tests/clients/link-routes.test.ts`, `tests/clients/client-link-connect.test.ts`, `tests/clients/client-link-relay.test.ts`, `tests/clients/client-machine-listener.test.ts`, `tests/clients/client-link-status.test.ts`, `tests/clients/client-link-runtime.test.ts`, `tests/codex-integration/injection-link-websocket.test.ts`, `tests/clients/link-supervisor.test.ts`, `tests/clients/link-status-projection.test.ts`, `tests/clients/link-admission-wait.test.ts`, `tests/clients/link-fingerprint.test.ts`, `tests/cli/cli-link.test.ts`, `tests/server/link-management-routes.test.ts`, `tests/server/link-join-route.test.ts`, `tests/server/port-reclaim.test.ts`, `tests/server/link-listener-lifecycle.test.ts`, `tests/clients/client-link-teardown.test.ts` and `gui/tests/remote-link.test.tsx`. +Regression coverage lives in `tests/clients/link-ssh-argv.test.ts`, `tests/clients/link-ssh-config.test.ts`, `tests/clients/link-tunnel-state.test.ts`, `tests/clients/link-tunnel-history.test.ts`, `tests/clients/link-readiness.test.ts`, `tests/clients/link-store.test.ts`, `tests/clients/link-boundary.test.ts`, `tests/clients/link-routes.test.ts`, `tests/clients/client-link-connect.test.ts`, `tests/clients/client-link-relay.test.ts`, `tests/clients/client-machine-listener.test.ts`, `tests/clients/client-link-status.test.ts`, `tests/clients/client-link-runtime.test.ts`, `tests/codex-integration/injection-link-websocket.test.ts`, `tests/clients/link-supervisor.test.ts`, `tests/clients/link-status-projection.test.ts`, `tests/clients/link-admission-wait.test.ts`, `tests/clients/link-fingerprint.test.ts`, `tests/cli/cli-link.test.ts`, `tests/server/link-management-routes.test.ts`, `tests/server/link-join-route.test.ts`, `tests/server/port-reclaim.test.ts`, `tests/server/link-listener-lifecycle.test.ts`, `tests/clients/client-link-teardown.test.ts` and `gui/tests/remote-link.test.tsx`. Enrollment cancellation in `src/client/connect.ts` reaches actual network requests and every subsequent write boundary. `src/client/link-join.ts` awaits that transaction's commit or completed rollback instead of racing a separate failure against its terminal result. An exit observed before commit aborts enrollment, drains local rollback, and only then revokes the issued key; an exit queued after the synchronous commit retains the committed link for restart. Readiness polling also observes tunnel exit while sleeping, rather than waiting out its deadline. `tests/server/link-join-route.test.ts` and `tests/clients/client-link-connect.test.ts` pin those boundaries. +## Bounded diagnostics and preflight recovery + +On Windows, a successful append refreshes the existing ACL memo only after matching the path to the open file's identity and closing the handle. Repeated writes avoid redundant ACL subprocesses; replacing the file or changing its permissions still requires fresh hardening. Exits from a bound client restart drain the same bounded diagnostic queue as signal shutdown and standalone recycle. + +The `diagnostics.ts` module in `src/link/` owns best-effort local evidence in `/link/diagnostics.jsonl` and its single rotated backup, `diagnostics.jsonl.1`, each capped at 256 KiB. The link directory is private (0700 on POSIX), files are private (0600), and Windows uses the secret-path ACL helpers. Diagnostics accept only fixed event/outcome/state values, generated UUID request/check IDs, numeric generation, elapsed time and attempt metadata, and a timestamp; they exclude secrets, request bodies, headers, URLs, account identifiers and raw errors. Each event/outcome pair emits at most once per second; the next emitted record carries the intervening `suppressed` count. A backward wall-clock adjustment starts a new rate-limit window, including when the writer queue is full, and preserves the suppression count. An asynchronous writer keeps at most 56 pending records, counts overflow as suppression, and yields while Windows ACL work runs. Write failures are also rate-limited and never change admission. Graceful signal shutdown and standalone recycling wait up to one second for queued diagnostics before exiting; stalled storage, forced termination and power loss can still lose records. Relay-generated 503 responses carry a server-generated `x-opencodex-link-request-id` for correlation; incoming caller IDs are not trusted as this identifier. + +`owner_lookup` records only refused ownership checks; routine successful checks are silent. `owner_lookup.elapsedMs` measures the owner lookup itself. For `auth_timeout`, `relay_retry` and `relay_failure` report the actual transport-attempt duration in `elapsedMs` (normally about five seconds). A `relay_recovered` event with outcome `recovered` is emitted only after a retry receives an upstream response with `upstream.ok` (HTTP 2xx); it does not prove the streamed body completed. Rate limiting makes request-ID correlation sampled evidence, not a guaranteed trace of every failed request. + +The Home writes `tunnel_transition` records into its own `/link/diagnostics.jsonl` through the same sink, wired in `src/server/index/optional-listeners.ts`: `previousState`, `state`, the cause as `outcome` (the history's causes, or `connected`, `spawned` and `stopped` when nothing failed), `attempt`, the link's `tunnelPort` and, when a retry is scheduled, `retryAfterMs`. The supervisor's `stop()` drains the sink for at most one second. The one-per-second limit per event/outcome applies across links. + +`src/client/link-relay.ts` shares one preflight recovery attempt per request between an owner-lookup `deadline` and the typed `LinkRelayAuthTimeoutError` from `src/client/link-relay-transport.ts`. Only expiry of the credential-free authentication deadline before data dispatch produces that type; arbitrary aborts, authentication failures and transport errors do not qualify. Recovery obtains fresh owner proof where required and opens a fresh physical socket, authenticating it before sending any link credential or body. It never caches owner approval, bypasses security, restarts SSH or OpenCodex, or replays a dispatched body. The owner-lookup and authentication deadlines remain two and five seconds respectively, per attempt. Existing reconnect holding remains separate. + +This recovers those transient preflight deadline paths and records bounded evidence for future failures. It does not establish the historical cause of [#7297](https://github.com/lidge-jun/opencodex/issues/7297); timing clusters and synthetic reproductions alone do not classify past production failures. + ## Connection-bound relay authentication The production relay's complete-send boundary is `src/client/link-relay-transport.ts`. A dedicated HTTP agent opens exactly one TCP socket. Before any admission header or body is sent, a credential-free, five-second challenge proves the stored link/key identity using direction-separated HMACs from `src/link/relay-auth.ts`. The data request uses that same authenticated socket, with a one-use session nonce; a closed socket cannot be silently replaced. A peer lacking this protocol is refused with an upgrade/re-link error, never retried through an unauthenticated send. That refusal is a 503 with `code: "link_relay_auth_failed"` (`src/client/link-relay.ts`), never `home_unreachable`: the peer answered, so it is not a Home outage and must not open device-local fallback ([fallback policy](fallback-policy.md#hub-down)). Home-initiated reverse tunnels use the same check. `LinkRelayDeps.fetchImpl` is a trusted complete-send test seam, not a user setting; production supplies no substitute. Bun's Node HTTP compatibility is confined to this leaf so streamed request caps, backpressure, credential replacement, decompression, response status, caller cancellation and SSE limits remain owned by the existing relay. @@ -91,7 +109,7 @@ The Home learns each Child's version only from what the link already carries; it `src/link/compatibility.ts` publishes `LINK_PROTOCOL_RELEASES`, the first release of each link protocol part (`relay-auth/2` 2.69.0, `link-management-relay` 2.84.0, `link-peer-report/1` 2.91.0), and `linkPeerStatus` turns an observation into `compatible`, `degraded` (the Child reports a feature the Home does not serve), `incompatible` (a refused relay-auth version newer than the last verified challenge) or `unknown` (nothing observed, or a reporting Child whose version or feature list was missing or rejected). Only a Child that sent neither report header gets `versionRange` from its relay-auth release up to the report's release. `GET /api/link/status?compatibility=1` adds `local` and, on a Home, a `peer` block on every link (`attachLinkPeers` in `src/link/status-projection.ts`); without the query the document stays the exact K16 shape an older `ocx link status` validates key by key. The CLI sends the query and accepts either shape. Home `/readyz` advertises the report as `link-peer-report/`. Coverage: `tests/clients/link-peer-compatibility.test.ts` (verdicts, header parsing, a real Home lifecycle against a current and a pre-report Child challenge, the CLI round trip) and `tests/server/link-management-routes.test.ts` (the opt-in query). -`src/server/index/link-relay-sessions.ts` owns a per-listener reservation book. A valid caller proof reserves its peer socket for five seconds (at most 256 outstanding reservations); the data request must consume that nonce on the same peer address/port while its link and credential remain valid. Current and unexpired pending-rotation fingerprints use the same expiry rule as ordinary API-key admission, and the server signs with the fingerprint that matched. The nonce is not an alternative data-plane credential: dispatch still applies its ordinary key and route policy. Removal, rotation expiry, abort and commit are observed anew at dispatch. Replays and a different socket are refused. +`src/server/index/link-relay-sessions.ts` owns a per-listener reservation book. A valid caller proof reserves its peer socket for five seconds (at most 256 outstanding reservations), except a probe-only challenge marked `x-opencodex-link-probe: 1`: the Child's readiness preflight closes after the proof, so the Home answers it with the same listener proof and creates no session and no drain lease, and its nonce redeems nothing. An older Home ignores the marker and reserves as before. Otherwise the data request must consume that nonce on the same peer address/port while its link and credential remain valid. Current and unexpired pending-rotation fingerprints use the same expiry rule as ordinary API-key admission, and the server signs with the fingerprint that matched. The nonce is not an alternative data-plane credential: dispatch still applies its ordinary key and route policy. Removal, rotation expiry, abort and commit are observed anew at dispatch. Replays and a different socket are refused. Last-link `close()` refuses new challenges/work, retains both the bound port and hub-link ingress identity through pending reservations and active response bodies, then closes. An abandoned proof expires; EOF, error and cancellation release an active response. Process `stop()` remains forceful: it retires reservations and closes every socket, which the client cannot reconnect. The regression file `tests/clients/link-relay-bound-transport.test.ts` exercises the actual sockets, shutdown, same-connection proof, refusal to reconnect, bounded reservations, current/pending key lookup and response decoding. This is port-replacement protection, not a claim of encryption against an active transparent local intermediary. SSH continues to own remote-host authentication/encryption; enrollment and supervisor probes are separate boundaries. diff --git a/tests/cli/cli-link.test.ts b/tests/cli/cli-link.test.ts index 1c1df2d698c..771d7c84eff 100644 --- a/tests/cli/cli-link.test.ts +++ b/tests/cli/cli-link.test.ts @@ -144,7 +144,7 @@ describe("ocx link", () => { ...statusResponse, links: statusResponse.links.map(link => ({ ...link, ...NO_PARITY })), }); - expect(calls[0]?.url).toBe("http://127.0.0.1:19101/api/link/status?compatibility=1"); + expect(calls[0]?.url).toBe("http://127.0.0.1:19101/api/link/status?compatibility=1&diagnostics=1"); expect(new Headers(calls[0]?.init?.headers).get("x-opencodex-api-key")).toBe("ocx_admin_test-token"); } finally { output.restore(); diff --git a/tests/clients/client-home-unreachable.test.ts b/tests/clients/client-home-unreachable.test.ts index b8c7103a37c..9ce57ed293f 100644 --- a/tests/clients/client-home-unreachable.test.ts +++ b/tests/clients/client-home-unreachable.test.ts @@ -68,7 +68,7 @@ describe("home_unreachable answers", () => { headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "m", input: "hello" }), }), { tunnelPort: 41001, admissionKey: "fixture-link-key", linkId: "lnk_0123456789abcdef", apiKeyId: "key_link" }, { - tunnel: { connected: () => true, pending: () => false, waitForConnected: async () => true }, + tunnel: { connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => true }, fetchImpl: (async () => { throw new LinkRelayAuthenticationError(unrecognized); }) as unknown as typeof fetch, }); expect(response.status).toBe(503); diff --git a/tests/clients/client-link-relay.test.ts b/tests/clients/client-link-relay.test.ts index 6842a69a2b4..4fd8d8bd41e 100644 --- a/tests/clients/client-link-relay.test.ts +++ b/tests/clients/client-link-relay.test.ts @@ -15,7 +15,11 @@ import { sanitizeLinkResponseHeaders, type LinkRelayClock, type LinkTunnelGate, + type LinkAdmissionResult, } from "../../src/client/link-relay"; +import { LinkRelayAuthenticationError, LinkRelayAuthTimeoutError } from "../../src/client/link-relay-transport"; +import type { LinkDiagnosticRecord } from "../../src/link/diagnostics"; +import { LINK_RELAY_AUTH_PATH } from "../../src/link/relay-auth"; import { HUB_RELAY_REQUEST_BODY_MAX_BYTES } from "../../src/client/hub-relay"; import { HOME_UNREACHABLE_MESSAGE } from "../../src/client/home-unreachable"; import { startMachineListener } from "../../src/client/machine-listener"; @@ -25,7 +29,7 @@ import { removeTreeWithRetry } from "../helpers/remove-tree"; const LINK_KEY = `ocx_data_${"d".repeat(40)}`; const target = { tunnelPort: 12000, admissionKey: LINK_KEY }; -const connectedTunnel: LinkTunnelGate = { connected: () => true, pending: () => false, waitForConnected: async () => true }; +const connectedTunnel: LinkTunnelGate = { connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => true }; // Payload and framing cases provide a proven tunnel; admission cases call the implementation directly. function relayLinkDataRequest( req: Request, @@ -474,7 +478,7 @@ function manualGate(initiallyPending = true) { const state = { pending: initiallyPending, connected: !initiallyPending }; let open: (connected: boolean) => void = () => {}; const tunnel: LinkTunnelGate = { - connected: () => state.connected, + connected: () => ({ kind: state.connected ? "match" : "unavailable" }), pending: () => state.pending, waitForConnected: timeoutMs => { waits.push(timeoutMs); @@ -497,7 +501,7 @@ describe("client link relay while the tunnel reconnects", () => { for (const state of ["missing", "failed", "stopped"] as const) { test(`does not send the committed key or fetch while supervision is ${state}`, async () => { const forwarded: string[] = []; - const unavailable = { pending: () => false, connected: () => false, waitForConnected: async () => false }; + const unavailable = { pending: () => false, connected: () => ({ kind: "unavailable" as const }), waitForConnected: async () => false }; const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "{}" }), target, { ...(state === "missing" ? {} : { tunnel: unavailable }), fetchImpl: (async (_input, init) => { @@ -535,7 +539,7 @@ describe("client link relay while the tunnel reconnects", () => { let calls = 0; const response = await relayLinkDataRequestImpl(new Request("http://127.0.0.1:10100/v1/models"), target, { tunnel: { - connected: () => connected, + connected: () => ({ kind: connected ? "match" : "unavailable" }), pending: () => pending, waitForConnected: async () => { pending = false; connected = false; return true; }, }, @@ -597,22 +601,25 @@ describe("client link relay while the tunnel reconnects", () => { expect(calls).toBe(0); }); - test("a connected tunnel adds no wait", async () => { + test("a connected tunnel adds no wait or routine diagnostic", async () => { let waited = 0; - const tunnel: LinkTunnelGate = { connected: () => true, pending: () => false, waitForConnected: async () => { waited += 1; return true; } }; + const records: LinkDiagnosticRecord[] = []; + const tunnel: LinkTunnelGate = { connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => { waited += 1; return true; } }; const response = await relayLinkDataRequest(relayRequest({ method: "POST", body: "{}" }), target, { tunnel, + diagnostic: row => { records.push(row); }, fetchImpl: (async () => Response.json({ ok: true })) as typeof fetch, }); expect(response.status).toBe(200); expect(waited).toBe(0); + expect(records).toEqual([]); }); test("never replays a request after a reset, even while the tunnel reconnects", async () => { let pending = false; let calls = 0; let waited = 0; - const tunnel: LinkTunnelGate = { connected: () => !pending, pending: () => pending, waitForConnected: async () => { waited += 1; return true; } }; + const tunnel: LinkTunnelGate = { connected: () => ({ kind: pending ? "unavailable" : "match" }), pending: () => pending, waitForConnected: async () => { waited += 1; return true; } }; const response = await relayLinkDataRequest(relayRequest({ method: "POST", body: '{"input":"once"}' }), target, { tunnel, fetchImpl: (async (_input, init) => { @@ -634,7 +641,7 @@ describe("client link relay while the tunnel reconnects", () => { let clock = 1_000; const waits: number[] = []; const tunnel: LinkTunnelGate = { - connected: () => !pending, + connected: () => ({ kind: pending ? "unavailable" : "match" }), pending: () => pending, waitForConnected: async timeoutMs => { waits.push(timeoutMs); @@ -669,7 +676,7 @@ describe("client link relay while the tunnel reconnects", () => { calls = 0; let spentConnected = false; const spent = await relayLinkDataRequest(relayRequest({ method: "POST", body: "{}" }), target, { - tunnel: { connected: () => spentConnected, pending: () => !spentConnected, + tunnel: { connected: () => ({ kind: spentConnected ? "match" : "unavailable" }), pending: () => !spentConnected, waitForConnected: async () => { clock += LINK_RELAY_HOLD_MS; spentConnected = true; return true; } }, now: () => clock, fetchImpl: (async () => { @@ -693,7 +700,7 @@ describe("client link relay while the tunnel reconnects", () => { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ input: "x".repeat(100_000) }), }), { tunnelPort: port, admissionKey: LINK_KEY }, { tunnel: { - connected: () => !pending, + connected: () => ({ kind: pending ? "unavailable" : "match" }), pending: () => pending, waitForConnected: async () => { home = Bun.serve({ hostname: "127.0.0.1", port, async fetch(req) { received.push(await req.text()); return Response.json({ ok: true }); } }); @@ -737,3 +744,227 @@ describe("client link relay while the tunnel reconnects", () => { expect(await response.json()).toEqual({ relayed: true }); }); }); + +test("a transient owner deadline retries before forwarding even when diagnostics throw", async () => { + const { gate, checks } = observedGate(["deadline", "match"]); + let forwarded = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "untouched" }), target, { + tunnel: gate, + diagnostic: () => { throw new Error("diagnostic sink unavailable"); }, + // This trusted fetch seam records only the final data dispatch. + fetchImpl: (async (_url, init) => { forwarded += 1; expect(await new Response(init?.body).text()).toBe("untouched"); return Response.json({ ok: true }); }) as typeof fetch, + }); + expect(response.status).toBe(200); + expect(checks()).toBe(2); + expect(forwarded).toBe(1); +}); + + +function observedGate(outcomes: Array, onCheck?: () => void) { + let checks = 0; + const gate: LinkTunnelGate = { + connected() { + const kind = outcomes[Math.min(checks++, outcomes.length - 1)]; + onCheck?.(); + return { kind, elapsedMs: kind === "deadline" ? 2000 : 1, generation: 2, state: "connected" }; + }, pending: () => false, waitForConnected: async () => false, + }; + return { gate, checks: () => checks }; +} + +test.each(["mismatch", "lookup_error", "stale", "identity_unknown", "identity_mismatch", "unavailable"] as const)( + "does not retry an owner refusal without a deadline: %s", async kind => { + const { gate, checks } = observedGate([kind]); + let forwards = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "private" }), target, { + tunnel: gate, fetchImpl: (async () => { forwards++; return new Response("unexpected"); }) as typeof fetch, + }); + expect(response.status).toBe(503); + expect(checks()).toBe(1); + expect(forwards).toBe(0); + }); + +test("repeated owner deadlines stop after one recovery and correlate the final refusal", async () => { + const { gate, checks } = observedGate(["deadline"]); + const records: LinkDiagnosticRecord[] = []; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST" }), target, { tunnel: gate, diagnostic: row => records.push(row) }); + expect(response.status).toBe(503); + expect(checks()).toBe(2); + expect(response.headers.get("retry-after")).toBe("1"); + const requestId = response.headers.get("x-opencodex-link-request-id"); + expect(requestId).toMatch(/^[0-9a-f-]{36}$/); + expect(records.map(row => [row.event, row.outcome, row.attempt, row.requestId])).toEqual([ + ["relay_retry", "deadline", 1, requestId], ["relay_failure", "deadline", 1, requestId], + ]); +}); + +test("an auth deadline retries fresh admission and delivers the original body once", async () => { + const { gate, checks } = observedGate(["match"]); + const records: LinkDiagnosticRecord[] = []; + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "original bytes" }), target, { + tunnel: gate, diagnostic: row => { records.push(row); throw new Error("logging unavailable"); }, + fetchImpl: (async (_url, init) => { + if (++attempts === 1) throw new LinkRelayAuthTimeoutError(); + expect(await new Response(init?.body).text()).toBe("original bytes"); + return new Response("ok"); + }) as typeof fetch, + }); + expect(await response.text()).toBe("ok"); + expect(checks()).toBe(2); + expect(attempts).toBe(2); + expect(records.map(row => [row.event, row.outcome])).toEqual([ + ["relay_retry", "auth_timeout"], ["relay_recovered", "recovered"], + ]); +}); + +test("an upstream 500 after a preflight retry is not reported as recovered", async () => { + const { gate, checks } = observedGate(["match"]); + const records: LinkDiagnosticRecord[] = []; + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST" }), target, { + tunnel: gate, diagnostic: row => records.push(row), + fetchImpl: (async () => { + if (++attempts === 1) throw new LinkRelayAuthTimeoutError(); + return new Response("upstream failed", { status: 500 }); + }) as typeof fetch, + }); + expect(response.status).toBe(500); + expect(await response.text()).toBe("upstream failed"); + expect(checks()).toBe(2); + expect(attempts).toBe(2); + expect(records.map(row => [row.event, row.outcome])).toEqual([["relay_retry", "auth_timeout"]]); +}); + +test.each([false, true])("owner and auth deadlines share a single recovery budget (auth first: %s)", async authFirst => { + const { gate, checks } = observedGate(authFirst ? ["match", "deadline"] : ["deadline", "match"]); + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "original" }), target, { + tunnel: gate, fetchImpl: (async () => { attempts++; throw new LinkRelayAuthTimeoutError(); }) as typeof fetch, + }); + expect(response.status).toBe(503); + expect(checks()).toBe(2); + expect(attempts).toBe(1); +}); + +test.each([new LinkRelayAuthenticationError(), new Error("socket reset after dispatch")])( + "never retries authentication refusal or ambiguous transport failure: %s", async failure => { + const { gate, checks } = observedGate(["match"]); + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST" }), target, { + tunnel: gate, fetchImpl: (async () => { attempts++; throw failure; }) as typeof fetch, + }); + expect(response.status).toBe(503); + expect(checks()).toBe(1); + expect(attempts).toBe(1); + }); + +test("caller cancellation during an owner deadline prevents both recovery and dispatch", async () => { + const abort = new AbortController(); + const { gate, checks } = observedGate(["deadline", "match"], () => abort.abort()); + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", signal: abort.signal }), target, { + tunnel: gate, fetchImpl: (async () => { attempts++; return new Response("unexpected"); }) as typeof fetch, + }); + expect(response.status).toBe(503); + expect(checks()).toBe(1); + expect(attempts).toBe(0); +}); + +test("a real five-second silent preflight recovers on a new socket and nonce without replaying data", async () => { + let challenges = 0, dispatches = 0; + const records: LinkDiagnosticRecord[] = []; + const nonces: string[] = []; + const ports: number[] = []; + const sessions = createLinkRelaySessions({ fingerprints: () => [serviceApiTokenFingerprint(LINK_KEY)] }); + const home = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req, server) { + const url = new URL(req.url); + if (url.pathname === LINK_RELAY_AUTH_PATH) { + challenges++; + nonces.push(url.searchParams.get("nonce")!); + ports.push(server.requestIP(req)!.port); + expect(req.headers.has("authorization")).toBe(false); + if (challenges === 1) return new Promise(() => {}); + } + return sessions.dispatch(req, server, async data => { + dispatches++; + expect(await data.text()).toBe("sent exactly once"); + return new Response("recovered"); + }); + } }); + servers.push(home); + const { gate, checks } = observedGate(["match"]); + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "sent exactly once" }), { + ...target, tunnelPort: home.port!, linkId: `lnk_${"a".repeat(16)}`, apiKeyId: "ocx_data_fixture", + }, { tunnel: gate, diagnostic: row => records.push(row) }); + expect(response.status).toBe(200); + expect(await response.text()).toBe("recovered"); + expect(challenges).toBe(2); + expect(new Set(nonces).size).toBe(2); + expect(new Set(ports).size).toBe(2); + expect(checks()).toBe(2); + expect(dispatches).toBe(1); + const retry = records.find(row => row.event === "relay_retry" && row.outcome === "auth_timeout"); + expect(retry?.elapsedMs).toBeGreaterThanOrEqual(4500); + expect(retry?.elapsedMs).toBeLessThan(10_000); +}, 12_000); + + +test("repeated auth deadlines stop after the second preflight without dispatching data", async () => { + const { gate, checks } = observedGate(["match"]); + const records: LinkDiagnosticRecord[] = []; + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: "untouched" }), target, { + tunnel: gate, diagnostic: row => records.push(row), fetchImpl: (async () => { + attempts++; + await Bun.sleep(20); + throw new LinkRelayAuthTimeoutError(); + }) as typeof fetch, + }); + expect(response.status).toBe(503); + expect(checks()).toBe(2); + expect(attempts).toBe(2); + expect(records.map(row => [row.event, row.outcome])).toEqual([ + ["relay_retry", "auth_timeout"], ["relay_failure", "auth_timeout"], + ]); + for (const record of records) expect(record.elapsedMs).toBeGreaterThanOrEqual(15); +}); + +test("concurrent requests keep their recovery budget and correlation separate", async () => { + const records: LinkDiagnosticRecord[] = []; + const first = observedGate(["deadline"]); + const second = observedGate(["deadline"]); + const responses = await Promise.all([first, second].map(({ gate }) => relayLinkDataRequestImpl( + relayRequest({ method: "POST" }), target, { tunnel: gate, diagnostic: row => records.push(row) }, + ))); + const ids = responses.map(response => response.headers.get("x-opencodex-link-request-id")); + expect(new Set(ids).size).toBe(2); + for (const id of ids) { + expect(records.filter(row => row.requestId === id).map(row => [row.event, row.attempt])).toEqual([ + ["relay_retry", 1], ["relay_failure", 1], + ]); + } + expect(first.checks()).toBe(2); + expect(second.checks()).toBe(2); +}); + + +test("an oversized streamed body remains a 413 when the credential-free auth deadline expires", async () => { + const records: LinkDiagnosticRecord[] = []; + let attempts = 0; + const response = await relayLinkDataRequestImpl(relayRequest({ method: "POST", body: new ReadableStream({ + start(controller) { controller.enqueue(new Uint8Array(4)); controller.close(); }, + }) }), target, { + tunnel: connectedTunnel, bodyLimitBytes: 3, diagnostic: row => records.push(row), + fetchImpl: (async () => { + attempts++; + // The cap may prefetch while authentication is pending, before data dispatch starts. + await new Promise(resolve => setTimeout(resolve, 0)); + throw new LinkRelayAuthTimeoutError(); + }) as typeof fetch, + }); + expect(response.status).toBe(413); + expect(await response.json()).toEqual({ error: "link relay request body too large" }); + expect(attempts).toBe(1); + expect(records.some(row => row.event === "relay_retry")).toBe(false); +}); diff --git a/tests/clients/client-link-runtime.test.ts b/tests/clients/client-link-runtime.test.ts index cbe3c033165..d4299b341e0 100644 --- a/tests/clients/client-link-runtime.test.ts +++ b/tests/clients/client-link-runtime.test.ts @@ -8,7 +8,7 @@ import { bindClientListener, scheduleStandaloneRecycle, startClientRuntime } fro import type { MachineListenerDeps } from "../../src/client/machine-listener"; import type { OcxClientConnectionConfig } from "../../src/types"; import { clientLinkStatePath, writeClientLinkState } from "../../src/client/link-state"; -import { fixturePath } from "../helpers/repo-root"; +import { fixturePath, repoPath } from "../helpers/repo-root"; import { removeTreeWithRetry } from "../helpers/remove-tree"; import { watchdogMs } from "../helpers/ci-watchdog"; @@ -84,6 +84,77 @@ test("link runtime refuses a busy configured port instead of selecting an epheme } }); +test.each(["SIGINT", "SIGTERM", "restart"])("%s shutdown persists queued diagnostics before process exit", async signal => { + const root = mkdtempSync(join(tmpdir(), "ocx-client-signal-diagnostics-")); + const source = ` + import { mock } from "bun:test"; + import { readFileSync } from "node:fs"; + import { join } from "node:path"; + import { getDefaultConfig, saveConfig } from ${JSON.stringify(repoPath("src/config.ts"))}; + import { startClientRuntime } from ${JSON.stringify(repoPath("src/client/runtime.ts"))}; + import { recordLinkDiagnostic } from ${JSON.stringify(repoPath("src/link/diagnostics.ts"))}; + import { createLocalAttestationChallenge } from ${JSON.stringify(repoPath("src/lib/local-management-attestation.ts"))}; + import { createSystemRestartCapability, SYSTEM_RESTART_CAPABILITY_HEADER, + SYSTEM_RESTART_EXPECTED_PID_HEADER, SYSTEM_RESTART_NONCE_HEADER, SYSTEM_RESTART_PATH + } from ${JSON.stringify(repoPath("src/lib/system-restart-contract.ts"))}; + const record = () => { + recordLinkDiagnostic({ event: "relay_failure", outcome: "auth_failed" }); + recordLinkDiagnostic({ event: "relay_retry", outcome: "deadline" }); + recordLinkDiagnostic({ event: "relay_recovered", outcome: "recovered" }); + }; + if (${JSON.stringify(signal)} === "restart") { + const replacement = await import(${JSON.stringify(repoPath("src/server/restart-replacement.ts"))}); + mock.module(${JSON.stringify(repoPath("src/server/restart-replacement.ts"))}, () => ({ + ...replacement, + spawnReplacementStart: async () => { record(); }, + })); + } + const config = getDefaultConfig(); + config.port = ${freePort()}; + config.runtimeRole = "client"; + config.client = ${JSON.stringify(linkClientState())}; + saveConfig(config); + await startClientRuntime({ block: false }, { restartSupervisor: async () => null }); + if (${JSON.stringify(signal)} === "restart") { + const { attestationSecret, port } = JSON.parse(readFileSync(join(process.env.OPENCODEX_HOME, "runtime-port.json"), "utf8")); + const nonce = createLocalAttestationChallenge(); + const response = await fetch('http://127.0.0.1:' + port + SYSTEM_RESTART_PATH, { + method: "POST", headers: { + Connection: "close", + [SYSTEM_RESTART_EXPECTED_PID_HEADER]: String(process.pid), + [SYSTEM_RESTART_NONCE_HEADER]: nonce, + [SYSTEM_RESTART_CAPABILITY_HEADER]: createSystemRestartCapability(attestationSecret, nonce, "POST", SYSTEM_RESTART_PATH, process.pid, port), + }, + }); + await response.arrayBuffer(); + if (response.status !== 202) throw new Error('restart refused: ' + response.status); + } else { + record(); + process.emit(${JSON.stringify(signal)}, ${JSON.stringify(signal)}); + } + `; + const child = Bun.spawn([process.execPath, "-e", source], { + cwd: root, + env: { ...process.env, HOME: root, USERPROFILE: root, CODEX_HOME: join(root, "codex"), + OPENCODEX_HOME: join(root, "ocx"), OCX_SERVICE_MANAGED: "0" }, + stdout: "ignore", stderr: "pipe", + }); + const errors = new Response(child.stderr).text(); + try { + const exit = await waitFor(() => child.exitCode, "signal shutdown"); + expect({ exit, stderr: await errors }).toEqual({ exit: 0, stderr: "" }); + const records = readFileSync(join(root, "ocx", "link", "diagnostics.jsonl"), "utf8") + .trim().split("\n").map(line => JSON.parse(line)); + expect(records.map(({ event }) => event)).toEqual(["relay_failure", "relay_retry", "relay_recovered"]); + expect(existsSync(join(root, "ocx", "runtime-port.json"))).toBe(false); + } finally { + if (child.exitCode === null) child.kill(); + await child.exited; + await errors; + removeTreeWithRetry(root); + } +}, watchdogMs(60_000)); + test("an ended link recycles a connected sibling after listener cleanup", async () => { const root = mkdtempSync(join(tmpdir(), "ocx-client-sibling-recycle-")); const home = join(root, "ocx"); diff --git a/tests/clients/client-link-tunnel.test.ts b/tests/clients/client-link-tunnel.test.ts index aee48f0ddc4..8c96d3c1102 100644 --- a/tests/clients/client-link-tunnel.test.ts +++ b/tests/clients/client-link-tunnel.test.ts @@ -455,7 +455,7 @@ function supervisorHarness(overrides: Partial = {}) { random: () => 0.5, readProcessStartTime: pid => `fixture-start-${pid}`, linkKey: () => LINK_KEY, - ownsLoopbackListener: async (_port, pid) => ssh.live().some(item => item.child.pid === pid), + inspectLoopbackListener: async (_port, pid) => ({ kind: ssh.live().some(item => item.child.pid === pid) ? "match" : "mismatch", elapsedMs: 1 }), fetchImpl: (async (input, init) => { probes.push({ url: String(input), keyed: new Headers(init?.headers).get("x-opencodex-api-key") === LINK_KEY, cache: init?.cache }); if (view.readyz === "refused") throw Object.assign(new Error("Unable to connect"), { code: "ConnectionRefused" }); @@ -503,7 +503,7 @@ test("a competing listener cannot receive the probe key or relayed traffic befor } }); const h = supervisorHarness({ fetchImpl: fetch, - ownsLoopbackListener: async () => false, + inspectLoopbackListener: async () => ({ kind: "mismatch", elapsedMs: 1 }), }); h.view.sidecar = { ...sidecar(), tunnelPort: competing.port! }; try { @@ -513,7 +513,7 @@ test("a competing listener cannot receive the probe key or relayed traffic befor method: "POST", headers: { "Content-Type": "application/json" }, body: '{"input":"private"}', }), { tunnelPort: competing.port!, admissionKey: LINK_KEY }, { tunnel: h.supervisor, holdMs: 10, fetchImpl: fetch }); expect(response.status).toBe(503); - expect(await h.supervisor.connected()).toBe(false); + expect(await h.supervisor.connected()).toMatchObject({ kind: "unavailable" }); expect(received).toEqual([]); } finally { await h.close(); @@ -557,6 +557,77 @@ test("after a six-minute outage the tunnel is back within a minute of the Home r } }, 30_000); +/** A harness whose ssh spawn throws while `spawnError` is set, as when ssh is missing or the pidfile cannot be written. */ +function failingSpawnHarness() { + const control = { spawnError: null as Error | null }; + const h = supervisorHarness({ + runner: { + async run() { return { code: 0, stdout: "", stderr: "" }; }, + spawnTunnel(argv) { + if (control.spawnError) throw control.spawnError; + return h.ssh.runner.spawnTunnel(argv); + }, + }, + }); + const tunnel = () => { + const status = h.supervisor.status(); + if (status.kind !== "tunnel" || status.state.kind !== "failed") throw new Error(`expected a failed tunnel, got ${JSON.stringify(status)}`); + return status.state; + }; + /** Move the clock to the scheduled retry and run that check; returns the delay to the next one. */ + const toRetry = async (): Promise => { + await h.step(tunnel().retryAt! - h.view.clock); + return tunnel().retryAt! - h.view.clock; + }; + return { h, control, tunnel, toRetry }; +} + +test("a due retry that cannot start ssh counts as a failed retry, so the Child's backoff keeps doubling", async () => { + const { h, control, tunnel, toRetry } = failingSpawnHarness(); + try { + await h.start(); + await h.step(); + expect(h.state()).toBe("connected"); + h.ssh.children[0]!.exit(255, "bind [127.0.0.1]:19001: Address already in use\nCould not request local forwarding."); + await h.settle(); + const failedAt = h.view.clock; + expect(tunnel()).toMatchObject({ reason: "forward", since: failedAt, retryAt: failedAt + 60_000 }); + + control.spawnError = new Error("spawn ssh ENOENT"); + const delays: number[] = []; + for (let retry = 0; retry < 4; retry += 1) delays.push(await toRetry()); + expect(delays).toEqual([120_000, 240_000, 300_000, 300_000]); + expect(tunnel()).toMatchObject({ reason: "forward", since: failedAt, retries: 4, inFlight: false }); + expect(h.ssh.children).toHaveLength(1); + + control.spawnError = null; + await toRetry(); + expect(h.ssh.children).toHaveLength(2); + } finally { + await h.close(); + } +}); + +test("a spawn failure on an authentication retry keeps the five-minute cadence", async () => { + const { h, control, tunnel, toRetry } = failingSpawnHarness(); + try { + await h.start(); + await h.step(); + h.ssh.children[0]!.exit(255, "home@example: Permission denied (publickey)."); + await h.settle(); + expect(tunnel()).toMatchObject({ reason: "auth", retryAt: h.view.clock + 300_000 }); + + control.spawnError = new Error("pidfile is not writable"); + // The attempt never reached the Home's sshd, yet it must not open a faster lane to it. + expect(await toRetry()).toBe(300_000); + expect(tunnel()).toMatchObject({ reason: "auth", retries: 1 }); + expect(await toRetry()).toBe(300_000); + expect(tunnel()).toMatchObject({ reason: "auth", retries: 2 }); + } finally { + await h.close(); + } +}); + test("a reconnect is promoted by a keyed readyz and the healthy ssh is never cut afterwards", async () => { const h = supervisorHarness(); try { @@ -824,7 +895,7 @@ test("an adopted tunnel is probed only when pidfile identity and socket owner ag platform: "darwin", isAlive: pid => pid === 42 || pid === process.pid, readProcessInfo: pid => pid === 42 ? { ppid: process.pid, args: argv.join(" "), startTime: "adopted-start" } : null, - ownsLoopbackListener: async (_port, pid) => pid === 42, + inspectLoopbackListener: async (_port, pid) => ({ kind: pid === 42 ? "match" : "mismatch", elapsedMs: 1 }), }); mkdirSync(join(h.configDir, "link"), { recursive: true }); writeFileSync(clientTunnelPidfilePath(h.configDir), JSON.stringify({ version: 1, @@ -833,7 +904,7 @@ test("an adopted tunnel is probed only when pidfile identity and socket owner ag await h.start(); expect(h.ssh.children).toHaveLength(0); await h.step(); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); expect(h.probes).toHaveLength(1); expect(h.probes[0]?.keyed).toBe(true); } finally { @@ -853,7 +924,7 @@ test("one unreadable adopted identity denies only this admission and recovers", if (unreadable) { unreadable = false; return null; } return { ppid: process.pid, args: argv.join(" "), startTime: "adopted-start" }; }, - ownsLoopbackListener: async (_port, pid) => pid === 42, + inspectLoopbackListener: async (_port, pid) => ({ kind: pid === 42 ? "match" : "mismatch", elapsedMs: 1 }), }); mkdirSync(join(h.configDir, "link"), { recursive: true }); writeFileSync(clientTunnelPidfilePath(h.configDir), JSON.stringify({ version: 1, @@ -868,7 +939,7 @@ test("one unreadable adopted identity denies only this admission and recovers", try { await h.start(); await h.step(); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); unreadable = true; expect((await relay()).status).toBe(503); expect(forwarded).toBe(0); @@ -893,7 +964,7 @@ test("an adopted PID with changed start identity cannot receive another probe or isAlive: pid => pid === 42 || pid === process.pid, readProcessInfo: pid => pid === 42 ? { ppid: process.pid, args: argv.join(" "), startTime: currentStart } : null, - ownsLoopbackListener: async (_port, pid) => pid === 42 || pid >= 31_000, + inspectLoopbackListener: async (_port, pid) => ({ kind: pid === 42 || pid >= 31_000 ? "match" : "mismatch", elapsedMs: 1 }), signal: (_pid, signal) => { signals.push(signal); }, }); mkdirSync(join(h.configDir, "link"), { recursive: true }); @@ -903,7 +974,7 @@ test("an adopted PID with changed start identity cannot receive another probe or try { await h.start(); await h.step(); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); expect(h.probes).toHaveLength(1); currentStart = "reused-start"; h.view.clock += 1_500; @@ -928,13 +999,13 @@ test("an adopted PID with changed start identity cannot receive another probe or test("a connected tunnel stops relaying when another PID takes its port", async () => { let owners: number[] = []; - const h = supervisorHarness({ ownsLoopbackListener: async (_port, pid) => owners.includes(pid) }); + const h = supervisorHarness({ inspectLoopbackListener: async (_port, pid) => ({ kind: owners.includes(pid) ? "match" : "mismatch", elapsedMs: 1 }) }); let forwarded = 0; try { await h.start(); owners = [h.ssh.children[0]!.child.pid]; await h.step(); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); owners = [process.pid]; const response = await relayLinkDataRequest(new Request("http://127.0.0.1:10100/v1/responses", { method: "POST", body: '{"input":"private"}', @@ -944,7 +1015,7 @@ test("a connected tunnel stops relaying when another PID takes its port", async }); expect(response.status).toBe(503); expect(forwarded).toBe(0); - expect(await h.supervisor.connected()).toBe(false); + expect(await h.supervisor.connected()).toMatchObject({ kind: "mismatch" }); } finally { await h.close(); } @@ -953,7 +1024,7 @@ test("a connected tunnel stops relaying when another PID takes its port", async test("each relayed request obtains a fresh asynchronous owner proof", async () => { let lookups = 0; const h = supervisorHarness({ - ownsLoopbackListener: async () => { lookups += 1; await Bun.sleep(1); return true; }, + inspectLoopbackListener: async () => { lookups += 1; await Bun.sleep(1); return { kind: "match", elapsedMs: 1 }; }, }); const relay = () => relayLinkDataRequest(new Request("http://127.0.0.1:10100/v1/models"), { tunnelPort: sidecar().tunnelPort, admissionKey: LINK_KEY }, { @@ -977,7 +1048,7 @@ test("each relayed request obtains a fresh asynchronous owner proof", async () = test("a replacement SSH child needs a new ownership proof before its first keyed probe", async () => { let lookups = 0; - const h = supervisorHarness({ ownsLoopbackListener: async () => { lookups += 1; return true; } }); + const h = supervisorHarness({ inspectLoopbackListener: async () => { lookups += 1; return { kind: "match", elapsedMs: 1 }; } }); try { await h.start(); await h.step(); @@ -1040,19 +1111,19 @@ test("an unreadable read does not end the link; three in a row or an explicit di test("requests wait on a reconnect only while it lasts and are released when it connects or fails", async () => { const h = supervisorHarness(); try { - expect(await h.supervisor.connected()).toBe(false); + expect(await h.supervisor.connected()).toMatchObject({ kind: "unavailable" }); expect(h.supervisor.pending()).toBe(false); await h.start(); expect(h.supervisor.pending()).toBe(true); await h.step(); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); expect(h.supervisor.pending()).toBe(false); expect(await h.supervisor.waitForConnected(15_000)).toBe(true); h.view.readyz = "refused"; h.ssh.children[0]!.exit(255, "Connection reset by peer"); await h.settle(); - expect(await h.supervisor.connected()).toBe(false); + expect(await h.supervisor.connected()).toMatchObject({ kind: "unavailable" }); expect(h.supervisor.pending()).toBe(true); let released: boolean | undefined; const waiting = h.supervisor.waitForConnected(15_000).then(value => { released = value; return value; }); @@ -1062,7 +1133,7 @@ test("requests wait on a reconnect only while it lasts and are released when it h.view.readyz = 200; await h.step(); expect(await waiting).toBe(true); - expect(await h.supervisor.connected()).toBe(true); + expect(await h.supervisor.connected()).toMatchObject({ kind: "match" }); h.ssh.children[1]!.exit(255, "Connection reset by peer"); await h.settle(); @@ -1072,7 +1143,7 @@ test("requests wait on a reconnect only while it lasts and are released when it await h.settle(); expect(await failing).toBe(false); expect(h.supervisor.status()).toMatchObject({ state: { kind: "failed", reason: "hostkey" } }); - expect(await h.supervisor.connected()).toBe(false); + expect(await h.supervisor.connected()).toMatchObject({ kind: "unavailable" }); expect(h.supervisor.pending()).toBe(false); expect(await h.supervisor.waitForConnected(15_000)).toBe(false); } finally { @@ -1124,3 +1195,153 @@ test("the supervisor's own reads parse a file again only after it changed", () = rmSync(dir, { recursive: true, force: true }); } }); + +for (const kind of ["match", "mismatch", "deadline"] as const) { + test(`fresh ${kind} owner proof reports correlated timing and admits only a match`, async () => { + let result: Awaited>> = { kind: "match", elapsedMs: 1 }; + const checks: import("../../src/client/link-relay").LinkAdmissionResult[] = []; + const events: import("../../src/link/diagnostics").LinkDiagnosticRecord[] = []; + const h = supervisorHarness({ inspectLoopbackListener: async () => result, diagnostic: event => { events.push(event); } }); + try { + await h.start(); + await h.step(); + expect(h.state()).toBe("connected"); + events.length = 0; + result = { kind, elapsedMs: 2_003.25 }; + for (let request = 0; request < 2; request += 1) { + checks.push(await h.supervisor.connected({ requestId: `request-${request}` })); + } + expect(checks).toHaveLength(2); + expect(events).toHaveLength(kind === "match" ? 0 : 2); + for (const [index, check] of checks.entries()) { + expect(check).toMatchObject({ kind, elapsedMs: 2_003.25, state: "connected" }); + expect(check.checkId).toEqual(expect.any(String)); + expect(check.checkId!.length).toBeGreaterThan(0); + expect(Number.isSafeInteger(check.generation)).toBe(true); + expect(check.generation!).toBeGreaterThan(0); + if (kind !== "match") expect(events[index]).toEqual({ event: "owner_lookup", outcome: kind, requestId: `request-${index}`, + checkId: check.checkId, generation: check.generation, state: "connected", elapsedMs: 2_003.25 }); + } + expect(checks[1]!.checkId).not.toBe(checks[0]!.checkId); + expect(checks[1]!.generation).toBe(checks[0]!.generation); + } finally { + await h.close(); + } + }); +} + +for (const transition of ["stop", "replacement", "sidecar failure"] as const) { + for (const kind of ["match", "deadline"] as const) { + test(`a delayed ${kind} after ${transition} is stale and cannot admit or report a deadline`, async () => { + type OwnerResult = Awaited>>; + const delayed = deferred(); + const entered = deferred(); + let delayNext = false; + const events: import("../../src/link/diagnostics").LinkDiagnosticRecord[] = []; + const h = supervisorHarness({ + inspectLoopbackListener: async () => { + if (delayNext) { + delayNext = false; + entered.resolve(); + return delayed.promise; + } + return { kind: "match", elapsedMs: 1 }; + }, + diagnostic: event => { events.push(event); }, + }); + try { + await h.start(); + await h.step(); + expect(h.state()).toBe("connected"); + const baseline = await h.supervisor.connected(); + expect(baseline.kind).toBe("match"); + events.length = 0; + delayNext = true; + const admission = h.supervisor.connected({ requestId: "delayed-request" }); + await entered.promise; + if (transition === "stop") { + await h.supervisor.stop(); + expect(h.state()).toBe("idle"); + } else if (transition === "sidecar failure") { + h.view.sidecar = "unreadable"; + await h.step(); + await h.step(); + expect(h.state()).toBe("connected"); + await h.step(); + expect(h.supervisor.status()).toEqual({ kind: "failed", reason: "sidecar_invalid" }); + } else { + h.ssh.children[0]!.exit(255, "Connection reset by peer"); + await h.settle(); + for (let tick = 0; tick < 10 && h.ssh.children.length < 2; tick += 1) await h.step(); + expect(h.ssh.children).toHaveLength(2); + await h.step(); + expect(h.state()).toBe("connected"); + } + delayed.resolve({ kind, elapsedMs: 2_004.5 }); + const check = await admission; + expect(check).toMatchObject({ kind: "stale", elapsedMs: 2_004.5, + generation: baseline.generation, state: transition === "replacement" ? "connected" : "idle" }); + expect(check.checkId).not.toBe(baseline.checkId); + expect(events.filter(event => event.requestId === "delayed-request")).toEqual([ + { event: "owner_lookup", outcome: "stale", requestId: "delayed-request", checkId: check.checkId, + generation: baseline.generation, state: transition === "replacement" ? "connected" : "idle", elapsedMs: 2_004.5 }, + ]); + if (transition === "replacement") { + const fresh = await h.supervisor.connected(); + expect(fresh.kind).toBe("match"); + expect(fresh.generation!).toBeGreaterThan(check.generation!); + } + } finally { + delayed.resolve({ kind: "match", elapsedMs: 1 }); + await h.close(); + } + }); + } +} + +for (const kind of ["match", "mismatch", "deadline"] as const) { + test(`throwing diagnostic sinks preserve ${kind} admission`, async () => { + let result: "match" | "mismatch" | "deadline" = "match"; + let diagnostics = 0; + const h = supervisorHarness({ + inspectLoopbackListener: async () => ({ kind: result, elapsedMs: 7 }), + diagnostic: () => { diagnostics += 1; throw new Error("diagnostic sink unavailable"); }, + }); + try { + await h.start(); + await h.step(); + expect(h.state()).toBe("connected"); + diagnostics = 0; + result = kind; + expect(await h.supervisor.connected({ requestId: "throwing-sinks" })).toMatchObject({ kind, elapsedMs: 7 }); + expect(diagnostics).toBe(kind === "match" ? 0 : 1); + expect(h.state()).toBe("connected"); + } finally { + await h.close(); + } + }); +} + +test("a stop triggered by failure diagnostics leaves admission closed", async () => { + let result: "match" | "mismatch" = "match"; + let stopping: Promise | undefined; + const h = supervisorHarness({ + inspectLoopbackListener: async () => ({ kind: result, elapsedMs: 1 }), + diagnostic: () => { stopping = h.supervisor.stop(); }, + }); + try { + await h.start(); + await h.step(); + expect(h.state()).toBe("connected"); + expect(stopping).toBeUndefined(); + result = "mismatch"; + expect(await h.supervisor.connected()).toMatchObject({ kind: "mismatch" }); + expect(stopping).toBeDefined(); + await stopping; + expect(h.state()).toBe("idle"); + expect(await h.supervisor.connected()).toMatchObject({ kind: "unavailable" }); + } finally { + await stopping; + await h.close(); + } +}); diff --git a/tests/clients/client-machine-listener.test.ts b/tests/clients/client-machine-listener.test.ts index 2b4ab69e3d5..3ef6a52179c 100644 --- a/tests/clients/client-machine-listener.test.ts +++ b/tests/clients/client-machine-listener.test.ts @@ -363,7 +363,7 @@ describe("client machine listener in link mode", () => { const server = startMachineListener(0, { state: linkConnection(options.fingerprint), managementAuthState: authState(), - linkTunnel: { connected: () => true, pending: () => false, waitForConnected: async () => true }, + linkTunnel: { connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => true }, fetchImpl: (async (input, init) => { upstream.push(new Request(String(input), init)); return options.reply?.() ?? Response.json({ relayed: true }); diff --git a/tests/clients/client-runtime.test.ts b/tests/clients/client-runtime.test.ts index 10463ddc30b..45da00f7d8a 100644 --- a/tests/clients/client-runtime.test.ts +++ b/tests/clients/client-runtime.test.ts @@ -1,8 +1,10 @@ import { describe, expect, test } from "bun:test"; -import { mkdtempSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { serviceApiTokenFingerprint } from "../../src/lib/service-secrets"; +import { recordLinkDiagnostic } from "../../src/link/diagnostics"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; import { recycleStandalone, standaloneRecycleEnv } from "../../src/client/runtime"; describe("standalone recycle environment", () => { @@ -129,6 +131,36 @@ describe("standalone recycle", () => { }); }); + test("persists queued diagnostics before a supervised recycle exits", async () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-recycle-diagnostics-")); + const previousHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = dir; + try { + await withServiceMarker("1", async () => { + recordLinkDiagnostic({ event: "relay_failure", outcome: "auth_failed" }); + recordLinkDiagnostic({ event: "relay_retry", outcome: "deadline" }); + recordLinkDiagnostic({ event: "relay_recovered", outcome: "recovered" }); + let exited = false; + await recycleStandalone(serviceApiTokenFingerprint("disconnected-hub-token"), { + configuredPort: () => 10456, + isDesktopSupervised: () => false, + exitProcess: code => { + expect(code).toBe(1); + const records = readFileSync(join(dir, "link", "diagnostics.jsonl"), "utf8") + .trim().split("\n").map(line => JSON.parse(line)); + expect(records.map(({ event }) => event)).toEqual(["relay_failure", "relay_retry", "relay_recovered"]); + exited = true; + }, + }); + expect(exited).toBe(true); + }); + } finally { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(dir); + } + }); + test("a supervised client still exits 1 without spawning", async () => { await withServiceMarker("1", async () => { const calls: string[] = []; diff --git a/tests/clients/link-diagnostics.test.ts b/tests/clients/link-diagnostics.test.ts new file mode 100644 index 00000000000..53cea0da6cc --- /dev/null +++ b/tests/clients/link-diagnostics.test.ts @@ -0,0 +1,296 @@ +import { afterEach, expect, test } from "bun:test"; +import { randomUUID } from "node:crypto"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, renameSync, statSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createLinkDiagnosticSink, type LinkDiagnosticRecord } from "../../src/link/diagnostics"; +import { resetHardenedStateForTests, setAsyncIcaclsRunnerForTests, setIcaclsRunnerForTests, + setPlatformForTests, setStatForTests } from "../../src/lib/windows-secret-acl"; +import { setAsyncWindowsPrincipalRunnerForTests, resetWindowsPrincipalForTests } from "../../src/lib/windows-user-principal"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const roots: string[] = []; +afterEach(() => { for (const root of roots.splice(0)) removeTreeWithRetry(root); }); +function fixture() { + const configDir = mkdtempSync(join(tmpdir(), "ocx-link-diagnostics-")); + roots.push(configDir); + const dir = join(configDir, "link"); + const path = join(dir, "diagnostics.jsonl"); + return { configDir, dir, path }; +} +function records(path: string): Record[] { + return readFileSync(path, "utf8").trim().split("\n").map(line => JSON.parse(line)); +} + +test("writes only bounded metadata and opaque UUID correlation without config ownership side effects", async () => { + const { configDir, path } = fixture(); + const sink = createLinkDiagnosticSink({ configDir, now: () => 100 }); + expect(readdirSync(configDir)).toEqual([]); + const requestId = randomUUID(); + const checkId = randomUUID(); + sink({ + event: "owner_lookup", outcome: "match", requestId, checkId, + generation: 3, state: "connected", elapsedMs: 1.5, attempt: 2, + error: "private-error", body: "private-body", key: "private-key", url: "https://private.test", + linkId: "lnk_0123456789abcdef", suppressed: "private", timestamp: "private", + } as LinkDiagnosticRecord); + sink({ event: "relay_failure", outcome: "auth_timeout", requestId: "lnk_0123456789abcdef", + checkId: `${checkId}\nprivate`, state: "private-state", elapsedMs: Infinity, generation: -1, attempt: 0.5 }); + sink({ event: "private-event", outcome: "match" } as unknown as LinkDiagnosticRecord); + sink({ event: "owner_lookup", outcome: "private-outcome" } as unknown as LinkDiagnosticRecord); + await sink.flush(); + expect(records(path)).toEqual([ + { event: "owner_lookup", outcome: "match", requestId, checkId, generation: 3, + state: "connected", elapsedMs: 1.5, attempt: 2, timestamp: 100 }, + { event: "relay_failure", outcome: "auth_timeout", timestamp: 100 }, + ]); + expect(readFileSync(path, "utf8")).not.toContain("private"); + expect(readdirSync(configDir)).toEqual(["link"]); +}); + +test("deduplicates across changing IDs and reports suppression on the next eligible write", async () => { + const { configDir, path } = fixture(); + let now = 0; + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + const requestId = randomUUID(); + const checkId = randomUUID(); + sink({ event: "owner_lookup", outcome: "match", requestId, checkId }); + now = 999; + for (let i = 0; i < 50; i++) sink({ event: "owner_lookup", outcome: "match", requestId: randomUUID(), generation: i }); + sink({ event: "owner_lookup", outcome: "deadline" }); + sink({ event: "relay_retry", outcome: "match" }); + await sink.flush(); + expect(records(path)).toHaveLength(3); + now = 1_000; + sink({ event: "owner_lookup", outcome: "match" }); + now = 2_000; + sink({ event: "owner_lookup", outcome: "match" }); + await sink.flush(); + const written = records(path); + expect(written[0]).toEqual({ event: "owner_lookup", outcome: "match", requestId, checkId, timestamp: 0 }); + expect(written[3]).toEqual({ event: "owner_lookup", outcome: "match", timestamp: 1_000, suppressed: 50 }); + expect(written[4]).toEqual({ event: "owner_lookup", outcome: "match", timestamp: 2_000 }); +}); + +test("a backward clock step starts a new diagnostic window and preserves suppression counts", async () => { + const { configDir, path } = fixture(); + let now = 10_000; + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + const record: LinkDiagnosticRecord = { event: "relay_failure", outcome: "auth_timeout" }; + sink(record); + now = 10_500; + sink(record); + now = 5_000; + sink(record); + now = 5_500; + sink(record); + now = 6_000; + sink(record); + await sink.flush(); + expect(records(path)).toEqual([ + { ...record, timestamp: 10_000 }, + { ...record, timestamp: 5_000, suppressed: 1 }, + { ...record, timestamp: 6_000, suppressed: 1 }, + ]); +}); + +test("keeps private modes and rotates at 256 KiB with only one bounded backup", async () => { + const { configDir, dir, path } = fixture(); + let now = 0; + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + sink({ event: "owner_lookup", outcome: "match" }); + await sink.flush(); + if (process.platform !== "win32") { + expect(statSync(dir).mode & 0o777).toBe(0o700); + expect(statSync(path).mode & 0o777).toBe(0o600); + chmodSync(dir, 0o755); + chmodSync(path, 0o644); + } + // Seed the boundary directly instead of writing megabytes through the sink. + const seed = `${" ".repeat(256 * 1024 - 1)}\n`; + writeFileSync(path, seed); + writeFileSync(`${path}.1`, "old backup\n"); + now = 1_000; + sink({ event: "owner_lookup", outcome: "match" }); + await sink.flush(); + expect(readFileSync(`${path}.1`, "utf8")).toBe(seed); + expect(records(path)).toEqual([{ event: "owner_lookup", outcome: "match", timestamp: now }]); + writeFileSync(path, seed); + now = 2_000; + sink({ event: "relay_failure", outcome: "transport_error" }); + await sink.flush(); + expect(readdirSync(dir).sort()).toEqual(["diagnostics.jsonl", "diagnostics.jsonl.1"]); + expect(statSync(path).size).toBeLessThanOrEqual(256 * 1024); + expect(statSync(`${path}.1`).size).toBeLessThanOrEqual(256 * 1024); + expect(statSync(path).size + statSync(`${path}.1`).size).toBeLessThanOrEqual(512 * 1024); + if (process.platform !== "win32") { + expect(statSync(dir).mode & 0o777).toBe(0o700); + expect(statSync(path).mode & 0o777).toBe(0o600); + expect(statSync(`${path}.1`).mode & 0o777).toBe(0o600); + } +}); + +test("filesystem and clock failures never escape the sink", async () => { + const { configDir, dir, path } = fixture(); + let now = 0; + writeFileSync(dir, "not a directory"); + const record: LinkDiagnosticRecord = { event: "relay_failure", outcome: "transport_error" }; + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + expect(() => sink(record)).not.toThrow(); + await sink.flush(); + expect(existsSync(path)).toBe(false); + expect(() => createLinkDiagnosticSink({ configDir, now: () => { throw new Error("private"); } })(record)).not.toThrow(); + expect(readFileSync(dir, "utf8")).toBe("not a directory"); + unlinkSync(dir); + now = 999; + sink(record); + await sink.flush(); + expect(existsSync(path)).toBe(false); + now = 1_000; + sink(record); + await sink.flush(); + expect(records(path)).toEqual([{ ...record, timestamp: 1_000, suppressed: 1 }]); +}); + +test.skipIf(process.platform === "win32")("refuses symlink logs without modifying their targets", async () => { + const { configDir, dir, path } = fixture(); + const target = join(configDir, "foreign"); + mkdirSync(dir); + writeFileSync(target, "preserve"); + symlinkSync(target, path); + const sink = createLinkDiagnosticSink({ configDir }); + expect(() => sink({ event: "owner_lookup", outcome: "match" })).not.toThrow(); + await sink.flush(); + expect(readFileSync(target, "utf8")).toBe("preserve"); +}); + + +test("Windows diagnostic appends reuse ACLs only until the file or its permissions change", async () => { + const { configDir, path } = fixture(); + let now = 0, grants = 0; + let permissionFreshness: bigint | undefined; + const observedStat = (target: string) => { + const { dev, ino, ctimeNs } = statSync(target, { bigint: true }); + return { dev, ino, ctimeNs: target === path ? permissionFreshness ?? ctimeNs : ctimeNs }; + }; + setPlatformForTests("win32"); + resetHardenedStateForTests(); + setStatForTests(observedStat); + setAsyncWindowsPrincipalRunnerForTests(async () => ({ success: true, exitCode: 0, timedOut: false, + stdout: "S-1-5-21-1-2-3-1001\nTEST\\user\n" })); + setAsyncIcaclsRunnerForTests(async args => { + if (args[0] === path && args.includes("/grant:r")) grants++; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + const append = async () => { + sink({ event: "relay_failure", outcome: "auth_timeout" }); + await sink.flush(); + now += 1_000; + }; + try { + await append(); + expect(grants).toBe(1); + await append(); + expect(records(path)).toHaveLength(2); + expect(grants).toBe(1); + + // Keep the old object alive so replacing the path cannot recycle its inode. + renameSync(path, `${path}.old`); + writeFileSync(path, ""); + await append(); + expect(records(path)).toHaveLength(1); + expect(grants).toBe(2); + await append(); + expect(grants).toBe(2); + + const beforePermissions = observedStat(path); + chmodSync(path, 0o400); + chmodSync(path, 0o600); + // Model an observable permission change regardless of the host filesystem's ctime granularity. + permissionFreshness = beforePermissions.ctimeNs + 1n; + const afterPermissions = observedStat(path); + expect(afterPermissions.dev).toBe(beforePermissions.dev); + expect(afterPermissions.ino).toBe(beforePermissions.ino); + expect(afterPermissions.ctimeNs).not.toBe(beforePermissions.ctimeNs); + await append(); + expect(records(path)).toHaveLength(3); + expect(grants).toBe(3); + } finally { + await sink.flush(); + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + setAsyncWindowsPrincipalRunnerForTests(null); + resetWindowsPrincipalForTests(); + resetHardenedStateForTests(); + } +}); + +test.each([false, true])("slow Windows ACL work yields and bounds the queue (clock rolls back: %s)", async rollback => { + const { configDir, path } = fixture(); + let enter!: () => void, release!: () => void; + const entered = new Promise(resolve => { enter = resolve; }); + const blocked = new Promise(resolve => { release = resolve; }); + let first = true, now = 0, syncCalls = 0; + setPlatformForTests("win32"); + resetHardenedStateForTests(); + setAsyncWindowsPrincipalRunnerForTests(async () => ({ success: true, exitCode: 0, timedOut: false, + stdout: "S-1-5-21-1-2-3-1001\nTEST\\user\n" })); + setIcaclsRunnerForTests(() => { syncCalls++; throw new Error("synchronous ACL work is forbidden"); }); + setAsyncIcaclsRunnerForTests(async () => { + if (first) { first = false; enter(); await blocked; } + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + const sink = createLinkDiagnosticSink({ configDir, now: () => now }); + try { + sink({ event: "owner_lookup", outcome: "match", attempt: 0 }); + await entered; + for (let attempt = 1; attempt <= 100; attempt++) { + now += 1_000; + sink({ event: "owner_lookup", outcome: "match", attempt }); + } + // A deadline must still be able to fire while the diagnostics child is stalled. + await new Promise(resolve => setTimeout(resolve, 0)); + expect(existsSync(path)).toBe(false); + expect(syncCalls).toBe(0); + if (rollback) { + now = 5_000; + sink({ event: "owner_lookup", outcome: "match", attempt: 101 }); + } + // Shutdown must return even while the ACL runner is still blocked. + let watchdog: ReturnType | undefined; + try { + expect(await Promise.race([ + sink.flush(10).then(() => "drained"), + new Promise(resolve => { watchdog = setTimeout(() => resolve("blocked"), 500); }), + ])).toBe("drained"); + expect(existsSync(path)).toBe(false); + } finally { + clearTimeout(watchdog); + } + release(); + await sink.flush(); + expect(records(path)).toHaveLength(57); // one active record plus at most 56 queued + if (rollback) { + now += 500; + sink({ event: "owner_lookup", outcome: "match", attempt: 102 }); + await sink.flush(); + expect(records(path)).toHaveLength(57); + } + now += rollback ? 500 : 1_000; + const attempt = rollback ? 103 : 101; + sink({ event: "owner_lookup", outcome: "match", attempt }); + await sink.flush(); + expect(records(path).at(-1)).toMatchObject({ attempt, timestamp: now, suppressed: rollback ? 46 : 44 }); + } finally { + release(); + await sink.flush(); + setAsyncIcaclsRunnerForTests(null); + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setAsyncWindowsPrincipalRunnerForTests(null); + resetWindowsPrincipalForTests(); + resetHardenedStateForTests(); + } +}); diff --git a/tests/clients/link-readiness.test.ts b/tests/clients/link-readiness.test.ts new file mode 100644 index 00000000000..6fad9f3c2aa --- /dev/null +++ b/tests/clients/link-readiness.test.ts @@ -0,0 +1,201 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { Server } from "bun"; +import { + clientInitiatedLinkReadiness, + createHomeInitiatedLinkReadiness, + LINK_READINESS_TTL_MS, + type LinkReadiness, +} from "../../src/client/link-readiness"; +import { probeLinkRelayPreflight, type LinkRelayPreflightResult } from "../../src/client/link-relay-transport"; +import type { ClientLinkSupervisorStatus } from "../../src/client/link-tunnel"; +import { startMachineListener } from "../../src/client/machine-listener"; +import { serviceApiTokenFingerprint } from "../../src/lib/service-secrets"; +import { LINK_RELAY_AUTH_PATH, linkRelayProof } from "../../src/link/relay-auth"; +import type { TunnelState } from "../../src/link/tunnel-state"; +import { validateReadyzBody } from "../../src/server/proxy-liveness"; +import type { ManagementAuthState } from "../../src/server/management-auth"; +import type { OcxClientConnectionConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const LINK_KEY = `ocx_data_${"k".repeat(40)}`; +const LINK_ID = `lnk_${"c".repeat(16)}`; +const KEY_ID = "link-key-1"; +const servers: Array<{ stop: (force?: boolean) => unknown }> = []; +let root = ""; +let previousHome: string | undefined; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + root = mkdtempSync(join(tmpdir(), "ocx-link-readiness-")); + process.env.OPENCODEX_HOME = root; + writeFileSync(join(root, "config.json"), JSON.stringify({ port: 0, hostname: "127.0.0.1", providers: {}, defaultProvider: "openai" })); +}); + +afterEach(async () => { + for (const server of servers.splice(0)) await server.stop(true); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (root) removeTreeWithRetry(root); +}); + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise(next => { resolve = next; }); + return { promise, resolve }; +} + +const tunnel = (state: TunnelState, probe?: "unauthorized" | "home_unreachable" | "home_not_ready"): ClientLinkSupervisorStatus => + ({ kind: "tunnel", linkId: LINK_ID, state, pid: 7, ...(probe ? { probe } : {}) }); + +test("a Child-initiated link is ready only while its tunnel is connected and the Home answers the key", () => { + const cases: Array<[ClientLinkSupervisorStatus, LinkReadiness]> = [ + [tunnel({ kind: "connected", since: 1 }), { ready: true }], + [tunnel({ kind: "connected", since: 1 }, "home_not_ready"), { ready: true }], + [tunnel({ kind: "connected", since: 1 }, "home_unreachable"), { ready: false, reason: "home_unreachable" }], + [tunnel({ kind: "connected", since: 1 }, "unauthorized"), { ready: false, reason: "home_unauthorized" }], + [tunnel({ kind: "connecting", since: 1 }), { ready: false, reason: "tunnel_connecting" }], + [tunnel({ kind: "reconnecting", since: 1, attempt: 2, retryAt: 9, inFlight: false }), { ready: false, reason: "tunnel_connecting" }], + [tunnel({ kind: "failed", since: 1, reason: "forward", retryAt: 9, inFlight: true }), { ready: false, reason: "tunnel_connecting" }], + [tunnel({ kind: "failed", since: 1, reason: "forward", retryAt: 9, inFlight: false }), { ready: false, reason: "tunnel_failed" }], + [tunnel({ kind: "idle" }), { ready: false, reason: "tunnel_idle" }], + [{ kind: "stopped" }, { ready: false, reason: "tunnel_idle" }], + [{ kind: "failed", reason: "sidecar_invalid" }, { ready: false, reason: "sidecar_invalid" }], + ]; + for (const [status, expected] of cases) expect(clientInitiatedLinkReadiness(status)).toEqual(expected); +}); + +test("a Home-initiated link shares one preflight, reuses its verdict briefly and never waits long for it", async () => { + const flights: Array>> = []; + const waits: Array<() => void> = []; + let now = 1_000; + let key: string | null = null; + const readiness = createHomeInitiatedLinkReadiness({ + target: { tunnelPort: 23456, linkId: LINK_ID, apiKeyId: KEY_ID }, + linkKey: () => key, + now: () => now, + probe: async target => { + expect(target).toEqual({ tunnelPort: 23456, linkId: LINK_ID, apiKeyId: KEY_ID, admissionKey: LINK_KEY }); + const flight = deferred(); + flights.push(flight); + return await flight.promise; + }, + delay: () => new Promise(resolve => { waits.push(resolve); }), + }); + // Without the committed key nothing is sent. + expect(await readiness()).toEqual({ ready: false, reason: "link_credential_unavailable" }); + expect(flights).toHaveLength(0); + + key = LINK_KEY; + const first = readiness(); + const second = readiness(); + await Promise.resolve(); + expect(flights).toHaveLength(1); + flights[0]!.resolve("ok"); + expect(await first).toEqual({ ready: true }); + expect(await second).toEqual({ ready: true }); + // Cached for the TTL: an unauthenticated caller cannot multiply preflights. + now += LINK_READINESS_TTL_MS - 1; + expect(await readiness()).toEqual({ ready: true }); + expect(flights).toHaveLength(1); + + // A slow Home: the caller gets link_unverified once the bounded wait ends, the preflight keeps going. + now += 1; + const slow = readiness(); + await Promise.resolve(); + expect(flights).toHaveLength(2); + waits.at(-1)!(); + expect(await slow).toEqual({ ready: false, reason: "link_unverified" }); + flights[1]!.resolve("unreachable"); + await flights[1]!.promise; + await Promise.resolve(); + expect(await readiness()).toEqual({ ready: false, reason: "home_unreachable" }); + expect(flights).toHaveLength(2); + + now += LINK_READINESS_TTL_MS; + const refused = readiness(); + await Promise.resolve(); + flights[2]!.resolve("auth_failed"); + expect(await refused).toEqual({ ready: false, reason: "link_auth_failed" }); +}); + +/** A fake Home link listener that answers only the credential-free preflight. */ +function fakeHome(answer: "proof" | "bad-proof" | "missing", seen: Request[]): Server { + const server = Bun.serve({ + port: 0, + hostname: "127.0.0.1", + fetch(req) { + // A copy taken inside the handler: the live request's headers are not readable once it is answered. + seen.push(new Request(req.url, { method: req.method, headers: new Headers(req.headers) })); + const url = new URL(req.url); + if (answer === "missing" || url.pathname !== LINK_RELAY_AUTH_PATH) return new Response("not found", { status: 404 }); + const nonce = url.searchParams.get("nonce") ?? ""; + const proof = answer === "proof" + ? linkRelayProof(serviceApiTokenFingerprint(LINK_KEY), "listener", KEY_ID, LINK_ID, nonce)! + : "0".repeat(64); + return new Response(null, { status: 204, headers: { "x-opencodex-link-proof": proof } }); + }, + }); + servers.push(server); + return server; +} + +test("the preflight probe proves the Home without sending the link key", async () => { + for (const [answer, expected] of [["proof", "ok"], ["bad-proof", "auth_failed"], ["missing", "unrecognized"]] as const) { + const seen: Request[] = []; + const home = fakeHome(answer, seen); + const result = await probeLinkRelayPreflight({ tunnelPort: home.port!, linkId: LINK_ID, apiKeyId: KEY_ID, admissionKey: LINK_KEY }, { timeoutMs: 2_000 }); + expect(result).toBe(expected); + expect(seen).toHaveLength(1); + expect(seen[0]!.method).toBe("GET"); + expect(seen[0]!.headers.get("authorization")).toBeNull(); + expect(seen[0]!.headers.get("x-opencodex-api-key")).toBeNull(); + expect(seen[0]!.headers.get("x-opencodex-link-probe")).toBe("1"); + expect(seen[0]!.url).not.toContain(LINK_KEY); + } + const closed = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response(null) }); + const port = closed.port!; + await closed.stop(true); + expect(await probeLinkRelayPreflight({ tunnelPort: port, linkId: LINK_ID, apiKeyId: KEY_ID, admissionKey: LINK_KEY }, { timeoutMs: 2_000 })) + .toBe("unreachable"); +}); + +function authState(): ManagementAuthState { + return { available: true, token: `ocx_admin_${"a".repeat(43)}`, source: "environment", sessions: new Map(), pairingGrants: new Map() }; +} + +function linkConnection(): OcxClientConnectionConfig { + return { + serverUrl: "http://127.0.0.1:23456", managementUrl: "http://127.0.0.1:23456", managementTransport: "direct", transport: "link", + link: { tunnelPort: 23456, linkId: LINK_ID }, selectedClients: ["codex"], tokenEnv: "OPENCODEX_API_AUTH_TOKEN", + apiKeyId: KEY_ID, tokenFingerprint: serviceApiTokenFingerprint(LINK_KEY), protocolVersion: 1, connectedAt: "2026-09-26T00:00:00.000Z", + }; +} + +test("a Child's /readyz reports an unusable link as 503 pending with a reason, and /healthz stays live", async () => { + let verdict: LinkReadiness = { ready: false, reason: "home_unreachable" }; + const server = startMachineListener(0, { + state: linkConnection(), managementAuthState: authState(), linkKeySource: () => LINK_KEY, + linkTunnel: { connected: () => ({ kind: "match" }), pending: () => false, waitForConnected: async () => false }, + linkReadiness: async () => verdict, + }); + servers.push(server); + expect((await fetch(new URL("/healthz", server.url))).status).toBe(200); + + const down = await fetch(new URL("/readyz", server.url)); + expect(down.status).toBe(503); + expect(down.headers.get("retry-after")).toBe("1"); + const downBody = await down.json() as Record; + expect(downBody).toMatchObject({ service: "opencodex", role: "client", status: "pending", link: { state: "unavailable", reason: "home_unreachable" } }); + // The strict /readyz contract (ocx ready) reads it as ours and not ready. + expect(validateReadyzBody(downBody, server.port!)).toMatchObject({ ready: false, status: "pending" }); + + verdict = { ready: true }; + const up = await fetch(new URL("/readyz", server.url)); + expect(up.status).toBe(200); + const upBody = await up.json() as Record; + expect(upBody).toMatchObject({ status: "ready", link: { state: "available" } }); + expect(validateReadyzBody(upBody, server.port!)).toMatchObject({ ready: true, status: "ready" }); +}); diff --git a/tests/clients/link-relay-bound-transport.test.ts b/tests/clients/link-relay-bound-transport.test.ts index 780f5e9213e..33c19a38f1e 100644 --- a/tests/clients/link-relay-bound-transport.test.ts +++ b/tests/clients/link-relay-bound-transport.test.ts @@ -1,8 +1,8 @@ import { describe, expect, test } from "bun:test"; -import { Agent, request as httpRequest } from "node:http"; +import { Agent, createServer, request as httpRequest, type RequestListener } from "node:http"; import type { Server } from "bun"; import { gzipSync } from "node:zlib"; -import { fetchBoundLinkRelay, LinkRelayAuthenticationError } from "../../src/client/link-relay-transport"; +import { fetchBoundLinkRelay, LinkRelayAuthenticationError, LinkRelayAuthTimeoutError, probeLinkRelayPreflight } from "../../src/client/link-relay-transport"; import { relayLinkDataRequest, HOME_INITIATED_LINK_TUNNEL } from "../../src/client/link-relay"; import { createLinkRelaySessions } from "../../src/server/index/link-relay-sessions"; import { createLinkListenerLifecycle } from "../../src/server/index/link-listener"; @@ -10,7 +10,7 @@ import { createOptionalListenerSet } from "../../src/server/index/optional-liste import { serviceApiTokenFingerprint } from "../../src/lib/service-secrets"; import type { LinkStore } from "../../src/link/store"; import type { OcxConfig } from "../../src/types"; -import { LINK_RELAY_AUTH_PATH, LINK_RELAY_SESSION_HEADER, linkRelayChallenge, linkRelayProof } from "../../src/link/relay-auth"; +import { LINK_RELAY_AUTH_PATH, LINK_RELAY_PROBE_HEADER, LINK_RELAY_SESSION_HEADER, linkRelayChallenge, linkRelayProof } from "../../src/link/relay-auth"; const key = `ocx_data_${"a".repeat(40)}`, pendingKey = `ocx_data_${"b".repeat(40)}`; const keyId = "test-link-key", linkId = "lnk_0123456789abcdef"; @@ -43,6 +43,19 @@ async function rawGet(url: URL, agent: Agent, headers?: Record) }); } +async function withHttpPeer(handler: RequestListener, run: (port: number) => Promise) { + const server = createServer(handler); + try { + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("missing peer port"); + await run(address.port); + } finally { + server.closeAllConnections(); + await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); + } +} + describe("connection-bound link relay", () => { test("direction and both identities are authenticated", () => { const nonce = "c".repeat(64); @@ -68,6 +81,8 @@ describe("connection-bound link relay", () => { expect(response.status).toBe(200); expect(await response.json()).toEqual({ ok: true }); expect(peers).toHaveLength(2); expect(peers[0]).toBe(peers[1]); expect(seen[0]!.has("authorization")).toBe(false); + // A relayed request's preflight reserves its session; only readiness probes are probe-only. + expect(seen[0]!.has(LINK_RELAY_PROBE_HEADER)).toBe(false); expect(seen[1]!.get("authorization")).toBe(`Bearer ${key}`); expect(bodies).toEqual(['{"input":"hello"}']); } finally { await home.stop(true); } @@ -108,16 +123,93 @@ describe("connection-bound link relay", () => { test("an unrecognized or invalid peer gets no credentials or body, and missing identity refuses locally", async () => { const requests: Request[] = []; - const home = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: req => { requests.push(req); return new Response(null, { status: 404 }); } }); + const home = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: req => { + requests.push(req); + return new Response(null, requests.length === 1 ? { status: 404 } + : { status: 204, headers: { "x-opencodex-link-proof": "0".repeat(64) } }); + } }); try { await expect(fetchBoundLinkRelay(target(home.port!), `${home.url}v1/models`, {})).rejects.toBeInstanceOf(LinkRelayAuthenticationError); expect(requests).toHaveLength(1); expect(requests[0]!.headers.has("authorization")).toBe(false); expect(requests[0]!.method).toBe("GET"); expect(requests[0]!.body).toBeNull(); + await expect(fetchBoundLinkRelay(target(home.port!), `${home.url}v1/models`, {})).rejects.toBeInstanceOf(LinkRelayAuthenticationError); + expect(requests).toHaveLength(2); + expect(requests[1]!.headers.has("authorization")).toBe(false); expect(requests[1]!.body).toBeNull(); await expect(fetchBoundLinkRelay({ tunnelPort: home.port!, admissionKey: key }, `${home.url}v1/models`, {})).rejects.toThrow(); - expect(requests).toHaveLength(1); + expect(requests).toHaveLength(2); } finally { await home.stop(true); } }); + test("a silent actual peer expires only the credential-free auth deadline without touching the body", async () => { + const seen: Array<{ url: string; authorization: string | undefined; body: string }> = []; + let bodyRead = false, bodyCancelled = false; + const body = new ReadableStream({ + pull() { bodyRead = true; }, cancel() { bodyCancelled = true; }, + }, { highWaterMark: 0 }); + await withHttpPeer(req => { + const record = { url: req.url!, authorization: req.headers.authorization, body: "" }; + seen.push(record); + req.on("data", chunk => { record.body += chunk.toString(); }); + }, async port => { + await expect(fetchBoundLinkRelay(target(port), `http://127.0.0.1:${port}/v1/responses`, { + method: "POST", headers: { authorization: `Bearer ${key}` }, body, + }, { authTimeoutMs: 20 })).rejects.toBeInstanceOf(LinkRelayAuthTimeoutError); + expect(seen).toHaveLength(1); + expect(new URL(seen[0]!.url, "http://peer").pathname).toBe(LINK_RELAY_AUTH_PATH); + expect(seen[0]!.authorization).toBeUndefined(); expect(seen[0]!.body).toBe(""); + expect(bodyRead).toBe(false); expect(bodyCancelled).toBe(false); expect(body.locked).toBe(false); + }); + }); + + test("parent abort during a real auth probe is not an auth deadline", async () => { + const parent = new AbortController(); + let probes = 0; + await withHttpPeer(() => { probes++; parent.abort(new DOMException("caller stopped", "AbortError")); }, async port => { + const error = await fetchBoundLinkRelay(target(port), `http://127.0.0.1:${port}/v1/models`, { + signal: parent.signal, + }, { authTimeoutMs: 40 }).catch(error => error); + expect(error).toBeInstanceOf(Error); + expect(error).not.toBeInstanceOf(LinkRelayAuthTimeoutError); + expect(probes).toBe(1); + }); + }); + + for (const failure of ["header timeout", "reset"] as const) { + test(`post-proof data ${failure} is never a preflight auth timeout`, async () => { + const parent = new AbortController(); + let timer: ReturnType | undefined; + let probes = 0, dataRequests = 0; + await withHttpPeer((req, res) => { + const url = new URL(req.url!, "http://peer"); + if (url.pathname === LINK_RELAY_AUTH_PATH) { + probes++; + res.writeHead(204, { "x-opencodex-link-proof": linkRelayProof(fp, "listener", keyId, linkId, + url.searchParams.get("nonce")!)! }); + res.end(); return; + } + dataRequests++; + if (failure === "reset") req.socket.destroy(); + else timer = setTimeout(() => parent.abort(new DOMException("data headers timed out", "TimeoutError")), 40); + }, async port => { + try { + const error = await fetchBoundLinkRelay(target(port), `http://127.0.0.1:${port}/v1/models`, { + headers: { authorization: `Bearer ${key}` }, signal: parent.signal, + }, { authTimeoutMs: 20 }).catch(error => error); + expect(error).toBeInstanceOf(Error); + expect(error).not.toBeInstanceOf(LinkRelayAuthTimeoutError); + expect(probes).toBe(1); expect(dataRequests).toBe(1); + } finally { clearTimeout(timer); parent.abort(); } + }); + }); + } + + test("auth deadline overrides must be finite and positive", async () => { + for (const authTimeoutMs of [0, -1, NaN, Infinity]) { + await expect(fetchBoundLinkRelay(target(1), "http://127.0.0.1:1/v1/models", {}, { authTimeoutMs })) + .rejects.toBeInstanceOf(RangeError); + } + }); + test("a removed last link cannot release its port while an authenticated response is still streaming", async () => { let current = store(); const body = gate(), entered = gate(); @@ -164,6 +256,30 @@ describe("connection-bound link relay", () => { } finally { agent.destroy(); await home.stop(true); } }); + test("a readiness preflight is proven without reserving a session or holding the listener's drain", async () => { + let dispatched = 0; + const book = createLinkRelaySessions({ fingerprints: () => [fp], reservationMs: 60_000 }); + const home = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: (req, server) => book.dispatch(req, server, + async () => { dispatched++; return new Response("data"); }) }); + const agent = new Agent({ keepAlive: true, maxSockets: 1 }); + try { + expect(await probeLinkRelayPreflight(target(home.port!), { timeoutMs: 2_000 })).toBe("ok"); + // A probe-marked proof is still a valid answer, but its nonce authorizes nothing afterwards. + const { url, proof } = proofUrl(home.port!); + const answered = await rawGet(url, agent, { [LINK_RELAY_PROBE_HEADER]: "1" }); + expect(answered.status).toBe(204); + expect(answered.headers["x-opencodex-link-proof"]).toBe(proof.expected); + expect((await rawGet(new URL("/v1/models", home.url), agent, { [LINK_RELAY_SESSION_HEADER]: proof.nonce })).status).toBe(404); + expect(dispatched).toBe(0); + // With nothing reserved, a drain settles in the same turn instead of after the lease. + const settled = await Promise.race([ + book.drain().then(() => "drained"), + new Promise(resolve => setImmediate(() => resolve("pending"))), + ]); + expect(settled).toBe("drained"); + } finally { book.abortReservations(); agent.destroy(); await home.stop(true); } + }); + test("real optional-listener lookup accepts current and pending keys, but expiry, commit, abort and removal revoke old proofs", async () => { let current = store(); const cfg = { port: 0, providers: {}, defaultProvider: "test", claudeCode: { enabled: false }, diff --git a/tests/clients/link-supervisor.test.ts b/tests/clients/link-supervisor.test.ts index 76a80ec3921..0f69c12bc3d 100644 --- a/tests/clients/link-supervisor.test.ts +++ b/tests/clients/link-supervisor.test.ts @@ -302,7 +302,7 @@ test("auth failures retry every five minutes, a changed host key never does, and await Promise.resolve(); await Promise.resolve(); const failed = supervisor.status()[0]!.state; - expect(failed).toEqual({ kind: "failed", since: 0, reason: "hostkey" }); + expect(failed).toEqual({ kind: "failed", since: 0, reason: "hostkey", retries: 1 }); for (let second = 0; second < 2 * 60 * 60; second += 10) { current += 10_000; timers[0]!(); @@ -312,7 +312,7 @@ test("auth failures retry every five minutes, a changed host key never does, and await supervisor.stop(); }); -test("the Home's -R supervisor retries a failed forward once a minute and promotes a retry that stays up", async () => { +test("the Home's -R supervisor retries a failed forward with a doubling delay and promotes a retry that stays up", async () => { const fake = fakeRunner(); const timers: Array<() => void> = []; let current = 0; @@ -335,7 +335,9 @@ test("the Home's -R supervisor retries a failed forward once a minute and promot fake.children[0]!.resolve(255); await Promise.resolve(); await Promise.resolve(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 65_000, inFlight: false }); + expect(supervisor.status()[0]!.state).toEqual({ + kind: "failed", since: 5_000, reason: "forward", retryAt: 65_000, inFlight: false, detail: "remote_forward_conflict", + }); current = 64_999; timers[0]!(); expect(fake.children).toHaveLength(1); @@ -347,17 +349,22 @@ test("the Home's -R supervisor retries a failed forward once a minute and promot fake.children[1]!.resolve(255); await Promise.resolve(); await Promise.resolve(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 125_000, inFlight: false }); - current = 125_000; + expect(supervisor.status()[0]!.state).toEqual({ + kind: "failed", since: 5_000, reason: "forward", retryAt: 185_000, inFlight: false, retries: 1, detail: "remote_forward_conflict", + }); + current = 184_999; + timers[0]!(); + expect(fake.children).toHaveLength(2); + current = 185_000; timers[0]!(); expect(fake.children).toHaveLength(3); // A retry in flight is not killed for reading failed, and staying up past the retry grace proves it. - current = 139_999; + current = 199_999; timers[0]!(); expect(supervisor.status()[0]!.state).toMatchObject({ kind: "failed", inFlight: true }); - current = 140_000; + current = 200_000; timers[0]!(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 140_000 }); + expect(supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 200_000 }); expect(supervisor.status()[0]!.pid).toBe(fake.children[2]!.child.pid); await supervisor.stop(); }); @@ -706,7 +713,9 @@ test("a reload neither skips the reconnect backoff nor a failed link's retry del fake.children[1]!.resolve(255); await Promise.resolve(); await Promise.resolve(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 6_000, reason: "forward", retryAt: 66_000, inFlight: false }); + expect(supervisor.status()[0]!.state).toEqual({ + kind: "failed", since: 6_000, reason: "forward", retryAt: 66_000, inFlight: false, detail: "remote_forward_conflict", + }); await supervisor.reload(); expect(fake.children).toHaveLength(2); await supervisor.stop(); @@ -743,9 +752,13 @@ test("the Home's -R supervisor keeps the retry cadence when ssh cannot be starte spawnError = new Error("spawn ssh ENOENT"); current = 65_000; timers[0]!(); - expect(supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 5_000, reason: "forward", retryAt: 125_000, inFlight: false }); + // A retry that cannot even start counts as a failed retry, so the delay doubles, and it keeps + // the peer's last answer because it learned nothing new about the link. + expect(supervisor.status()[0]!.state).toEqual({ + kind: "failed", since: 5_000, reason: "forward", retryAt: 185_000, inFlight: false, retries: 1, detail: "remote_forward_conflict", + }); spawnError = null; - current = 125_000; + current = 185_000; timers[0]!(); expect(fake.children).toHaveLength(2); await supervisor.stop(); diff --git a/tests/clients/link-tunnel-history.test.ts b/tests/clients/link-tunnel-history.test.ts new file mode 100644 index 00000000000..ed82c6ca635 --- /dev/null +++ b/tests/clients/link-tunnel-history.test.ts @@ -0,0 +1,248 @@ +import { afterEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { runLinkCommand, type LinkStatusPayload } from "../../src/cli/link"; +import { createLinkDiagnosticSink, type LinkDiagnosticRecord } from "../../src/link/diagnostics"; +import { attachLinkTunnelDiagnostics, projectLinkStatus } from "../../src/link/status-projection"; +import type { LinkStore } from "../../src/link/store"; +import { createLinkSupervisor, type LinkSupervisor } from "../../src/link/supervisor"; +import { TUNNEL_CAUSES } from "../../src/link/tunnel-history"; +import type { SshChild, SshRunner } from "../../src/link/ssh-runner"; +import { handleManagementAPI } from "../../src/server/management-api"; +import type { ManagementApiDeps } from "../../src/server/management/context"; +import type { OcxConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const LINK_ID = "lnk_0123456789abcdef"; +const ALIAS = "child-mac.example"; +const STALE_FORWARD = "Error: remote port forwarding failed for listen port 19002"; +const temps: string[] = []; + +afterEach(() => { + for (const dir of temps.splice(0)) removeTreeWithRetry(dir); +}); + +function link(): LinkStore["links"][number] { + return { + id: LINK_ID, alias: ALIAS, direction: "hub-initiated", hostKeyFingerprint: "SHA256:abcdefghijklmnop", + tunnelPort: 19002, apiKeyId: "key-link-1", createdAt: "2026-09-25T00:00:00.000Z", + }; +} + +function fakeRunner() { + const children: Array<{ child: SshChild; exit: (code: number) => void; stderr: string }> = []; + const runner: SshRunner = { + async run() { return { code: 0, stdout: "", stderr: "" }; }, + spawnTunnel(argv) { + let exit!: (code: number) => void; + const exited = new Promise(resolve => { exit = resolve; }); + const item = { stderr: "", exit, child: undefined as unknown as SshChild }; + item.child = { + pid: 500 + children.length, argv: [...argv], exited, + stderr: exited.then(() => item.stderr), kill: () => exit(143), + }; + children.push(item); + return item.child; + }, + }; + return { runner, children }; +} + +/** Drive one supervisor with a manual clock and a manual timer; an exit settles before it returns. */ +function harness(initial: LinkStore) { + const fake = fakeRunner(); + const records: LinkDiagnosticRecord[] = []; + let store = initial; + let tick!: () => void; + const clock = { now: 0 }; + const supervisor: LinkSupervisor = createLinkSupervisor({ + readStore: () => store, runner: fake.runner, now: () => clock.now, random: () => 0.5, + readPidfile: () => null, writePidfile: () => {}, removePidfile: () => {}, + setTimer: callback => { tick = callback; return 1 as unknown as ReturnType; }, clearTimer: () => {}, + diagnostic: record => { records.push(record); }, + }); + const settle = async () => { for (let turn = 0; turn < 6; turn += 1) await Promise.resolve(); }; + const exitWith = async (index: number, stderr: string) => { + fake.children[index]!.stderr = stderr; + fake.children[index]!.exit(255); + await settle(); + }; + const at = (ms: number) => { clock.now = ms; tick(); }; + return { supervisor, records, children: fake.children, clock, exitWith, at, setStore: (next: LinkStore) => { store = next; } }; +} + +function dto(supervisor: LinkSupervisor, store: LinkStore) { + return attachLinkTunnelDiagnostics(projectLinkStatus(store, supervisor.status(), { state: "listening", port: 19001 }, + { runtimeRole: "standalone" }, { version: 1, entries: {} }), supervisor.status()); +} + +test("a Home tunnel refused by a stale remote forward keeps retrying and records why", async () => { + const store: LinkStore = { version: 1, listenerPort: 19001, links: [link()] }; + const h = harness(store); + h.supervisor.start(); + try { + h.at(5_000); + expect(h.supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 5_000 }); + // The Home slept: its ssh died, and the Child's sshd still holds port 19002 for that session. + h.clock.now = 10_000; + await h.exitWith(0, "client_loop: send disconnect: Broken pipe"); + h.at(11_000); + expect(h.children).toHaveLength(2); + h.clock.now = 11_500; + await h.exitWith(1, STALE_FORWARD); + expect(h.supervisor.status()[0]!.state).toEqual({ + kind: "failed", since: 11_500, reason: "forward", retryAt: 71_500, inFlight: false, detail: "remote_forward_conflict", + }); + // Every later attempt meets the same held port; the link keeps retrying on a doubling delay. + h.at(71_500); + h.clock.now = 72_000; + await h.exitWith(2, STALE_FORWARD); + expect(h.supervisor.status()[0]!.state).toMatchObject({ kind: "failed", retryAt: 192_000, retries: 1 }); + const failing = dto(h.supervisor, store).links[0]!; + expect(failing).toMatchObject({ state: "failed", reason: "forward" }); + expect(failing.tunnel).toEqual({ + attempt: 2, + nextRetryAt: new Date(192_000).toISOString(), + lastTransition: { from: "failed", to: "failed", at: new Date(72_000).toISOString(), cause: "remote_forward_conflict" }, + lastFailure: { cause: "remote_forward_conflict", at: new Date(72_000).toISOString() }, + }); + // sshd dropped the dead session; the next retry holds the port and outlives its grace. + h.at(192_000); + expect(dto(h.supervisor, store).links[0]!.tunnel?.nextRetryAt).toBeNull(); + h.at(207_000); + expect(h.supervisor.status()[0]!.state).toEqual({ kind: "connected", since: 207_000 }); + expect(dto(h.supervisor, store).links[0]!.tunnel).toMatchObject({ + attempt: 0, nextRetryAt: null, lastTransition: { from: "failed", to: "connected", cause: null }, + lastFailure: { cause: "remote_forward_conflict" }, + }); + + expect(h.records.map(record => [record.previousState, record.state, record.outcome, record.attempt, record.retryAfterMs])).toEqual([ + ["idle", "connecting", "spawned", 1, undefined], + ["connecting", "connected", "connected", 0, undefined], + ["connected", "reconnecting", "network", 0, 1_000], + ["reconnecting", "failed", "remote_forward_conflict", 1, 60_000], + ["failed", "failed", "remote_forward_conflict", 2, 120_000], + ["failed", "connected", "connected", 0, undefined], + ]); + expect(h.records.every(record => record.event === "tunnel_transition" && record.tunnelPort === 19002)).toBe(true); + expect(JSON.stringify(h.records)).not.toContain(ALIAS); + } finally { + await h.supervisor.stop(); + } + expect(h.records.at(-1)).toMatchObject({ previousState: "connected", state: "idle", outcome: "stopped" }); +}); + +test("a Home link without a listener port is retried instead of failing for good", async () => { + const store: LinkStore = { version: 1, listenerPort: null, links: [link()] }; + const h = harness(store); + h.supervisor.start(); + try { + expect(h.children).toHaveLength(0); + expect(h.supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 0, reason: "forward", retryAt: 60_000, inFlight: false }); + h.at(60_000); + expect(h.supervisor.status()[0]!.state).toEqual({ kind: "failed", since: 0, reason: "forward", retryAt: 180_000, inFlight: false, retries: 1 }); + expect(h.records.map(record => record.outcome)).toEqual(["listener_unavailable", "listener_unavailable"]); + // The listener comes back: the changed record starts a fresh attempt at once. + h.setStore({ ...store, listenerPort: 19001 }); + await h.supervisor.reload(); + expect(h.children).toHaveLength(1); + expect(h.supervisor.status()[0]!.state).toEqual({ kind: "connecting", since: 60_000 }); + expect(h.supervisor.status()[0]!.history).toMatchObject({ attempt: 1, lastFailure: null }); + } finally { + await h.supervisor.stop(); + } +}); + +test("a changed host key stays terminal and is recorded once", async () => { + const store: LinkStore = { version: 1, listenerPort: 19001, links: [link()] }; + const h = harness(store); + h.supervisor.start(); + try { + await h.exitWith(0, "Host key verification failed."); + for (let at = 1_000; at <= 24 * 60 * 60_000; at += 60_000) h.at(at); + expect(h.children).toHaveLength(1); + expect(h.records.filter(record => record.outcome === "hostkey")).toHaveLength(1); + expect(dto(h.supervisor, store).links[0]!.tunnel).toMatchObject({ nextRetryAt: null, lastFailure: { cause: "hostkey" } }); + } finally { + await h.supervisor.stop(); + } +}); + +test("the diagnostics file keeps only the fixed tunnel fields", async () => { + const configDir = mkdtempSync(join(tmpdir(), "ocx-link-tunnel-diag-")); + temps.push(configDir); + const sink = createLinkDiagnosticSink({ configDir, now: () => 1_000 }); + sink({ + event: "tunnel_transition", outcome: "remote_forward_conflict", state: "failed", previousState: "reconnecting", + attempt: 3, tunnelPort: 19002, retryAfterMs: 120_000, + ...{ alias: ALIAS, argv: ["ssh", "-R", "127.0.0.1:19002:127.0.0.1:19001", "--", ALIAS], stderr: STALE_FORWARD }, + } as LinkDiagnosticRecord); + sink({ event: "tunnel_transition", outcome: "connected", state: "connected", previousState: "bogus", tunnelPort: 70_000 } as LinkDiagnosticRecord); + sink({ event: "tunnel_transition", outcome: "not-a-cause" } as unknown as LinkDiagnosticRecord); + // Every cause the history can name is an allowed outcome; none is dropped as unknown. + for (const cause of TUNNEL_CAUSES) sink({ event: "tunnel_transition", outcome: cause, state: "failed" }); + await sink.flush(); + const lines = readFileSync(join(configDir, "link", "diagnostics.jsonl"), "utf8").trim().split("\n").map(line => JSON.parse(line)); + expect(lines.slice(0, 2)).toEqual([ + { event: "tunnel_transition", outcome: "remote_forward_conflict", state: "failed", previousState: "reconnecting", + attempt: 3, retryAfterMs: 120_000, tunnelPort: 19002, timestamp: 1_000 }, + { event: "tunnel_transition", outcome: "connected", state: "connected", timestamp: 1_000 }, + ]); + // remote_forward_conflict was already written once in this one-second window. + expect(lines.slice(2).map(line => line.outcome)).toEqual(TUNNEL_CAUSES.filter(cause => cause !== "remote_forward_conflict")); +}); + +function statusDeps(supervisor: LinkSupervisor, store: LinkStore): ManagementApiDeps { + return { + readLinkStore: () => store, + linkSupervisor: () => supervisor, + linkListener: () => ({ + ensureStarted: async () => {}, close: async () => {}, onAuthenticatedCatalog: () => () => {}, + status: () => ({ state: "listening", port: store.listenerPort, reason: null }), + }), + } as unknown as ManagementApiDeps; +} + +async function getStatus(path: string, deps: ManagementApiDeps): Promise> { + const url = new URL(`http://127.0.0.1${path}`); + const config = { port: 10100, hostname: "127.0.0.1", runtimeRole: "standalone", defaultProvider: "mock", providers: {}, apiKeys: [] } as unknown as OcxConfig; + const response = await handleManagementAPI(new Request(url, { headers: { host: "127.0.0.1" } }), url, config, deps, "admin-token", + { isPaired: () => true, isCurrent: () => true, revokeCurrent: () => true }, { trustedLoopback: true, guiSessionIssuance: null }); + expect(response?.status).toBe(200); + return await response!.json() as Record; +} + +test("GET /api/link/status adds tunnel diagnostics only on request, and ocx link status validates them", async () => { + const store: LinkStore = { version: 1, listenerPort: 19001, links: [link()] }; + const h = harness(store); + h.supervisor.start(); + try { + await h.exitWith(0, STALE_FORWARD); + const deps = statusDeps(h.supervisor, store); + const plain = await getStatus("/api/link/status", deps); + expect(Object.keys((plain.links as object[])[0]!)).not.toContain("tunnel"); + const detailed = await getStatus("/api/link/status?compatibility=1&diagnostics=1", deps); + const tunnel = { attempt: 1, nextRetryAt: new Date(60_000).toISOString(), lastTransition: { from: "connecting", to: "failed", + at: new Date(0).toISOString(), cause: "remote_forward_conflict" }, lastFailure: { cause: "remote_forward_conflict", at: new Date(0).toISOString() } }; + expect((detailed.links as Array>)[0]!.tunnel).toEqual(tunnel); + + const printed: string[] = []; + const original = console.log; + console.log = (...args: unknown[]) => { printed.push(args.map(String).join(" ")); }; + try { + const fetchImpl = (async () => Response.json(detailed)) as unknown as typeof fetch; + expect(await runLinkCommand(["status", "--json"], { baseUrl: "http://127.0.0.1:19101", fetchImpl, readAdminToken: () => "ocx_admin_t" })).toBe(0); + expect((JSON.parse(printed[0]!) as LinkStatusPayload).links[0]!.tunnel).toEqual(tunnel); + for (const bad of [{ ...tunnel, stderr: STALE_FORWARD }, { ...tunnel, lastFailure: { cause: STALE_FORWARD, at: tunnel.lastFailure.at } }, { ...tunnel, attempt: -1 }]) { + const body = { ...detailed, links: [{ ...(detailed.links as object[])[0]!, tunnel: bad }] }; + const badFetch = (async () => Response.json(body)) as unknown as typeof fetch; + expect(await runLinkCommand(["status", "--json"], { baseUrl: "http://127.0.0.1:19101", fetchImpl: badFetch, readAdminToken: () => "ocx_admin_t" })).not.toBe(0); + } + } finally { + console.log = original; + } + } finally { + await h.supervisor.stop(); + } +}); diff --git a/tests/clients/link-tunnel-state.test.ts b/tests/clients/link-tunnel-state.test.ts index dd2a3d3db0e..fe5b848065e 100644 --- a/tests/clients/link-tunnel-state.test.ts +++ b/tests/clients/link-tunnel-state.test.ts @@ -86,13 +86,49 @@ test("ssh stderr is classified by its retry policy", () => { const cases = [ ["Permission denied (publickey).", "auth"], ["Host key verification failed.", "hostkey"], - ["Error: remote port forwarding failed for listen port 20100", "forward"], + ["Error: remote port forwarding failed for listen port 20100", "remote_forward"], + ["bind [127.0.0.1]:20100: Address already in use", "forward"], + ["channel_setup_fwd_listener_tcpip: cannot listen to port: 20100\nCould not request local forwarding.", "forward"], ["ssh: connect to host x port 22: Connection refused", "network"], ["unexpected diagnostic", "unknown"], ] as const; for (const [stderr, expected] of cases) expect(classifySshStderr(stderr)).toBe(expected); }); +test("a remote forward refused by the peer fails the link as forward and keeps the finer cause", () => { + const failed = reduceTunnel({ kind: "connected", since: 0 }, { type: "exit", now: 1_000, stderrClass: "remote_forward" }, () => 0.5, POLICY); + expect(failed).toEqual({ kind: "failed", since: 1_000, reason: "forward", retryAt: 61_000, inFlight: false, detail: "remote_forward_conflict" }); + const retry = reduceTunnel(failed, { type: "spawn", now: 61_000 }, () => 0.5, POLICY); + // The port was released meanwhile; a network exit of the retry drops the stale detail. + expect(reduceTunnel(retry, { type: "exit", now: 61_500, stderrClass: "network" }, () => 0.5, POLICY)) + .toEqual({ kind: "failed", since: 1_000, reason: "timeout", retryAt: 181_500, inFlight: false, retries: 1 }); + expect(reduceTunnel(retry, { type: "exit", now: 61_500, stderrClass: "remote_forward" }, () => 0.5, POLICY)) + .toEqual({ kind: "failed", since: 1_000, reason: "forward", retryAt: 181_500, inFlight: false, retries: 1, detail: "remote_forward_conflict" }); +}); + +test("each failed retry doubles the delay up to five minutes, and a link that never heals is still retried", () => { + let state: TunnelState = reduceTunnel({ kind: "connected", since: 0 }, { type: "exit", now: 0, stderrClass: "remote_forward" }, () => 0.5, POLICY); + const delays: number[] = []; + let last = 0; + for (let now = 0; now <= 24 * HOUR_MS; now += 1_000) { + if (!dueForSpawn(state, now)) continue; + delays.push(now - last); + state = reduceTunnel(state, { type: "spawn", now }, () => 0.5, POLICY); + state = reduceTunnel(state, { type: "exit", now, stderrClass: "remote_forward" }, () => 0.5, POLICY); + last = now; + } + expect(delays.slice(0, 5)).toEqual([60_000, 120_000, 240_000, 300_000, 300_000]); + expect(Math.max(...delays)).toBe(300_000); + // About 12 attempts an hour at the cap, for as long as the link stays down. + expect(delays.length).toBeGreaterThan(24 * 11); + expect(state).toMatchObject({ kind: "failed", since: 0, reason: "forward", detail: "remote_forward_conflict" }); + // Connecting resets the cadence: the next failure starts at a minute again. + const healed = reduceTunnel(reduceTunnel(state, { type: "spawn", now: 25 * HOUR_MS }, () => 0.5, POLICY), { type: "ready", now: 25 * HOUR_MS }, () => 0.5, POLICY); + expect(healed).toEqual({ kind: "connected", since: 25 * HOUR_MS }); + expect(reduceTunnel(healed, { type: "exit", now: 26 * HOUR_MS, stderrClass: "forward" }, () => 0.5, POLICY)) + .toMatchObject({ retryAt: 26 * HOUR_MS + 60_000 }); +}); + test("the client policy retries timeout and forward after a minute and auth after five minutes, never a host key", () => { const forward = reduceTunnel({ kind: "connected", since: 0 }, { type: "exit", now: 1_000, stderrClass: "forward" }, () => 0.5, POLICY); expect(forward).toEqual({ kind: "failed", since: 1_000, reason: "forward", retryAt: 61_000, inFlight: false }); @@ -119,16 +155,16 @@ test("a retry runs while the link still reads failed, connects on ready and fall let state = reduceTunnel(failed, { type: "spawn", now: 60_100 }, () => 0.5, POLICY); expect(state).toEqual({ ...failed, inFlight: true }); expect(dueForSpawn(state, 10 * HOUR_MS)).toBe(false); - // A transient exit of the retry keeps the original failure time and waits another minute. + // A transient exit of the retry keeps the original failure time and waits twice as long. state = reduceTunnel(state, { type: "exit", now: 61_000, stderrClass: "network" }, () => 0.5, POLICY); - expect(state).toEqual({ kind: "failed", since: 100, reason: "timeout", retryAt: 121_000, inFlight: false }); + expect(state).toEqual({ kind: "failed", since: 100, reason: "timeout", retryAt: 181_000, inFlight: false, retries: 1 }); // A hung retry is not timed out by the reducer; ssh's own keepalive bounds it. - state = reduceTunnel(state, { type: "spawn", now: 121_000 }, () => 0.5, POLICY); - expect(reduceTunnel(state, { type: "tick", now: 121_000 + 10 * FAILED_AFTER_MS }, () => 0.5, POLICY)).toBe(state); - expect(reduceTunnel(state, { type: "ready", now: 122_000 }, () => 0.5, POLICY)).toEqual({ kind: "connected", since: 122_000 }); + state = reduceTunnel(state, { type: "spawn", now: 181_000 }, () => 0.5, POLICY); + expect(reduceTunnel(state, { type: "tick", now: 181_000 + 10 * FAILED_AFTER_MS }, () => 0.5, POLICY)).toBe(state); + expect(reduceTunnel(state, { type: "ready", now: 182_000 }, () => 0.5, POLICY)).toEqual({ kind: "connected", since: 182_000 }); // A retry that meets a changed host key stops retrying. - expect(reduceTunnel(state, { type: "exit", now: 122_000, stderrClass: "hostkey" }, () => 0.5, POLICY)) - .toEqual({ kind: "failed", since: 100, reason: "hostkey" }); + expect(reduceTunnel(state, { type: "exit", now: 182_000, stderrClass: "hostkey" }, () => 0.5, POLICY)) + .toEqual({ kind: "failed", since: 100, reason: "hostkey", retries: 2 }); }); test("auth retries reach the Home's sshd at most 12 times in any hour", () => { diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index c431e963f2b..b6a14241cf2 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,4 @@ -{ "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", +{ "link-tunnel-history.test.ts": "clients", "link-readiness.test.ts": "clients", "link-diagnostics.test.ts": "clients", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "service-proxy-env.test.ts": "service", "bridge-terminal-refusal.test.ts": "adapters", "loop-terminal-refusal.test.ts": "images", diff --git a/tests/server/port-reclaim.test.ts b/tests/server/port-reclaim.test.ts index 8025e77317a..07eccd6a7a1 100644 --- a/tests/server/port-reclaim.test.ts +++ b/tests/server/port-reclaim.test.ts @@ -9,6 +9,8 @@ import { parseListenEntriesFromSs, scanListenPidsForAddress, ownsIpv4LoopbackListener, + inspectIpv4LoopbackListener, + type LoopbackOwnerLookupIo, parseIpv4LoopbackListenPidsFromNetstat, parseProcLoopbackListenInodes, reclaimListenPort, @@ -85,8 +87,16 @@ describe("exact IPv4 loopback listener ownership", () => { }; expect(await ownsIpv4LoopbackListener(25000, 42, io)).toBe(true); expect(paths).toEqual(["/proc/42/fd"]); + expect((await inspectIpv4LoopbackListener(25000, 42, io)).kind).toBe("match"); expect(await ownsIpv4LoopbackListener(25000, 43, { ...io, readFdLink: async () => "socket:[44002]" })).toBe(false); + expect((await inspectIpv4LoopbackListener(25000, 43, { ...io, + readFdLink: async () => "socket:[44002]" })).kind).toBe("mismatch"); + expect((await inspectIpv4LoopbackListener(25000, 42, { ...io, + readProc: async () => { throw new Error("private proc detail"); } })).kind).toBe("lookup_error"); + expect((await inspectIpv4LoopbackListener(25000, 42, { ...io, + readProc: async () => tcp + "1: 0100007F:61A8 00000000:0000 0A 0 0 0 501 0 44003 1\n", + })).kind).toBe("mismatch"); }); test("Windows netstat ignores a foreign IPv6 listener on the same numeric port", () => { @@ -97,6 +107,119 @@ describe("exact IPv4 loopback listener ownership", () => { expect(parseIpv4LoopbackListenPidsFromNetstat(rows, 25000)).toEqual([42]); }); + test.each(["darwin", "win32"] as const)("%s reports match, mismatch and ambiguous owners with boolean compatibility", async platform => { + const row = (pid: number) => platform === "darwin" ? `${pid}\n` + : `TCP 127.0.0.1:25000 0.0.0.0:0 LISTENING ${pid}\n`; + for (const [stdout, kind] of [[row(42), "match"], [row(43), "mismatch"], + [row(42) + row(43), "mismatch"], ["", "mismatch"]] as const) { + let now = 100; + const io: LoopbackOwnerLookupIo = { platform, nowMs: () => now, + run: async (_file, _args, timeoutMs) => { + expect(timeoutMs).toBe(2_000); + now += 7; + return stdout; + } }; + expect(await inspectIpv4LoopbackListener(25000, 42, io)).toEqual({ kind, elapsedMs: 7 }); + expect(await ownsIpv4LoopbackListener(25000, 42, io)).toBe(kind === "match"); + } + }); + + test.each([ + ["darwin", {}, "mismatch"], + ["darwin", { signal: undefined }, "mismatch"], + ["darwin", { killed: undefined }, "mismatch"], + ["win32", {}, "lookup_error"], + ["darwin", { killed: true }, "lookup_error"], + ["darwin", { signal: "SIGTERM" }, "lookup_error"], + ["darwin", { code: 2 }, "lookup_error"], + ["darwin", { code: "1" }, "lookup_error"], + ["darwin", { stdout: "42\n" }, "lookup_error"], + ["darwin", { stderr: "lsof: permission denied" }, "lookup_error"], + ["darwin", { stdout: "\n" }, "lookup_error"], + ["darwin", { stderr: "\n" }, "lookup_error"], + ["darwin", { stdout: undefined }, "lookup_error"], + ["darwin", { stderr: undefined }, "lookup_error"], + ] as const)("%s empty lsof exit 1 with overrides %j reports %s", async (platform, overrides, kind) => { + const failure = Object.assign(new Error("Command failed: lsof"), { + code: 1, killed: false, signal: null, stdout: "", stderr: "", ...overrides, + }); + const io: LoopbackOwnerLookupIo = { platform, nowMs: () => 100, + run: async () => { throw failure; } }; + expect(await inspectIpv4LoopbackListener(25000, 42, io)).toEqual({ kind, elapsedMs: 0 }); + expect(await ownsIpv4LoopbackListener(25000, 42, io)).toBe(false); + }); + + test.each([ + [Object.assign(new Error("private spawn detail"), { code: "ENOENT" }), "lookup_error"], + [{ killed: true, signal: "SIGTERM", code: null }, "deadline"], + [{ killed: true, signal: "SIGTERM", code: "ERR_CHILD_PROCESS_STDIO_MAXBUFFER" }, "lookup_error"], + [{ killed: false, signal: "SIGTERM", code: null }, "lookup_error"], + [null, "lookup_error"], + ] as const)("rejected lookup %j returns only bounded metadata", async (failure, kind) => { + const io: LoopbackOwnerLookupIo = { platform: "darwin", nowMs: () => 100, + run: async () => { throw failure; } }; + expect(await inspectIpv4LoopbackListener(25000, 42, io)).toEqual({ kind, elapsedMs: 0 }); + expect(await ownsIpv4LoopbackListener(25000, 42, io)).toBe(false); + }); + + test.each(["resolve", "reject"] as const)("deadline stays final after late lookup %s", async completion => { + let now = 100; + let expire!: () => void; + let finish!: (stdout: string) => void; + let fail!: (error: Error) => void; + let cancelled = 0; + const io: LoopbackOwnerLookupIo = { platform: "darwin", nowMs: () => now, + run: () => new Promise((resolve, reject) => { finish = resolve; fail = reject; }), + scheduleDeadline: (callback, timeoutMs) => { + expect(timeoutMs).toBe(2_000); + expire = callback; + return () => { cancelled += 1; }; + } }; + const pending = inspectIpv4LoopbackListener(25000, 42, io); + now += 2_000; + expire(); + const result = await pending; + expect(result).toEqual({ kind: "deadline", elapsedMs: 2_000 }); + expect(cancelled).toBe(1); + now += 100; + if (completion === "resolve") finish("42\n"); + else fail(new Error("late private error")); + await Promise.resolve(); + expect(await pending).toEqual({ kind: "deadline", elapsedMs: 2_000 }); + expect(result).toEqual({ kind: "deadline", elapsedMs: 2_000 }); + const booleanPending = ownsIpv4LoopbackListener(25000, 42, io); + expire(); + expect(await booleanPending).toBe(false); + finish("42\n"); + }); + + test("successful lookup cancels the deadline and retains its result", async () => { + let expire!: () => void; + let cancelled = false; + const pending = inspectIpv4LoopbackListener(25000, 42, { platform: "darwin", nowMs: () => 10, + run: async () => "42\n", scheduleDeadline: callback => { + expire = callback; + return () => { cancelled = true; }; + } }); + expect(await pending).toEqual({ kind: "match", elapsedMs: 0 }); + expect(cancelled).toBe(true); + expire(); + expect(await pending).toEqual({ kind: "match", elapsedMs: 0 }); + }); + + test.each([[0, 42], [65536, 42], [1.5, 42], [25000, 0], [25000, NaN]])( + "invalid port/PID %j/%j fails closed before lookup", async (port, pid) => { + expect(await inspectIpv4LoopbackListener(port, pid, { nowMs: () => 1, + run: async () => { throw new Error("must not run"); } })).toEqual({ kind: "mismatch", elapsedMs: 0 }); + }); + + test.skipIf(process.platform !== "darwin")("a closed IPv4 loopback port reports mismatch with real lsof", async () => { + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("ok") }); + const port = server.port!; + await server.stop(true); + expect((await inspectIpv4LoopbackListener(port, process.pid)).kind).toBe("mismatch"); + }); + test.skipIf(process.platform !== "darwin")("a foreign ::1 listener does not veto the owned IPv4 socket", async () => { const ipv4 = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: () => new Response("v4") }); const port = ipv4.port!;