fix(link): keep retrying failed Home tunnels, report Child link readiness, record tunnel diagnostics

docs-site/src/content/docs/fr/guides/remote-link.md

@@ -55,7 +55,7 @@ Le rôle **Child** exige également qu’OpenCodex fonctionne en mode autonome s
 
 - **Connected** signifie que le tunnel SSH est prêt et que Child peut utiliser la liaison Home.
 - **Reconnecting** signifie que le tunnel est réessayé. Les requêtes peuvent temporairement renvoyer `503` avec `Retry-After`. Sur un Child connecté depuis son propre tableau de bord, une requête attend d’abord jusqu’à 15 secondes le retour du tunnel.
-- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Le côté qui ouvre le tunnel SSH réessaie de lui-même après une mise en veille, une panne ou un redémarrage (Home pour un Child qu’il a ajouté, le Child pour une liaison qu’il a rejointe depuis son propre tableau de bord) : environ une fois par minute après un délai dépassé ou une erreur de redirection, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. Pour réessayer tout de suite, choisissez **Retry** dans le tableau de bord du Home ou exécutez `ocx link reconnect --link-id <id>` sur le Home. Le tunnel de cette liaison redémarre sans redémarrer OpenCodex.
+- **Failed** signifie que la liaison nécessite une intervention. Vérifiez l’authentification SSH, la clé d’hôte confirmée, la redirection ou le délai indiqué. Le côté qui ouvre le tunnel SSH réessaie de lui-même après une mise en veille, une panne ou un redémarrage (Home pour un Child qu’il a ajouté, le Child pour une liaison qu’il a rejointe depuis son propre tableau de bord) : une minute après un délai dépassé ou une erreur de redirection, puis avec une attente qui double après chaque nouvel échec, jusqu’à cinq minutes, et toutes les cinq minutes après une erreur d’authentification. Une clé d’hôte modifiée n’est jamais réessayée. Pour réessayer tout de suite, choisissez **Retry** dans le tableau de bord du Home ou exécutez `ocx link reconnect --link-id <id>` sur le Home. Le tunnel de cette liaison redémarre sans redémarrer OpenCodex.
 
 Une liaison en échec ne bascule pas silencieusement vers un fournisseur local.
 

docs-site/src/content/docs/guides/remote-link.md

@@ -59,12 +59,14 @@ The **Child** role also requires a standalone OpenCodex runtime running on its c
 
 - **Connected** means the SSH tunnel is ready and the Child can use the Home link.
 - **Reconnecting** means the tunnel is being retried. Requests can temporarily return `503` with `Retry-After` while the retry is in progress. On a Child that connected from its own dashboard, a request first waits up to 15 seconds for the tunnel to come back.
-- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. The side that runs the SSH tunnel keeps retrying by itself, after sleep, an outage or a restart: Home for a Child it added, and the Child for a link it joined from its own dashboard. It retries about once a minute after a timeout or forwarding error, and every five minutes after an authentication error. A changed host key is never retried. To try again at once, select **Retry** in the Home's dashboard or run `ocx link reconnect --link-id <id>` on the Home. It restarts that link's tunnel without restarting OpenCodex.
+- **Failed** means the link needs attention. Check SSH authentication, the confirmed host key, forwarding, or the timeout reason shown in the dashboard. The side that runs the SSH tunnel keeps retrying by itself, after sleep, an outage or a restart: Home for a Child it added, and the Child for a link it joined from its own dashboard. After a timeout or forwarding error it retries after one minute and doubles the wait after each retry that fails again, up to five minutes; after an authentication error it retries every five minutes. A changed host key is never retried. To try again at once, select **Retry** in the Home's dashboard or run `ocx link reconnect --link-id <id>` on the Home. It restarts that link's tunnel without restarting OpenCodex.
 
 A failed link does not silently switch to a local provider.
 The Child returns a retryable `503` without forwarding the link key or request when its tunnel is failed, stopped, or not supervised.
 It also keeps the key local when another process owns the tunnel port; free that port so the Child's SSH tunnel can bind it.
 
+On a Child, `GET /healthz` only says that OpenCodex is running. `GET /readyz` (and `ocx ready`) also says whether the link can carry a request right now: while it cannot, it answers `503` with `Retry-After: 1`, `"status": "pending"` and a `link` object such as `{ "state": "unavailable", "reason": "home_unreachable" }`, and while it can, `200` with `"status": "ready"` and `{ "state": "available" }`. The reasons are `tunnel_connecting`, `tunnel_failed`, `tunnel_idle`, `tunnel_unavailable`, `sidecar_invalid`, `home_unreachable`, `home_unauthorized`, `link_auth_failed`, `link_credential_unavailable` and `link_unverified`. A Child that joined from its own dashboard answers from its tunnel. A Child that its Home added has no tunnel of its own to look at, so it asks the Home through the tunnel with the same credential-free check every relayed request starts with; it repeats that check at most once every five seconds, and answers `link_unverified` when the Home has not replied within half a second.
+
 To confirm which Home is serving a Child, compare the `x-opencodex-served-by` header on the Child's `GET /v1/models` with the `instanceId` in the Home's `GET /healthz`. The Child passes the Home's header through unchanged, so the two match while the link works. The value is an opaque id, never a hostname; see [Home identity and the served-by header](/reference/management-api/#home-identity-and-the-served-by-header).
 
 ## What a Child shows and manages
@@ -98,6 +100,19 @@ When a step fails, the dashboard shows the reason and, when SSH reported one, a
 - **OpenCodex on the remote computer is too old**: run `ocx update` on that computer. Remote Link needs 2.66.0 or later.
 - **The remote computer did not report an OpenCodex version**: `ocx --version` on that computer printed something else, for example the usage text of an unsupported Windows install.
 - **Requests fail with `home_unreachable` while the Home is down**: a Child never serves requests from its own providers. When the tunnel is down before a request is forwarded, the Child answers a retryable 503 and sends nothing to the Home. If the tunnel drops after forwarding started, the answer is also `home_unreachable`, but the Home may already have received the request. A failed link relay authentication answers `link_relay_auth_failed` instead: upgrade the Home and Child or re-link. Whether GOAT may move that device's work to its own sign-ins is the per-device local fallback permission on the Home's **Remote Link** page, or `ocx fallback local on --device <alias>`; see [Fallback off](/guides/fallback-off/).
+- **A Home link stays Failed with a forwarding error after the Home slept or changed networks**: run `ocx link status --json` on the Home. Its `tunnel` field shows the attempts since the link was last connected, `nextRetryAt`, and the last failure. A `lastFailure.cause` of `remote_forward_conflict` means the Child's SSH server still holds the tunnel port for the session that died with the Home's old connection, so every new tunnel is refused until it lets go. OpenCodex keeps retrying and connects on its own once the port is free; it never stops processes on the Child. The Child's SSH server notices a dead session only when it probes it. Without `ClientAliveInterval` in the Child's `sshd_config`, that can take as long as the operating system's TCP keepalive (two hours by default on macOS). Setting `ClientAliveInterval 15` and `ClientAliveCountMax 3` there makes it drop such a session within about a minute.
+
+### Retryable 503 diagnostics
+
+A relay-generated `503` includes a server-generated `x-opencodex-link-request-id`. Use it to correlate a failure with local evidence under the Child's OpenCodex configuration home: `link/diagnostics.jsonl` and one rotated backup, `link/diagnostics.jsonl.1`. Each file is capped at 256 KiB, with private permissions (directory 0700 and files 0600 on POSIX; private ACLs on Windows). Records contain only fixed event/outcome/state categories, generated correlation IDs, numeric timing/generation/attempt metadata and timestamps, never secrets, request bodies, headers, URLs, account identifiers or raw errors. Each event/outcome pair is limited to one record per second; the next record includes a count of suppressed events, so an absent record does not prove an absent failure. A backward wall-clock adjustment starts a new rate-limit window, including when the writer queue is full, and preserves the suppression count. An asynchronous writer keeps at most 56 pending records, counts overflow as suppression, and yields while Windows ACL work runs. Logging is best effort and cannot change whether a request is admitted. Graceful signal shutdown and standalone recycling wait up to one second for queued diagnostics before exiting; stalled storage, forced termination and power loss can still lose records.
+
+`owner_lookup` records refused ownership checks; routine successful checks are silent. `owner_lookup.elapsedMs` measures the owner lookup itself. For `auth_timeout`, `relay_retry` and `relay_failure` report the actual transport-attempt duration in `elapsedMs` (normally about five seconds). A `relay_recovered` event with outcome `recovered` is emitted only after a retry receives an upstream response with `upstream.ok` (HTTP 2xx); it does not prove the streamed body completed. Rate limiting makes request-ID correlation sampled evidence, not a guaranteed trace of every failed request.
+
+The Home writes the same kind of file under its own configuration home for the tunnels it runs. Each change of a Home-initiated tunnel's state is a `tunnel_transition` record with `previousState`, `state`, the cause as `outcome` (`auth`, `hostkey`, `forward`, `remote_forward_conflict`, `timeout`, `network`, `unknown`, `spawn_failed` or `listener_unavailable`; `connected`, `spawned` or `stopped` when nothing failed), the `attempt` count since the link was last connected, the link's `tunnelPort` and, when a retry is scheduled, `retryAfterMs`. A retry of a failed link that fails again is recorded too, so an outage leaves one record per attempt. The same rate limit applies, across all links.
+
+Before sending data, OpenCodex allows one shared preflight retry for either an owner-lookup deadline or a specifically identified five-second authentication deadline. The retry obtains fresh owner proof where required and a fresh physical socket, then authenticates that socket before sending the link credential or body. It does not reuse cached owner approval, bypass security checks, restart SSH or OpenCodex, or replay a dispatched request body. The two-second owner-lookup and five-second authentication deadlines are unchanged for each attempt; other failures do not qualify for this preflight retry.
+
+This can recover these transient deadline paths and helps diagnose future failures. It does not establish the historical cause of [#7297](https://github.com/lidge-jun/opencodex/issues/7297).
 
 ## Security
 

docs-site/src/content/docs/ja/guides/remote-link.md

@@ -55,7 +55,7 @@ Home のプロバイダーを使うコンピューターで次の操作を行い
 
 - **Connected** は SSH トンネルが準備でき、Child が Home のリンクを使える状態です。
 - **Reconnecting** はトンネルを再試行している状態です。再試行中はリクエストが `Retry-After` 付きの `503` を一時的に返すことがあります。自分のダッシュボードから接続した Child では、リクエストはまずトンネルの復帰を最大 15 秒待ちます。
-- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。SSH トンネルを張る側が、スリープ、障害、再起動の後も自動で再試行します。Home が追加した Child では Home が、自分のダッシュボードから接続した Child では Child が再試行します。タイムアウトや転送エラーの後は約 1 分ごと、認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。すぐに再接続するには、Home のダッシュボードで **Retry** を選ぶか、Home で `ocx link reconnect --link-id <id>` を実行します。OpenCodex を再起動せずに、そのリンクのトンネルだけを張り直します。
+- **Failed** は対応が必要な状態です。SSH 認証、確認済みのホストキー、転送、タイムアウトの理由を確認してください。SSH トンネルを張る側が、スリープ、障害、再起動の後も自動で再試行します。Home が追加した Child では Home が、自分のダッシュボードから接続した Child では Child が再試行します。タイムアウトや転送エラーの後は 1 分後に再試行し、失敗が続くたびに間隔を倍にして最大 5 分ごとになります。認証エラーの後は 5 分ごとです。ホストキーが変わった場合は再試行しません。すぐに再接続するには、Home のダッシュボードで **Retry** を選ぶか、Home で `ocx link reconnect --link-id <id>` を実行します。OpenCodex を再起動せずに、そのリンクのトンネルだけを張り直します。
 
 リンクが失敗しても、ローカルプロバイダーへ自動的に切り替わることはありません。
 

docs-site/src/content/docs/ko/guides/remote-link.md

@@ -55,7 +55,7 @@ Home의 프로바이더를 사용할 컴퓨터에서 다음을 진행합니다.
 
 - **Connected**는 SSH 터널이 준비되어 Child가 Home 링크를 사용할 수 있다는 뜻입니다.
 - **Reconnecting**은 터널을 다시 연결하는 중이라는 뜻입니다. 재시도 중에는 요청이 일시적으로 `Retry-After`와 함께 `503`을 반환할 수 있습니다. 자기 대시보드에서 연결한 Child에서는 요청이 먼저 터널이 돌아오기를 최대 15초 기다립니다.
-- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. SSH 터널을 띄운 쪽이 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. Home이 추가한 Child라면 Home이, 자기 대시보드에서 연결한 Child라면 Child가 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 약 1분마다, 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. 바로 다시 연결하려면 Home 대시보드에서 **Retry**를 누르거나 Home에서 `ocx link reconnect --link-id <id>`를 실행하세요. OpenCodex를 재시작하지 않고 그 링크의 터널만 다시 띄웁니다.
+- **Failed**는 조치가 필요하다는 뜻입니다. SSH 인증, 확인한 호스트 키, 포워딩 또는 타임아웃 사유를 확인하세요. SSH 터널을 띄운 쪽이 절전, 장애, 재시작 뒤에도 스스로 다시 시도합니다. Home이 추가한 Child라면 Home이, 자기 대시보드에서 연결한 Child라면 Child가 시도합니다. 타임아웃이나 포워딩 오류 뒤에는 1분 뒤에 다시 시도하고, 다시 실패할 때마다 간격을 두 배로 늘려 최대 5분마다 시도합니다. 인증 오류 뒤에는 5분마다 시도합니다. 호스트 키가 바뀐 경우에는 다시 시도하지 않습니다. 바로 다시 연결하려면 Home 대시보드에서 **Retry**를 누르거나 Home에서 `ocx link reconnect --link-id <id>`를 실행하세요. OpenCodex를 재시작하지 않고 그 링크의 터널만 다시 띄웁니다.
 
 링크가 실패해도 로컬 프로바이더로 조용히 전환하지 않습니다.
 

docs-site/src/content/docs/reference/cli/lifecycle.md

@@ -426,6 +426,12 @@ The CLI's own `--json` output is deliberately narrower than the HTTP body: it em
 `unreachable`. Exit codes are 0 for ready; 1 for not-ready, pending, failed, timeout, or
 unreachable; and 64 for invalid arguments.
 
+On a Remote Link Child, readiness also covers the link. While the link cannot carry a request,
+`/readyz` answers `503` with `status: "pending"` and `link: {state: "unavailable", reason}`, so
+`ocx ready --wait` keeps waiting until the link is back; a working link adds
+`link: {state: "available"}` to the `200` answer. `/healthz` on a Child keeps meaning only that the
+process runs. The reasons are listed in [Remote Link](/guides/remote-link/#link-status).
+
 ### `ocx resolve [--json]`
 
 Resolve the runtime facts a shell needs without re-implementing them: the config home, the
@@ -1153,6 +1159,14 @@ starts and then once a minute; the Home keeps reports in memory only, so after a
 fields read `null` until the next report. A Child whose runtime started before `lastApplyAt` is
 running an older runtime than its last apply.
 
+For a link the Home added, each row also carries `tunnel`: `attempt` (ssh attempts since the link
+was last connected), `nextRetryAt` (ISO time of the next scheduled retry, `null` while one runs or
+none is due), `lastTransition` (`{from, to, at, cause}`) and `lastFailure` (`{cause, at}`). A cause
+is one of `auth`, `hostkey`, `forward`, `remote_forward_conflict`, `timeout`, `network`,
+`unknown`, `spawn_failed` and `listener_unavailable`. The Home keeps it in memory, so it starts
+empty after a Home restart. `GET /api/link/status` includes the field only when asked with
+`?diagnostics=1`, because an older `ocx link status` rejects fields it does not know.
+
 After `ocx sync` applies the Home's settings on a Child, it restarts the Child's client runtime when
 that runtime started before `lastApplyAt`, using the same restart as `ocx restart`. A runtime that
 already started after the apply is left alone, nothing is started when no client runtime is