-
Notifications
You must be signed in to change notification settings - Fork 45
docs: Document pod runtime status and specific SSH not-ready reasons in the runpodctl reference #758
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
docs: Document pod runtime status and specific SSH not-ready reasons in the runpodctl reference #758
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -51,7 +51,7 @@ Show all Pods including exited ones. By default, only running Pods are shown. | |
| </ResponseField> | ||
|
|
||
| <ResponseField name="--status" type="string"> | ||
| Filter by Pod status (e.g., `RUNNING`, `EXITED`). Cannot be used with `--all`. | ||
| Filter by desired status (`desiredStatus`), such as `RUNNING` or `EXITED`. Matching is case-insensitive. Does not accept `runtimeStatus` values like `initializing`; passing one returns no results. Cannot be used with `--all`. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="--since" type="string"> | ||
|
|
@@ -274,6 +274,79 @@ Delete a Pod: | |
| runpodctl pod delete <pod-id> | ||
| ``` | ||
|
|
||
| ## Pod runtime status | ||
|
|
||
| `runpodctl pod get` and `runpodctl pod list` now report a Pod's real runtime state in their JSON and YAML output, alongside the existing `desiredStatus`. `desiredStatus` is the state you asked for (for example, `RUNNING`), while `runtimeStatus` is what the Pod is actually doing. For example, a Pod still pulling a large image and a Pod that has been serving traffic for an hour both show `desiredStatus: RUNNING`, but their `runtimeStatus` values differ. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. desiredStatus vs runtimeStatus distinction (desiredStatus is what was requested, runtimeStatus is derived/actual) is documented verbatim in the podstate package comment, from open PR runpod/runpodctl#315 (CON-690, not yet merged). |
||
|
|
||
| <ResponseField name="runtimeStatus" type="string"> | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. podListOutput struct confirms runtimeStatus has no omitempty (always present); runtimeStatusReason and lastStatusChange both have omitempty. Same shape appears on pod get (cmd/pod/get.go response struct). From open PR runpodctl#315. |
||
| Always present. What the Pod is actually doing (see the table below). | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="runtimeStatusReason" type="string"> | ||
| Present only when there is more to say. A stable token you can safely branch on in scripts. | ||
| </ResponseField> | ||
|
|
||
| <ResponseField name="lastStatusChange" type="string"> | ||
| Present when available. The backend's raw status text (for example, `Exited by user: <date>` or `Outbid: <date>`). | ||
| </ResponseField> | ||
|
|
||
| For example, `runpodctl pod list --output json` returns entries like this for a running Pod: | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. JSON example field names (id, name, desiredStatus, runtimeStatus, imageName, gpuCount, costPerHr, createdAt, uptimeSeconds) all match real podListOutput fields populated in runList; no fabricated field names. --output flag consumed via cmd.Flag("output") (persistent flag, StringVarP "output"/"o", default "json"), so |
||
|
|
||
| ```json | ||
| { | ||
| "id": "abc123xyz", | ||
| "name": "my-training-pod", | ||
| "desiredStatus": "RUNNING", | ||
| "runtimeStatus": "running", | ||
| "imageName": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04", | ||
| "gpuCount": 1, | ||
| "costPerHr": 0.69, | ||
| "createdAt": "2025-01-15T09:30:00Z", | ||
| "uptimeSeconds": 3600 | ||
| } | ||
| ``` | ||
|
|
||
| When a Pod is initializing, `runtimeStatus` is `initializing`, `runtimeStatusReason` is `awaiting_container`, and `uptimeSeconds` is omitted because no container is reporting yet. | ||
|
|
||
| ### Runtime status values | ||
|
|
||
| | Value | Meaning | What to do | | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Runtime status values table (running/initializing/stopped/terminated/unknown) matches the Status constants and doc comments exactly, including "does not imply a port is reachable" and the terminated pod's narrow drop-out-of-list window (getMyPods filters terminatedAt: null, see lines 77-81 of same file). |
||
| | --- | --- | --- | | ||
| | `running` | desiredStatus is `RUNNING` and the platform is reporting runtime telemetry, so the container is up. Does not imply that any port (such as SSH) is reachable. | Ready to use. | | ||
| | `initializing` | desiredStatus is `RUNNING` but no telemetry is being reported yet. Usually the container is not up yet (image pull, container creation, or boot, which the platform does not distinguish). Read this as "no container reported," not "the container is provably down." | Keep polling. | | ||
| | `stopped` | desiredStatus is `EXITED` and the last status change does not name a termination. The container is gone, but the Pod's disk is kept. | Run `runpodctl pod start <pod-id>` to bring it back. | | ||
| | `terminated` | The Pod is being destroyed. A terminated Pod drops out of `runpodctl pod list` shortly after, so this is a narrow window. | None; the Pod is gone. Because a terminated Pod quickly drops out of `runpodctl pod list`, a teardown script should treat "Pod not found"—not `runtimeStatus: terminated`—as the reliable signal that a Pod is gone. | | ||
| | `unknown` | The runtime state cannot be derived (an uncommon `desiredStatus`, or the runtime lookup failed). | Read `desiredStatus` from the same output. | | ||
|
|
||
| ### Runtime status reasons | ||
|
|
||
| | Token | Paired status | Meaning | | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Runtime status reasons table matches the eight Reason constants and their doc comments verbatim (awaiting_container, stopped/terminated_by_user, stopped/terminated_by_runpod, stopped/terminated_outbid, runtime_unavailable). |
||
| | --- | --- | --- | | ||
| | `awaiting_container` | `initializing` | No container is being reported yet for a Pod that should be running. | | ||
| | `stopped_by_user`, `terminated_by_user` | `stopped` or `terminated` | You stopped or terminated the Pod. | | ||
| | `stopped_by_runpod`, `terminated_by_runpod` | `stopped` or `terminated` | Runpod stopped or terminated the Pod. No machine-readable cause is recorded; in practice this is a low account balance, a fatal image-pull failure, or host action. | | ||
| | `stopped_outbid`, `terminated_outbid` | `stopped` or `terminated` | A spot or Community Cloud Pod lost its machine to a higher bid. Retry on a different machine or at on-demand pricing. To avoid another outbid, redeploy with `runpodctl pod create` on Secure Cloud or an on-demand configuration. | | ||
| | `runtime_unavailable` | `unknown` | The runtime lookup could not be made, so `running` and `initializing` cannot be told apart. | | ||
|
|
||
| <Tip> | ||
| To poll a Pod to readiness in a script, wait for `runtimeStatus: running`. Branch on the `runtimeStatus` and `runtimeStatusReason` tokens rather than parsing the free-text `lastStatusChange`. | ||
|
|
||
| ```bash | ||
| # Wait until the Pod's container is up | ||
| until [ "$(runpodctl pod get <pod-id> --output json | jq -r '.runtimeStatus')" = "running" ]; do | ||
| sleep 5 | ||
| done | ||
| ``` | ||
|
|
||
| A `runtimeStatus` of `unknown` with reason `runtime_unavailable` during polling usually means the runtime lookup momentarily failed, not that the Pod is down—keep polling rather than treating it as terminal. | ||
| </Tip> | ||
|
|
||
| <Note> | ||
| `runtimeStatus` and `desiredStatus` come from different sources, so a single command can briefly show them disagreeing (for example, `desiredStatus: RUNNING` next to `runtimeStatus: stopped`). When they disagree, trust `runtimeStatus`. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirms runtimeStatus is derived from the GraphQL snapshot while desiredStatus is REST's, and that the two surfaces can momentarily disagree (matches PR README language "when they disagree, trust runtimeStatus"). |
||
| </Note> | ||
|
|
||
| The `uptimeSeconds` field now reports the container's actual uptime and is omitted entirely when no container is reporting, instead of always showing `0`. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. uptimeSeconds is set only when runtimeStatus is running and runtime telemetry exists; otherwise left nil, which the omitempty tag drops from output instead of always reporting 0. |
||
|
|
||
| ## Pod URLs | ||
|
|
||
| Access exposed ports on your Pod using the following URL pattern: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,7 +29,7 @@ Include Pod ID and name in output. Shorthand: `-v`. | |
|
|
||
| <Note> | ||
|
|
||
| The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session. | ||
| The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session. When a Pod is not reachable, `runpodctl ssh info` now reports the specific reason instead of a generic "pod not ready" message. | ||
|
|
||
| To connect to your Pod, use the SSH command provided in the output: | ||
| ```bash | ||
|
|
@@ -38,6 +38,25 @@ ssh user@host -p <port> -i <key-path> | |
|
|
||
| </Note> | ||
|
|
||
| `runpodctl ssh info` explains which of the following cases applies. It also reports the Pod's `runtimeStatus`, so you can match a not-ready result to the runtime status vocabulary: | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirms runSSHInfoWithArgs includes a "runtimeStatus" key in the not-ready JSON/YAML output map alongside the error/id/name/status fields. |
||
|
|
||
| | Reason | Meaning | What to do | | ||
| | --- | --- | --- | | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not-ready reasons table (initializing/stopped/terminated via state.Explain; no port 22 published, no public IP, mapping not ready via sshPortDetail) matches NotReadyMessage/sshPortDetail exactly, including the three distinct running-pod SSH-unreachable causes. |
||
| | Pod is initializing | No container is reported yet (image pull, container creation, or boot). | Wait and try again. | | ||
| | Pod is stopped | The Pod's container is gone but its disk is kept. | Start it with `runpodctl pod start <pod-id>`. | | ||
| | Pod is terminated | The Pod has been destroyed. | None; deploy a new Pod. | | ||
| | No SSH port published | The Pod is running but does not publish `22/tcp`. | Add it with the `runpodctl pod update` command the CLI provides (see the Warning below). | | ||
| | No public IP | Port 22 is mapped, but the machine has no publicly routable IP. This affects direct SSH over a public IP specifically; basic SSH (proxied through Runpod, no public IP required) may still work. | Use [Basic SSH](/pods/configuration/use-ssh#basic-ssh-with-key-authentication), which does not need a public IP. If you need direct SSH over a public IP, redeploy on a machine that provides one. | | ||
| | Mapping not ready yet | Port 22 is declared, but the host has not published the mapping yet. | Wait and try again. | | ||
|
|
||
| For the "No SSH port published" case, `runpodctl ssh info` gives you a ready-to-run command of the form `runpodctl pod update <pod-id> --ports <existing-ports>,22/tcp` that preserves your existing ports. | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirms the exact remedy command form |
||
|
|
||
| <Warning> | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirms --ports is wholesale replacement (not additive like --env), sourced from cmd/pod/update.go setting req.Ports to exactly what was passed; changing ports bumps pod version (editJob increments version), which can trigger a container recreate/restart. |
||
| `runpodctl pod update --ports` replaces the Pod's entire port list (unlike `--env`, which merges), so always include your current ports when you add `22/tcp`. Changing the port list also bumps the Pod's version, which can restart the container. Processes and container-local state outside the volume disk may not survive. If the Pod is running an active job, checkpoint your work to the volume disk before changing ports, since the container may restart. | ||
| </Warning> | ||
|
|
||
| For the full list of runtime status values, see [Pod runtime status](/runpodctl/reference/runpodctl-pod#pod-runtime-status). | ||
|
|
||
| ### List SSH keys | ||
|
|
||
| List all SSH keys associated with your account: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Line 77)
Confirms Basic SSH is described elsewhere in this same file as proxied through Runpod's systems, supporting the runpodctl-ssh.mdx claim that basic SSH does not require a public IP.
Source:
docs/pods/configuration/use-ssh.mdx
Line 77 in eeff71f