Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pods/configuration/use-ssh.mdx

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Line 77)

Confirms Basic SSH is described elsewhere in this same file as proxied through Runpod's systems, supporting the runpodctl-ssh.mdx claim that basic SSH does not require a public IP.

Source:

All Pods provide a basic SSH connection that is proxied through Runpod's systems. This method does not support commands like SCP (Secure Copy Protocol) or SFTP (SSH File Transfer Protocol).

Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,10 @@ Where:

If you're asked for a password when connecting to your Pod via SSH, this means something is not set up correctly. Runpod does not require a password for SSH connections, as authentication is handled entirely through your SSH key pair.

<Tip>
If your Pod is unreachable or stuck, run `runpodctl ssh info <pod-id>` to see the specific reason. See [Get SSH connection info](/runpodctl/reference/runpodctl-ssh#get-ssh-connection-info).
</Tip>

Here are some common reasons why this might happen:

- If you copy and paste the key *fingerprint* (which starts with `SHA256:`) into your Runpod user settings instead of the actual public key (the contents of your `id_ed25519.pub` file), authentication will fail.
Expand Down
75 changes: 74 additions & 1 deletion runpodctl/reference/runpodctl-pod.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Show all Pods including exited ones. By default, only running Pods are shown.
</ResponseField>

<ResponseField name="--status" type="string">
Filter by Pod status (e.g., `RUNNING`, `EXITED`). Cannot be used with `--all`.
Filter by desired status (`desiredStatus`), such as `RUNNING` or `EXITED`. Matching is case-insensitive. Does not accept `runtimeStatus` values like `initializing`; passing one returns no results. Cannot be used with `--all`.
</ResponseField>

<ResponseField name="--since" type="string">
Expand Down Expand Up @@ -274,6 +274,79 @@ Delete a Pod:
runpodctl pod delete <pod-id>
```

## Pod runtime status

`runpodctl pod get` and `runpodctl pod list` now report a Pod's real runtime state in their JSON and YAML output, alongside the existing `desiredStatus`. `desiredStatus` is the state you asked for (for example, `RUNNING`), while `runtimeStatus` is what the Pod is actually doing. For example, a Pod still pulling a large image and a Pod that has been serving traffic for an hour both show `desiredStatus: RUNNING`, but their `runtimeStatus` values differ.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

desiredStatus vs runtimeStatus distinction (desiredStatus is what was requested, runtimeStatus is derived/actual) is documented verbatim in the podstate package comment, from open PR runpod/runpodctl#315 (CON-690, not yet merged).

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/podstate/podstate.go#L1-L27


<ResponseField name="runtimeStatus" type="string">

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

podListOutput struct confirms runtimeStatus has no omitempty (always present); runtimeStatusReason and lastStatusChange both have omitempty. Same shape appears on pod get (cmd/pod/get.go response struct). From open PR runpodctl#315.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/cmd/pod/list.go#L34-L53

Always present. What the Pod is actually doing (see the table below).
</ResponseField>

<ResponseField name="runtimeStatusReason" type="string">
Present only when there is more to say. A stable token you can safely branch on in scripts.
</ResponseField>

<ResponseField name="lastStatusChange" type="string">
Present when available. The backend's raw status text (for example, `Exited by user: <date>` or `Outbid: <date>`).
</ResponseField>

For example, `runpodctl pod list --output json` returns entries like this for a running Pod:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JSON example field names (id, name, desiredStatus, runtimeStatus, imageName, gpuCount, costPerHr, createdAt, uptimeSeconds) all match real podListOutput fields populated in runList; no fabricated field names. --output flag consumed via cmd.Flag("output") (persistent flag, StringVarP "output"/"o", default "json"), so --output json is a valid invocation form.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/cmd/pod/list.go#L182-L200


```json
{
"id": "abc123xyz",
"name": "my-training-pod",
"desiredStatus": "RUNNING",
"runtimeStatus": "running",
"imageName": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04",
"gpuCount": 1,
"costPerHr": 0.69,
"createdAt": "2025-01-15T09:30:00Z",
"uptimeSeconds": 3600
}
```

When a Pod is initializing, `runtimeStatus` is `initializing`, `runtimeStatusReason` is `awaiting_container`, and `uptimeSeconds` is omitted because no container is reporting yet.

### Runtime status values

| Value | Meaning | What to do |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runtime status values table (running/initializing/stopped/terminated/unknown) matches the Status constants and doc comments exactly, including "does not imply a port is reachable" and the terminated pod's narrow drop-out-of-list window (getMyPods filters terminatedAt: null, see lines 77-81 of same file).

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/podstate/podstate.go#L90-L120

| --- | --- | --- |
| `running` | desiredStatus is `RUNNING` and the platform is reporting runtime telemetry, so the container is up. Does not imply that any port (such as SSH) is reachable. | Ready to use. |
| `initializing` | desiredStatus is `RUNNING` but no telemetry is being reported yet. Usually the container is not up yet (image pull, container creation, or boot, which the platform does not distinguish). Read this as "no container reported," not "the container is provably down." | Keep polling. |
| `stopped` | desiredStatus is `EXITED` and the last status change does not name a termination. The container is gone, but the Pod's disk is kept. | Run `runpodctl pod start <pod-id>` to bring it back. |
| `terminated` | The Pod is being destroyed. A terminated Pod drops out of `runpodctl pod list` shortly after, so this is a narrow window. | None; the Pod is gone. Because a terminated Pod quickly drops out of `runpodctl pod list`, a teardown script should treat "Pod not found"—not `runtimeStatus: terminated`—as the reliable signal that a Pod is gone. |
| `unknown` | The runtime state cannot be derived (an uncommon `desiredStatus`, or the runtime lookup failed). | Read `desiredStatus` from the same output. |

### Runtime status reasons

| Token | Paired status | Meaning |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runtime status reasons table matches the eight Reason constants and their doc comments verbatim (awaiting_container, stopped/terminated_by_user, stopped/terminated_by_runpod, stopped/terminated_outbid, runtime_unavailable).

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/podstate/podstate.go#L127-L165

| --- | --- | --- |
| `awaiting_container` | `initializing` | No container is being reported yet for a Pod that should be running. |
| `stopped_by_user`, `terminated_by_user` | `stopped` or `terminated` | You stopped or terminated the Pod. |
| `stopped_by_runpod`, `terminated_by_runpod` | `stopped` or `terminated` | Runpod stopped or terminated the Pod. No machine-readable cause is recorded; in practice this is a low account balance, a fatal image-pull failure, or host action. |
| `stopped_outbid`, `terminated_outbid` | `stopped` or `terminated` | A spot or Community Cloud Pod lost its machine to a higher bid. Retry on a different machine or at on-demand pricing. To avoid another outbid, redeploy with `runpodctl pod create` on Secure Cloud or an on-demand configuration. |
| `runtime_unavailable` | `unknown` | The runtime lookup could not be made, so `running` and `initializing` cannot be told apart. |

<Tip>
To poll a Pod to readiness in a script, wait for `runtimeStatus: running`. Branch on the `runtimeStatus` and `runtimeStatusReason` tokens rather than parsing the free-text `lastStatusChange`.

```bash
# Wait until the Pod's container is up
until [ "$(runpodctl pod get <pod-id> --output json | jq -r '.runtimeStatus')" = "running" ]; do
sleep 5
done
```

A `runtimeStatus` of `unknown` with reason `runtime_unavailable` during polling usually means the runtime lookup momentarily failed, not that the Pod is down—keep polling rather than treating it as terminal.
</Tip>

<Note>
`runtimeStatus` and `desiredStatus` come from different sources, so a single command can briefly show them disagreeing (for example, `desiredStatus: RUNNING` next to `runtimeStatus: stopped`). When they disagree, trust `runtimeStatus`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirms runtimeStatus is derived from the GraphQL snapshot while desiredStatus is REST's, and that the two surfaces can momentarily disagree (matches PR README language "when they disagree, trust runtimeStatus").

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/cmd/pod/list.go#L153-L180

</Note>

The `uptimeSeconds` field now reports the container's actual uptime and is omitted entirely when no container is reporting, instead of always showing `0`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uptimeSeconds is set only when runtimeStatus is running and runtime telemetry exists; otherwise left nil, which the omitempty tag drops from output instead of always reporting 0.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/cmd/pod/list.go#L176-L180


## Pod URLs

Access exposed ports on your Pod using the following URL pattern:
Expand Down
21 changes: 20 additions & 1 deletion runpodctl/reference/runpodctl-ssh.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Include Pod ID and name in output. Shorthand: `-v`.

<Note>

The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session.
The `ssh info` command returns connection details that you can use to connect via SSH manually. It does not start an interactive SSH session. When a Pod is not reachable, `runpodctl ssh info` now reports the specific reason instead of a generic "pod not ready" message.

To connect to your Pod, use the SSH command provided in the output:
```bash
Expand All @@ -38,6 +38,25 @@ ssh user@host -p <port> -i <key-path>

</Note>

`runpodctl ssh info` explains which of the following cases applies. It also reports the Pod's `runtimeStatus`, so you can match a not-ready result to the runtime status vocabulary:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirms runSSHInfoWithArgs includes a "runtimeStatus" key in the not-ready JSON/YAML output map alongside the error/id/name/status fields.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/cmd/ssh.go#L191-L211


| Reason | Meaning | What to do |
| --- | --- | --- |

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not-ready reasons table (initializing/stopped/terminated via state.Explain; no port 22 published, no public IP, mapping not ready via sshPortDetail) matches NotReadyMessage/sshPortDetail exactly, including the three distinct running-pod SSH-unreachable causes.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/sshconnect/notready.go#L10-L50

| Pod is initializing | No container is reported yet (image pull, container creation, or boot). | Wait and try again. |
| Pod is stopped | The Pod's container is gone but its disk is kept. | Start it with `runpodctl pod start <pod-id>`. |
| Pod is terminated | The Pod has been destroyed. | None; deploy a new Pod. |
| No SSH port published | The Pod is running but does not publish `22/tcp`. | Add it with the `runpodctl pod update` command the CLI provides (see the Warning below). |
| No public IP | Port 22 is mapped, but the machine has no publicly routable IP. This affects direct SSH over a public IP specifically; basic SSH (proxied through Runpod, no public IP required) may still work. | Use [Basic SSH](/pods/configuration/use-ssh#basic-ssh-with-key-authentication), which does not need a public IP. If you need direct SSH over a public IP, redeploy on a machine that provides one. |
| Mapping not ready yet | Port 22 is declared, but the host has not published the mapping yet. | Wait and try again. |

For the "No SSH port published" case, `runpodctl ssh info` gives you a ready-to-run command of the form `runpodctl pod update <pod-id> --ports <existing-ports>,22/tcp` that preserves your existing ports.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirms the exact remedy command form runpodctl pod update <pod-id> --ports <existing-ports>,22/tcp built by addSSHPortCommand, preserving existing declared ports before appending 22/tcp.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/sshconnect/notready.go#L52-L82


<Warning>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirms --ports is wholesale replacement (not additive like --env), sourced from cmd/pod/update.go setting req.Ports to exactly what was passed; changing ports bumps pod version (editJob increments version), which can trigger a container recreate/restart.

Source: https://github.com/runpod/runpodctl/blob/ea017f65f5c6bc1bcaad506553ff4e08825a0356/internal/sshconnect/notready.go#L52-L68

`runpodctl pod update --ports` replaces the Pod's entire port list (unlike `--env`, which merges), so always include your current ports when you add `22/tcp`. Changing the port list also bumps the Pod's version, which can restart the container. Processes and container-local state outside the volume disk may not survive. If the Pod is running an active job, checkpoint your work to the volume disk before changing ports, since the container may restart.
</Warning>

For the full list of runtime status values, see [Pod runtime status](/runpodctl/reference/runpodctl-pod#pod-runtime-status).

### List SSH keys

List all SSH keys associated with your account:
Expand Down
Loading