Fleet-wide live monitor with streaming, filterable logs

.claude/skills/shipnode/SKILL.md

@@ -85,10 +85,11 @@ Target one app: `shipnode deploy --app api`, `shipnode logs --app web`. `rollbac
 ## Workflows
 
 ### First deploy
-1. `shipnode init`
-2. `shipnode setup` — creates `deploy` user by default (`--no-deploy-user` to skip)
-3. `shipnode env` (per app if multi-app: `--app api`)
-4. `shipnode deploy`
+1. `shipnode init` — asks only for the server IP and an optional domain; `--host <ip> --domain <d> --non-interactive` skips prompts
+2. `shipnode setup` — creates the `deploy` user (`--no-deploy-user` to skip). When the config says `user: 'deploy'` and that user does not exist yet, it logs in as `root` for this run
+3. `shipnode deploy` — the first deploy uploads the local `envFile` when the server has none (outside CI). With no local `.env`, it starts from an empty one. Use `shipnode env` to push later changes
+
+The health check passes on any response below 500 until `.healthCheck(path)` is set; a configured path must return 2xx/3xx.
 
 ### CI/CD
 
@@ -170,15 +171,18 @@ shipnode cloudflare init   # one tunnel, ingress per app domain
 
 ### Monitor
 ```bash
-shipnode monitor           # live TUI: PM2, system, health, logs
-shipnode monitor --app api
+shipnode monitor           # live TUI across every server: fleet overview, replica detail, streaming logs
+shipnode monitor --app api # one app on all its servers
+shipnode monitor --on web-2
+shipnode monitor --once    # one snapshot; --json for machine-readable
 ```
 
 ## Day-to-day
 
 ```bash
 shipnode status [--app name]
 shipnode logs [--app name] [--lines 500]
+shipnode logs --follow [--level error] [--grep '/timeout/i'] [--on server]   # live, merged across servers
 shipnode restart [--app name]
 shipnode stop [--app name]
 shipnode run "pnpm db:apply" [--app name]

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to `@devalade/shipnode` will be documented here.
 
 ## [Unreleased]
 
+### Added
+- **`monitor` watches the whole fleet.** With no `--on` it connects to every server and opens a fleet overview: each app on each server it runs on, the release each replica serves, process and health state, and a one-line verdict per app (a half-finished roll reads `split across 2 releases` and marks the replica that is `behind`; an unreachable server keeps its row). `Enter` opens the per-replica view the old single-server monitor showed. `--on` and `--app` still narrow it. Rollback from the monitor is refused for an app that runs on several servers, since it would split the fleet; use `shipnode rollback`.
+- **Live log streaming with filters.** `f` opens a merged, streaming view of every server's logs (PM2, systemd and Caddy access logs) over the existing SSH connections, replacing the 2-second `pm2 logs --nostream` poll and its duplicated or dropped lines. Filter by server (`s`), app (`a`), process (`p`), level (`v`: all, warn+, error) and text, `/regex/` or `!exclude` search (`/`), in hide or dim mode (`m`), with live `ERR`/`WARN` counts. Pause, scroll back, and clear are supported. A dropped connection is retried with backoff without replaying lines already shown.
+- **The monitor has a consistent visual design.** Panels carry their title in the border, data sits in aligned tables with column headings, and colour is reserved for state (green ok, amber degraded, red failing) with one accent for titles and selection. A breadcrumb header shows where you are and how fresh the data is (`live`, `stale` once polls stop arriving), alerts get their own line, and the footer lists the keys for the current view and briefly confirms actions. Tables drop their least important columns on narrow terminals instead of overflowing, and nothing paints a background, so light terminal themes work.
+- **`shipnode logs --follow`** streams the same merged logs to the terminal, with `--level`, `--grep`, `--app`, `--on` and `--process`. `--level` and `--grep` also filter the one-shot `logs` output.
+
+### Changed
+- **`init` asks at most six questions instead of up to fifteen:** what you're deploying (pre-selected from detection), server IP, port, domain, and for backends whether to install a database or Redis. SSH user and port, deploy path, PM2 name, runtime, health check path and extra users take defaults you can edit in the file. `--host` and `--domain` let `init --non-interactive` write a complete config. A DB password is no longer written into the config as a literal; it reads `process.env.DB_PASSWORD`. If `shipnode.config.ts` already exists, `init` says so before asking anything.
+- **The default health check accepts any HTTP answer below 500.** Apps without a `/health` route no longer fail their first deploy on a 404. A path set with `.healthCheck(path)` (or `healthCheck.path`) is still held to 2xx/3xx; `healthCheck.strict` overrides either way. Failures now say whether nothing answered on the port or the configured route returned 404.
+
+### Fixed
+- **`setup` could not reach a fresh server with the config `init` wrote.** `init` sets `user: 'deploy'`, which only exists after `setup` creates it. When the server refuses `deploy`, `setup` now logs in as `root` for that run. If the config uses another user, the "switch ssh.user" hint is shown as before.
+- **The first deploy failed with "Remote environment file is missing".** When the server has no env file, `deploy` now uploads the local one (what `shipnode env` would do). With no local `.env` either, it creates an empty one. A custom `envFile` name that is missing locally still fails. None of this happens in CI, where the env belongs to `ci env-sync` (ADR-0006).
+
 ## [3.2.0-beta.3] - 2026-10-01
 
 ### Changed

README.md

@@ -11,22 +11,17 @@ npm install -g @devalade/shipnode
 ## Quick start
 
 ```bash
-# 1. Generate config
-shipnode init
-
-# 2. Provision the server. Installs Node, PM2, Caddy, mise, DB/Redis if
-#    configured, and bootstraps a `deploy` user (sudo, NOPASSWD) keyed off
-#    ${ssh.identityFile}.pub so subsequent runs don't need root SSH.
-shipnode setup
-
-# 3. Switch ssh.user in shipnode.config.ts to 'deploy', then lock down SSH:
-shipnode harden
-
-# 4. Deploy
-shipnode deploy
+shipnode init     # asks for your server's IP (and a domain, if you have one)
+shipnode setup    # once per server: installs Node, PM2, Caddy, creates a 'deploy' user
+shipnode deploy   # every time you ship
 ```
 
-Pass `--no-deploy-user` to `setup` if you want to manage users yourself.
+That's it. All you need is a fresh Ubuntu/Debian VPS you can `ssh root@<ip>` into with your key.
+
+- `setup` logs in as `root` the first time, because the `deploy` user it creates doesn't exist yet. After that, everything runs as `deploy`.
+- On the first deploy, your local `.env` is uploaded to the server. After you change it, run `shipnode env` to push it again.
+- The health check passes once your app answers HTTP on its port. Add `.healthCheck('/health')` to require a real 2xx from a specific route.
+- `shipnode harden` (optional) disables root and password SSH and turns on a firewall.
 
 ## Configuration
 
@@ -329,9 +324,9 @@ See [ADR-0007](docs/adr/0007-fleet-replication.md) and [ADR-0008](docs/adr/0008-
 | `.zeroDowntime(altPort?)` | automatic for Caddy backends | Force blue-green releases and optionally choose the green port |
 | `.blueGreenRetention('warm' \| 'rollback' \| 'none')` | `'warm'` | What happens to the old colour after the switch: stop it but keep its release so `rollback` can start it again (`warm`), keep it running for an instant flip (`rollback`), or stop it with no rollback (`none`) |
 | `.noZeroDowntime()` | — | Opt out and recreate PM2 processes during deploy |
-| `.healthCheck(path, opts?)` | `/health`, 30s, 3 retries | Post-deploy health check |
+| `.healthCheck(path, opts?)` | `/health`, 30s, 3 retries; any response below 500 passes until you set a path | Post-deploy health check — a configured path must return 2xx/3xx |
 | `.noHealthCheck()` | — | Skip health check |
-| `.envFile(f)` | `.env` | Local .env file to upload |
+| `.envFile(f)` | `.env` | Local .env file to upload (automatically on the first deploy) |
 | `.sharedDirs(dirs)` | — | Dirs persisted across releases |
 | `.sharedFiles(files)` | — | Files persisted across releases |
 | `.database(opts)` | — | Database connection config |

docs/adr/0006-ci-environment-ownership.md

@@ -24,6 +24,10 @@ Generated workflows do not run a generic repository-root build. Frontend strateg
 
 Production jobs use GitHub Environment protection, least-privilege repository permissions, a timeout, and serialized concurrency with `cancel-in-progress: false`. Root-level monorepo changes continue to trigger deployments; no `paths` filter is generated.
 
+## First deploy from a developer machine
+
+When the server has no env file, `deploy` uploads the local one, which is what `shipnode env` would have done. With no local file and the default `.env` name, it creates an empty one. This never happens when `CI` is set: a missing file there means a misconfigured workflow, and the preflight still fails.
+
 ## Trade-offs
 
 - Server-managed env minimizes secret exposure but requires an out-of-band env upload or rotation step.

src/cli/commands/config.ts

@@ -46,7 +46,7 @@ function showApp(app: ShipnodeApp, nodeVersion: string): void {
   if (app.appType === 'backend') {
     ui.section('Health Check', [
       ['enabled', String(app.healthCheck.enabled)],
-      ['path', app.healthCheck.path],
+      ['path', app.healthCheck.strict === false ? `${app.healthCheck.path} (any response below 500)` : app.healthCheck.path],
       ['timeout', String(app.healthCheck.timeout)],
       ['retries', String(app.healthCheck.retries)],
       ['startupDelay', String(app.healthCheck.startupDelay)],

src/cli/commands/deploy.ts

@@ -390,7 +390,9 @@ function renderAppPlan(
   if (app.hooks?.preDeploy) steps.push('Run preDeploy hook');
   steps.push('Switch symlink (atomic)');
   if (app.appType === 'backend') steps.push('Reload PM2');
-  if (app.healthCheck.enabled) steps.push(`Health check ${app.healthCheck.path}`);
+  if (app.healthCheck.enabled) {
+    steps.push(`Health check ${app.healthCheck.path}${app.healthCheck.strict === false ? ' (any response below 500)' : ''}`);
+  }
   steps.push('Record release');
   if (app.hooks?.postDeploy) steps.push('Run postDeploy hook');
   if (app.hooks?.afterFleet) steps.push('Run afterFleet hook (last replica only)');

src/cli/commands/env.ts

@@ -5,30 +5,7 @@ import { runRemoteCommandForTargets } from '../runner.js';
 import { ui } from '../ui.js';
 import { getDeploymentName } from '../../domain/pm2/apps.js';
 import { isWatt, resolveWattUnits, restartUnitCommand } from '../../domain/runtime/watt.js';
-import type { RemoteExecutor } from '../../domain/remote/executor.js';
-
-function shellSingleQuote(value: string): string {
-  return `'${value.replace(/'/g, `'"'"'`)}'`;
-}
-
-/** Atomically replace a remote environment file without exposing its raw content to shell parsing. */
-export async function uploadEnvironmentFile(
-  executor: RemoteExecutor,
-  remotePath: string,
-  content: Buffer,
-): Promise<void> {
-  const b64 = content.toString('base64');
-  await executor.execOrThrow(`mkdir -p "$(dirname ${shellSingleQuote(remotePath)})"`);
-  const temporaryEnv = `${remotePath}.shipnode.XXXXXX`;
-  await executor.execOrThrow([
-    `tmp=$(mktemp ${shellSingleQuote(temporaryEnv)})`,
-    `trap 'rm -f "$tmp"' EXIT`,
-    `printf '%s' ${shellSingleQuote(b64)} | base64 -d > "$tmp"`,
-    'chmod 600 "$tmp"',
-    `mv -f "$tmp" ${shellSingleQuote(remotePath)}`,
-    'trap - EXIT',
-  ].join(' && '));
-}
+import { uploadEnvironmentFile } from '../../domain/deploy/dotenv.js';
 
 export async function cmdEnv(
   cwd: string,