diff --git a/CHANGELOG.md b/CHANGELOG.md index 2497e61..72b2fa2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to `@devalade/shipnode` will be documented here. +## [Unreleased] + +### Added +- **Opt-in `watt` runtime (wattpm).** `.runtime('watt', { main: 'dist/server.js' })` runs the web app as worker threads sharing one port via `SO_REUSEPORT` instead of PM2 processes — no supervisor in the request path and no per-process V8 duplication. `instances` becomes the thread count; workers run as systemd units (`shipnode-[-]`). Blue-green, `rollback`, `restart`, `stop`, `logs`, `env`, `deploy --watch`, `doctor`, `status`, `metrics` and the monitor all support it. PM2 stays the default. Your app must depend on `wattpm` and `@platformatic/node`; scaling past one worker needs Linux. See [ADR-0009](docs/adr/0009-watt-runtime.md). + ## [3.2.0-beta.1] - 2026-09-18 ### Added diff --git a/README.md b/README.md index a4c9e0e..ca3f938 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,25 @@ export default shipnode Each app gets its own release directory, Caddy site, PM2 ecosystem, and health check. Deploy everything with `shipnode deploy`, or target one app with `shipnode deploy --app api`. +### Opt-in: wattpm runtime + +Run the web app as worker threads sharing one port (`SO_REUSEPORT`) instead of PM2 processes — no supervisor in the request path and no per-process V8 duplication. PM2 remains the default. + +```ts +export default shipnode + .backend() + .ssh({ host: '1.2.3.4', user: 'deploy' }) + .deployTo('/var/www/myapp') + .pm2('myapp', { instances: 4, maxMemory: '512M' }) // instances = worker threads + .port(3000) + .domain('api.example.com') + .runtime('watt', { main: 'dist/server.js' }) // file each thread loads; must listen on process.env.PORT + .worker({ name: 'mailer', command: 'node dist/worker.js' }) // runs as its own systemd unit + .build(); +``` + +Shipnode installs `wattpm` and `@platformatic/node` for you if your app doesn't list them; add them to your own dependencies to pin the versions. Zero-downtime blue-green, `rollback`, `logs`, `stop` and `env` work as with PM2; supervision is systemd (`shipnode-[-]`). Scaling past one worker needs Linux. See `docs/adr/0009-watt-runtime.md` for the trade-offs. + ### Web + workers A backend can run additional long-running processes alongside the web server. PM2 supervises all of them under one deployment. diff --git a/docs/adr/0009-watt-runtime.md b/docs/adr/0009-watt-runtime.md new file mode 100644 index 0000000..10000b2 --- /dev/null +++ b/docs/adr/0009-watt-runtime.md @@ -0,0 +1,24 @@ +# Opt-in `watt` runtime (wattpm) alongside PM2 + +PM2 stays the default. `runtime: 'watt'` (builder: `.runtime('watt', { main })`) runs the web app under wattpm instead: N worker threads in one process, each accepting straight from the kernel via `SO_REUSEPORT`, so no supervisor process sits in the request path and V8's startup state is not duplicated per worker. + +## Mechanism + +- **Web app** → wattpm. `instances` becomes the worker-thread count. Shipnode renders two per-release files into the app root (ADR-0001): `shipnode.watt.json` (runtime config, `port: "{PORT}"`) and `shipnode.platformatic.json` (tells `@platformatic/node` which file to load — `watt.main`). The runtime file has its own name and is passed with `wattpm start -c`: an app at path `.` would otherwise pick up a same-directory `watt.json` as *its own* config and fail to start. +- **Workers** (declared processes without a port) → one plain systemd unit each. wattpm supervises HTTP threads, not arbitrary commands, so workers keep an OS process and get systemd's restart/logging. +- **Supervision** → systemd units `shipnode-[-][-]`, each running a per-release launcher script (`shipnode-run-*.sh`). A script file sidesteps systemd's `%`/`$` expansion and keeps dotenv handling identical to PM2 (ADR-0003): the env file is parsed as data, never sourced. +- **Dependencies** → if the release does not already have `wattpm` and `@platformatic/node`, shipnode installs both at the version its rendered configs target (`WATT_VERSION`, also the schema version) with the app's own package manager, after the normal install and relink so nothing prunes them. An app that lists them itself keeps its own versions and nothing is installed. Nothing is installed globally, so every release carries the runtime it was deployed with and a rollback restores the matching one. The auto-installed packages are not in the app's lockfile; pin them there for fully reproducible production installs. A custom `module` is never auto-installed. + +## Zero-downtime + +Blue-green (ADR-0005) works unchanged in shape: the idle colour is a separate unit (`shipnode-api-green`) on its own port, health-checked before Caddy flips. Workers are a single unit set restarted in `afterHealthy`. Rollback flips Caddy to the previous colour after checking its unit is `active`. Adopting watt on a host that ran PM2 retires the PM2 process after the first successful flip (`pm2 delete `, a no-op when PM2 was never there). + +## Trade-offs + +- **`SO_REUSEPORT` is Linux-only.** On other OSes wattpm forces a single worker for the entrypoint (observed on macOS). Production VPSes are Linux; local runs are not representative of scaling. +- **Thread isolation is weaker than processes.** A native crash or OOM takes down every worker in the process; `health.maxHeapUsed` (from `maxMemory`) recycles a bloated worker before that. +- **`shipnode restart` is an in-place systemd restart**, not PM2's rolling `reload`. Use `deploy` (blue-green) for a zero-drop roll. +- **Load spread is kernel-hashed**, so clients that share few source ports (e.g. a local proxy over loopback) can land unevenly. Validate with a real traffic split before rolling out widely. +- **Observation reads systemd, not PM2.** `status`, the monitor and `--json` sample `systemctl show` for each candidate unit. CPU is a rate but systemd only exposes a cumulative counter, so the observe script samples it twice ~200ms apart (adds ~0.2s per poll on hosts running watt apps). `metrics` shows a refreshing `systemctl status` since there is no `pm2 monit` equivalent. +- **Deploy health is systemd-aware.** A unit must be `active` with `NRestarts=0`; a crash loop under `Restart=always` would otherwise look healthy between crashes. Failures include the last journal lines. +- `harden`'s PM2 steps do not apply to watt apps: units are enabled at install time and start at boot without a saved process list. diff --git a/docs/superpowers/specs/2026-08-30-monitor-redesign-design.md b/docs/superpowers/specs/2026-08-30-monitor-redesign-design.md new file mode 100644 index 0000000..d4ff091 --- /dev/null +++ b/docs/superpowers/specs/2026-08-30-monitor-redesign-design.md @@ -0,0 +1,214 @@ +# Monitor redesign: a shared observation layer + +## Problem + +`shipnode monitor` is a single-server Ink TUI with its data layer welded to React. +Four things are wrong with it at once: + +1. **Fleet-blind.** `resolveMonitorSession` narrows to one replica because "the + monitor holds one live connection". The one failure mode v3 introduced — a + partly-rolled fleet — is therefore invisible in the live view and visible only + in `status`, a one-shot text command. +2. **Overlapping surfaces.** `monitor`, `status`, and `logs` each reach the server + their own way, so the same facts are gathered by three code paths. +3. **No machine-readable output.** No `--once`, no `--json`. Nothing to pipe into + a script, a CI gate, or an alert. +4. **Layout hand-tuned to a 24-row terminal.** `releaseBoxExtra = min(6, rows-30)`, + a fixed 60/40 split, fixed panel heights, and a 389-line `App.tsx` that owns + keybindings, overlay routing, action dispatch, and layout at once. + +## Scope + +In scope: a shared observation layer, fleet visibility, `--once`/`--json`, and +rebuilding `status` and `logs` on the shared layer (they may change shape). + +Out of scope for this spec: the TUI layout, the two views, and the keymap. Those +are designed separately once the foundation lands. + +`shipnode metrics` stays as it is. It hands the terminal to PM2's own `pm2 monit` +over `ssh -t`, which is a deliberate escape hatch to the unfiltered truth when +shipnode's view and reality disagree — `monitor` filters PM2 to the app's +namespace and structurally cannot show that. It gains `--on` (so a fleet app is +targeted deliberately rather than by whichever host `getServerTargetResult` +returns) and honest help text; nothing else. + +## Decision: collection is server-shaped, presentation pivots to app-shaped + +Three models were considered. + +- **App-shaped** — the subject is one app across its replicas. Fits v3's model + (`on`, roll, convergence are all app-scoped), but duplicates and mis-attributes + host-level facts: a box hosting three apps reports its load three times. +- **Server-shaped** — the subject is one server, N apps. Closest to today, and + system stats and accessories land naturally, but an app can only be compared + across replicas by switching servers and remembering. +- **Two-level (chosen)** — collect server-shaped, pivot to app-shaped for + presentation. + +Collection stays server-shaped, so nothing is polled twice and host-level facts +have one home. The pivot is a pure function over snapshots — no I/O, trivially +testable — and it is what lets `status` be a renderer rather than a second data +path. + +## Architecture + +``` +domain/observe/ + snapshot.ts ServerSnapshot / AppSnapshot / FleetView types + parse.ts section parsers (moved from cli/monitor/state.ts) + script.ts buildObserveScript — composite shell, section-selectable + collector.ts MetricsCollector — one server, N apps -> ServerSnapshot + pivot.ts pivotByApp(ServerSnapshot[]) -> FleetView[] + +services/observe/ + session.ts ObserveSession — N collectors, scheduling, history, events + history.ts MetricsHistory + +cli/monitor/ Ink TUI: subscribes to an ObserveSession +cli/commands/ status / logs / monitor — three presenters, one session +``` + +**Boundary:** nothing under `domain/observe/` or `services/observe/` imports +React, Ink, or chalk. Today `use-monitor-data.ts` mixes polling, alerting, and +`chalk.red(...)` event strings — presentation decisions baked into the data +layer. Events become typed values; colour is chosen at render. + +This follows the existing `domain/deploy` + `services` split rather than +inventing a new shape. + +## Types + +Today's `MetricsSnapshot` conflates three scopes: host-level (`system`), +app-level (`processes`, `releases`, `health`, `caddy`), and server-level +(`accessories`, `deployLock`). The split follows those seams. + +```ts +interface ServerSnapshot { + server: string; // the host string — identity everywhere, per ADR-0008 + timestamp: string; + system: SystemInfo; // collected once, not once per app + accessories?: AccessoryInfo[]; + deployLock: DeployLockInfo | null; + apps: AppSnapshot[]; + error?: string; // whole-server failure: unreachable, timeout +} + +interface AppSnapshot { + app: string; + appType: 'backend' | 'frontend'; + processes: ProcessInfo[]; + currentRelease: string | null; + releases: ReleaseRecord[]; + health?: HealthInfo; + caddy?: CaddyInfo; + error?: string; // per-app failure: PM2 down for this namespace +} + +interface FleetView { + app: string; + appType: 'backend' | 'frontend'; + replicas: Array<{ + server: string; + snapshot: AppSnapshot; + system: SystemInfo; + reachable: boolean; + }>; + convergence: FleetConvergence; +} + +function pivotByApp(snapshots: ServerSnapshot[]): FleetView[]; +``` + +`SystemInfo`, `ProcessInfo`, `HealthInfo`, `AccessoryInfo`, `CaddyInfo`, and +`ReleaseRecord` move over unchanged — they are already well-shaped. + +`deployLock` sits on the server because that is where the lock lives +(`{remotePath}/.shipnode/deploy.lock`), which today's per-app snapshot quietly +misrepresents. + +`assembleConvergence` is reused as-is: `assessConvergence` already takes +`ReplicaObservation[]` (`{ server, release }`), so the pivot feeds it directly. +No new convergence logic, and `status`'s fleet reporting keeps working through a +path now shared with the live view. + +## Data flow + +``` +ObserveSession.tick() + -> for each server, bounded-parallel: MetricsCollector.collect({ apps, sections }) + -> ServerSnapshot[] (partial: an unreachable server yields error, not a throw) + -> history.push per (server, app) + -> health streaks + typed events + -> pivotByApp() -> FleetView[] + -> notify subscribers with { servers, fleets, events, lastUpdate } +``` + +Both shapes are published every tick. The TUI's fleet view reads `fleets`, its +server view reads `servers`, `--json` emits `servers` (the collected truth) with +`fleets` derivable, and `status` renders the pivot. No mode re-polls. + +## Collection decisions + +Carried forward from today's poller: + +- One SSH round trip per server per poll. The composite shell script with + `@@SHIPNODE:@@` section markers is the good part of the existing poller + and survives intact — it now emits N apps per script instead of one, since + `getAppsForServer` already says what is on the box. +- Accessories stay sampled on a slower cadence (~10s) than the main poll, with + the previous value carried forward. `docker inspect` is the expensive section. +- The health probe stays bounded by `--max-time` derived from the interval, so a + hanging probe cannot stretch a tick. +- Each section carries its own fallback, so one failing probe cannot blank the + rest of the snapshot. + +New: + +- Parallel collection across servers is bounded at 4 concurrent connections; the + rest queue within the tick. A 12-host fleet must not open 12 SSH connections. +- A tick that cannot finish within the interval is skipped rather than + overlapped — today's `inFlightRef` guard, generalised per server. + +## Error handling + +Per AGENTS.md, `better-result` with typed `TaggedError` classes for expected +domain failures; exceptions only for programmer defects and adapter-boundary +infrastructure failures. + +Failure is per-scope and never aborts a tick: + +- A server that is unreachable or times out yields a `ServerSnapshot` with + `error` set and no app data. Other servers still report. +- An app whose PM2 section fails yields an `AppSnapshot` with `error` set. Other + apps on the same server still report. +- `assessConvergence` over a fleet with an unreachable replica reports the skew + it can see and names the replica it could not reach, rather than claiming + convergence from a partial observation. This matches the existing rule in + `reportFleetConvergence` that a narrowed run reports nothing rather than + reporting "converged" from one observation. + +## Testing + +`FakeRemoteExecutor` (`tests/testing/fake-executor.ts`) already supports +predicate-matched canned responses and command-history assertions, which covers +the collector without a network. + +- `parse.ts` — the existing parser tests in `tests/unit/monitor.test.ts` move + across unchanged; they are good and already cover malformed input. +- `script.ts` — section selection: which sections appear for backend vs frontend, + with and without health checks and accessories, and N apps in one script. +- `collector.ts` — a canned multi-section stdout parses into a `ServerSnapshot`; + a failing section degrades that section only; a timeout yields a server-level + error. +- `pivot.ts` — pure function, so table-driven: converged fleet, skewed fleet, + unreachable replica, single-server app, app absent from one server. +- `session.ts` — with fake collectors and fake timers: concurrency cap honoured, + overlapping ticks skipped, accessory cadence, health-streak transitions in both + directions, subscriber notification. + +## Migration + +The existing `cli/monitor/` TUI keeps working throughout: `state.ts` and +`poller.ts` become thin re-exports over `domain/observe/` until the TUI is +rewritten in the follow-up spec. No user-visible change lands until the surfaces +are rebuilt. diff --git a/src/cli/commands/config.ts b/src/cli/commands/config.ts index 649c1e5..05c20b8 100644 --- a/src/cli/commands/config.ts +++ b/src/cli/commands/config.ts @@ -15,17 +15,27 @@ function showApp(app: ShipnodeApp, nodeVersion: string): void { ['keepReleases', String(app.keepReleases)], ]); + if (app.runtime === 'watt' && app.watt) { + ui.section('Runtime: watt (wattpm)', [ + ['main', app.watt.main], + ['module', app.watt.module ?? '@platformatic/node'], + ['maxHeapUsed', app.watt.maxHeapUsed ?? '(from maxMemory)'], + ]); + } + if (app.pm2) { + const watt = app.runtime === 'watt'; for (const pm2App of app.pm2.apps) { const rows: [string, string][] = [['name', pm2App.name]]; if (pm2App.command) rows.push(['command', pm2App.command]); if (pm2App.port !== undefined) rows.push(['port', String(pm2App.port)]); - if (pm2App.instances !== undefined) rows.push(['instances', String(pm2App.instances)]); + if (pm2App.instances !== undefined) rows.push([watt && pm2App.port !== undefined ? 'threads' : 'instances', String(pm2App.instances)]); if (pm2App.maxMemory !== undefined) rows.push(['maxMemory', pm2App.maxMemory]); if (pm2App.env) { for (const [k, v] of Object.entries(pm2App.env)) rows.push([`env.${k}`, v]); } - ui.section(pm2App.port !== undefined ? `PM2 process: ${pm2App.name} (web)` : `PM2 process: ${pm2App.name}`, rows); + const label = watt ? 'Process' : 'PM2 process'; + ui.section(pm2App.port !== undefined ? `${label}: ${pm2App.name} (web)` : `${label}: ${pm2App.name}`, rows); } } diff --git a/src/cli/commands/doctor.ts b/src/cli/commands/doctor.ts index 977d21c..2479dee 100644 --- a/src/cli/commands/doctor.ts +++ b/src/cli/commands/doctor.ts @@ -42,6 +42,10 @@ function checkLocal(config: ShipnodeConfig): void { issues.push('PM2 apps are not configured for backend app'); } + for (const app of config.apps) { + if (app.runtime === 'watt' && !app.watt?.main) issues.push(`App '${app.name}' uses runtime 'watt' but has no watt.main`); + } + if (issues.length === 0) { ui.success('Local configuration looks good'); } else { @@ -59,7 +63,8 @@ export async function checkRemote( ui.info('Checking remote server...'); const needsNode = config.apps.length > 0; - const needsPm2 = config.apps.some((app) => app.appType === 'backend' && app.pm2); + const needsPm2 = config.apps.some((app) => app.appType === 'backend' && app.pm2 && app.runtime !== 'watt'); + const needsSystemd = config.apps.some((app) => app.appType === 'backend' && app.runtime === 'watt'); const needsCaddy = config.apps.some((app) => app.domain); const needsDocker = Object.keys(config.accessories ?? {}).length > 0; @@ -68,6 +73,7 @@ export async function checkRemote( const checks = [ ...(needsNode ? [{ name: 'Node', cmd: `${mise}; mise exec "node@${nodeVersion}" -- node --version` }] : []), ...(needsPm2 ? [{ name: 'PM2', cmd: `${mise}; mise exec "node@${nodeVersion}" -- pm2 --version` }] : []), + ...(needsSystemd ? [{ name: 'systemd', cmd: 'systemctl --version' }] : []), ...(needsCaddy ? [{ name: 'Caddy', cmd: 'caddy version' }] : []), ...(needsDocker ? [{ name: 'Docker', cmd: 'docker --version' }] : []), { name: 'rsync', cmd: 'rsync --version' }, diff --git a/src/cli/commands/env.ts b/src/cli/commands/env.ts index c1d9cdc..c96c353 100644 --- a/src/cli/commands/env.ts +++ b/src/cli/commands/env.ts @@ -4,6 +4,7 @@ import { resolve } from 'path'; import { runRemoteCommandForTargets } from '../runner.js'; import { ui } from '../ui.js'; import { getDeploymentName } from '../../domain/pm2/apps.js'; +import { isWatt, resolveWattUnits, restartUnitCommand } from '../../domain/runtime/watt.js'; import type { RemoteExecutor } from '../../domain/remote/executor.js'; function shellSingleQuote(value: string): string { @@ -82,6 +83,14 @@ export async function cmdEnv( continue; } + if (isWatt(app)) { + const units = await resolveWattUnits(executor, appPath, app, { colors: 'active' }); + ui.info(`Restarting '${app.name}' (${units.join(', ')}) to pick up environment variables...`); + for (const unit of units) await executor.execOrThrow(restartUnitCommand(unit)); + ui.success(`'${app.name}' restarted with new environment variables`); + continue; + } + const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion; const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; const checkResult = await executor.exec( diff --git a/src/cli/commands/harden.ts b/src/cli/commands/harden.ts index 6540c3d..6f3f333 100644 --- a/src/cli/commands/harden.ts +++ b/src/cli/commands/harden.ts @@ -218,7 +218,8 @@ export async function cmdHarden(cwd: string, options: { config?: string; on?: st ); changes.push(`PM2: refreshed dump for ${currentUser}`); } - } else { + } else if (config.apps.some((a) => a.appType === 'backend' && a.pm2 && a.runtime !== 'watt')) { + // Watt-only hosts have no PM2 unit by design: systemd starts their units at boot. ui.warn(`No ${wanted} found. If you switched ssh.user recently, re-run 'shipnode setup' as the new user or install pm2 startup manually.`); } diff --git a/src/cli/commands/help.ts b/src/cli/commands/help.ts index fe43f6f..d7adcd9 100644 --- a/src/cli/commands/help.ts +++ b/src/cli/commands/help.ts @@ -27,7 +27,7 @@ ${chalk.bold('PROCESS MANAGEMENT')} ${chalk.green('logs')} Show application logs ${chalk.green('restart')} Restart the application ${chalk.green('stop')} Stop the application - ${chalk.green('metrics')} Open PM2 monitoring dashboard + ${chalk.green('metrics')} Open process monitoring (pm2 monit, or systemctl status for watt) ${chalk.green('monitor')} Live TUI dashboard with stats and logs ${chalk.bold('SECURITY & MAINTENANCE')} @@ -60,7 +60,7 @@ ${chalk.bold('CONFIGURATION')} ${chalk.green('config path')} Print path to config file ${chalk.bold('CUSTOMIZATION')} - ${chalk.green('eject')} [target] Eject PM2/Caddy templates (pm2, caddy, all) + ${chalk.green('eject')} [target] Eject PM2/Caddy templates (pm2, caddy, all) — PM2 only, not the watt runtime ${chalk.green('upgrade')} Upgrade shipnode to the latest version ${chalk.bold('OPTIONS')} diff --git a/src/cli/commands/init.ts b/src/cli/commands/init.ts index 6963adc..acf885e 100644 --- a/src/cli/commands/init.ts +++ b/src/cli/commands/init.ts @@ -87,6 +87,7 @@ export async function cmdInit(cwd: string, options: { nonInteractive?: boolean; cancelIfNeeded(remotePath); let pm2Name = ''; + let wattMain: string | undefined; let backendPort = defaultPort; let domain = ''; let healthCheckPath = '/health'; @@ -105,6 +106,24 @@ export async function cmdInit(cwd: string, options: { nonInteractive?: boolean; cancelIfNeeded(pm2Val); pm2Name = pm2Val as string; + const runtimeVal = await select({ + message: 'Process runtime', + options: [ + { value: 'pm2', label: 'PM2 (default)' }, + { value: 'watt', label: 'wattpm — worker threads sharing one port (Linux, opt-in)' }, + ], + initialValue: 'pm2', + }); + cancelIfNeeded(runtimeVal); + if (runtimeVal === 'watt') { + const mainVal = await text({ + message: 'Entry file each worker thread loads (must listen on process.env.PORT)', + initialValue: 'dist/server.js', + }); + cancelIfNeeded(mainVal); + wattMain = mainVal as string; + } + const portVal = await text({ message: 'Backend port', initialValue: String(defaultPort) }); cancelIfNeeded(portVal); backendPort = parseInt(portVal as string, 10); @@ -203,6 +222,7 @@ export async function cmdInit(cwd: string, options: { nonInteractive?: boolean; sshPort, remotePath: remotePath as string, pm2Name, + wattMain, backendPort, domain: domain || undefined, healthCheckPath, @@ -252,6 +272,7 @@ interface ConfigOptions { sshPort?: number; remotePath?: string; pm2Name?: string; + wattMain?: string; backendPort?: number; domain?: string; healthCheckPath?: string; @@ -336,6 +357,10 @@ function generateConfig(opts: ConfigOptions): string { lines.push(` .port(${opts.backendPort ?? 3000})`); } + if (opts.wattMain) { + lines.push(` .runtime('watt', { main: '${opts.wattMain}' })`); + } + if (opts.domain) { lines.push(` .domain('${opts.domain}')`); } diff --git a/src/cli/commands/logs.ts b/src/cli/commands/logs.ts index 0f46b8e..9b54454 100644 --- a/src/cli/commands/logs.ts +++ b/src/cli/commands/logs.ts @@ -1,4 +1,5 @@ import { runRemoteCommandForTargets } from '../runner.js'; +import { isWatt, logsCommand, resolveWattUnits } from '../../domain/runtime/watt.js'; /** * Prefix every line, not every block. @@ -39,6 +40,16 @@ export async function cmdLogs(cwd: string, options: { lines?: number; config?: s const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; for (const app of apps) { + if (isWatt(app)) { + const units = await resolveWattUnits(executor, `${config.remotePath}/${app.name}`, app, { process: options.process, colors: 'active' }); + for (const unit of units) { + const result = await executor.exec(logsCommand(unit, { lines })); + const label = `[${serverName} ${app.name} ${unit}]`; + if (result.stdout) process.stdout.write(`${prefixLines(result.stdout, label)}\n`); + if (result.stderr) process.stderr.write(`${prefixLines(result.stderr, label)}\n`); + } + continue; + } const namespace = app.pm2!.apps[0].name; const target = options.process ? app.pm2!.apps.find((a) => a.name === options.process)?.name ?? namespace diff --git a/src/cli/commands/metrics.ts b/src/cli/commands/metrics.ts index df3eeeb..bb9968c 100644 --- a/src/cli/commands/metrics.ts +++ b/src/cli/commands/metrics.ts @@ -1,27 +1,36 @@ import { execa } from 'execa'; import { loadConfig } from '../../config/loader.js'; import { getActiveApp } from '../../domain/workspace.js'; -import { getDeploymentName } from '../../domain/pm2/apps.js'; import { getServerTargetResult } from '../../domain/servers.js'; import { ui } from '../ui.js'; +import { isWatt } from '../../domain/runtime/watt.js'; -export async function cmdMetrics(cwd: string, options: { config?: string; app?: string }): Promise { +export async function cmdMetrics(cwd: string, options: { config?: string; app?: string; on?: string }): Promise { const config = await loadConfig(cwd, options.config); const app = options.app ? getActiveApp(config, options.app) : config.apps[0]; - const target = getServerTargetResult(config, app.on); + const target = getServerTargetResult( + config, + options.on ?? app.on, + `App '${app.name}'`, + ); if (target.isErr()) { ui.error(target.error.message); process.exit(1); return; } - if (app.appType !== 'backend' || !getDeploymentName({ ...config, apps: [app] } as any)) { + if (app.appType !== 'backend' || app.pm2?.apps[0]?.name === undefined) { throw new Error('Metrics only available for backend apps with PM2'); } + const namespace = app.pm2.apps[0].name; const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion; const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; - const remoteCmd = `${mise}; mise exec "node@${nodeVersion}" -- pm2 monit`; + // systemd has no `pm2 monit`; a refreshing `systemctl status` shows the same + // essentials per unit (state, memory, CPU time, cgroup processes, recent log). + const remoteCmd = isWatt(app) + ? `watch -n 2 -t "systemctl status --no-pager 'shipnode-${namespace}' 'shipnode-${namespace}-*'"` + : `${mise}; mise exec "node@${nodeVersion}" -- pm2 monit`; const sshArgs = [ '-t', diff --git a/src/cli/commands/migrate.ts b/src/cli/commands/migrate.ts index f1d3ca4..1ab90ba 100644 --- a/src/cli/commands/migrate.ts +++ b/src/cli/commands/migrate.ts @@ -82,7 +82,11 @@ export async function cmdMigrate(cwd: string, options: { config?: string }): Pro // Reload PM2 if applicable const namespace = getDeploymentName(config); - if (app.appType === 'backend' && namespace) { + if (app.appType === 'backend' && app.runtime === 'watt') { + // Units and launchers embed the deploy path, so a restart would keep + // running from the old location. A deploy regenerates them. + ui.warn(`'${app.name}' uses the watt runtime: run 'shipnode deploy --app ${app.name}' to regenerate its systemd units for the new path.`); + } else if (app.appType === 'backend' && namespace) { const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion; const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; ui.info('Reloading PM2 from new path...'); diff --git a/src/cli/commands/monitor.ts b/src/cli/commands/monitor.ts index 5e65eb0..3c16903 100644 --- a/src/cli/commands/monitor.ts +++ b/src/cli/commands/monitor.ts @@ -1,16 +1,55 @@ import { loadConfig } from '../../config/loader.js'; import { SshConnection } from '../../infrastructure/ssh/connection.js'; +import { configForServer } from '../../domain/servers.js'; +import type { ShipnodeConfig } from '../../shared/types.js'; import { runMonitor } from '../monitor/index.js'; -import { getAccessoriesForMonitorTarget, getAppsForMonitorTarget, resolveMonitorSession } from '../monitor/monitor-session.js'; +import { + getAccessoriesForMonitorTarget, + getAppsForMonitorTarget, + resolveMonitorSession, +} from '../monitor/monitor-session.js'; +import { observeStateJson, printObserveStatus, takeSnapshot } from '../observe.js'; import { ui } from '../ui.js'; export async function cmdMonitor( cwd: string, - options: { interval?: string; app?: string; config?: string }, + options: { interval?: string; app?: string; config?: string; on?: string; once?: boolean; json?: boolean }, ): Promise { const interval = Math.max(1, parseInt(options.interval ?? '2', 10) || 2); const config = await loadConfig(cwd, options.config); - const session = resolveMonitorSession(config, options.app); + + if (options.once === true || options.json === true) { + await printSnapshot(config, options, interval); + return; + } + + await runLiveDashboard(config, { app: options.app, on: options.on, interval }); +} + +async function printSnapshot( + config: ShipnodeConfig, + options: { app?: string; on?: string; json?: boolean }, + intervalSeconds: number, +): Promise { + const snapshot = await takeSnapshot(config, { + app: options.app, + on: options.on, + intervalSeconds, + }); + if (snapshot.isErr()) { + ui.error(snapshot.error.message); + process.exit(1); + return; + } + if (options.json) process.stdout.write(observeStateJson(snapshot.value)); + else printObserveStatus(snapshot.value, { narrowed: options.on !== undefined }); +} + +async function runLiveDashboard( + config: ShipnodeConfig, + options: { app?: string; on?: string; interval: number }, +): Promise { + const session = resolveMonitorSession(config, options.app, options.on); if (session.isErr()) { ui.error(session.error.message); process.exit(1); @@ -23,7 +62,6 @@ export async function cmdMonitor( process.exit(1); return; } - const accessoryNames = getAccessoriesForMonitorTarget(config, session.value.target.name); if (accessoryNames.isErr()) { ui.error(accessoryNames.error.message); @@ -31,11 +69,11 @@ export async function cmdMonitor( return; } - const host = `${session.value.target.ssh.user}@${session.value.target.ssh.host}:${session.value.target.ssh.port}`; + const { target, app } = session.value; + const host = `${target.ssh.user}@${target.ssh.host}:${target.ssh.port}`; const ssh = new SshConnection(); - try { - await ssh.connect(session.value.target.ssh); + await ssh.connect(target.ssh); } catch (err) { const msg = err instanceof Error ? err.message : String(err); ui.error(`Failed to connect to ${host}: ${msg}`); @@ -46,13 +84,13 @@ export async function cmdMonitor( try { await runMonitor({ executor: ssh, - config, - app: session.value.app, + config: configForServer(config, target.name), + app, apps: apps.value, accessoryNames: accessoryNames.value, - targetName: session.value.target.name, + targetName: target.name, host, - interval, + interval: options.interval, }); } finally { ssh.disconnect(); diff --git a/src/cli/commands/restart.ts b/src/cli/commands/restart.ts index 0e32619..e303ee8 100644 --- a/src/cli/commands/restart.ts +++ b/src/cli/commands/restart.ts @@ -1,5 +1,6 @@ import { runRemoteCommandForTargets } from '../runner.js'; import { ui } from '../ui.js'; +import { isWatt, resolveWattUnits, restartUnitCommand } from '../../domain/runtime/watt.js'; export async function cmdRestart(cwd: string, options: { config?: string; process?: string; app?: string; on?: string }): Promise { await runRemoteCommandForTargets( @@ -24,6 +25,12 @@ export async function cmdRestart(cwd: string, options: { config?: string; proces const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; for (const app of apps) { + if (isWatt(app)) { + const units = await resolveWattUnits(executor, `${config.remotePath}/${app.name}`, app, { process: options.process, colors: 'active' }); + for (const unit of units) await executor.execOrThrow(restartUnitCommand(unit)); + ui.success(`App '${app.name}' restarted (systemd; in-place restart — redeploy for a zero-downtime roll)`); + continue; + } const namespace = app.pm2!.apps[0].name; const target = options.process ? app.pm2!.apps.find((a) => a.name === options.process)?.name ?? namespace diff --git a/src/cli/commands/rollback.ts b/src/cli/commands/rollback.ts index fa61012..95743e0 100644 --- a/src/cli/commands/rollback.ts +++ b/src/cli/commands/rollback.ts @@ -19,6 +19,7 @@ import { rollFleet, type FleetEvent } from '../../domain/deploy/fleet.js'; import { isFleet } from '../../domain/servers.js'; import type { RemoteExecutor } from '../../domain/remote/executor.js'; import type { ShipnodeConfig, ShipnodeApp } from '../../shared/types.js'; +import { isActiveCommand, isWatt, resolveWattUnits, restartUnitCommand, wattUnitName } from '../../domain/runtime/watt.js'; import { configForAppResult, configForServer, getServerTargets } from '../../domain/servers.js'; /** @@ -35,7 +36,7 @@ const alreadyConfirmed: Confirmer = async () => true; export async function cmdRollback( cwd: string, - options: { steps?: number; app?: string; config?: string; on?: string }, + options: { steps?: number; app?: string; config?: string; on?: string; yes?: boolean }, ): Promise { if (!options.app) { throw new Error( @@ -55,12 +56,12 @@ export async function cmdRollback( const stepsBack = options.steps ?? 1; if (isFleet(appConfig, app)) { - await rollbackFleet(appConfig, app, stepsBack, options.on); + await rollbackFleet(appConfig, app, stepsBack, options.on, options.yes === true); return; } await runRemoteCommandForConfig(appConfig, async ({ config, executor }) => { - await rollbackReplica(executor, config, app, stepsBack, confirm); + await rollbackReplica(executor, config, app, stepsBack, options.yes ? alreadyConfirmed : confirm); }); } @@ -79,6 +80,7 @@ async function rollbackFleet( app: ShipnodeApp, stepsBack: number, on: string | undefined, + yes: boolean, ): Promise { const allReplicas = getServerTargets(appConfig).map((target) => target.name); let replicas = allReplicas; @@ -92,7 +94,7 @@ async function rollbackFleet( } ui.warn(`Rolling ${app.name} back ${stepsBack} release(s) across ${replicas.join(', ')}, one replica at a time.`); - if (!(await confirm('Proceed with rollback?'))) { + if (!yes && !(await confirm('Proceed with rollback?'))) { ui.info('Rollback cancelled.'); return; } @@ -192,7 +194,13 @@ async function rollbackReplica( ui.success('Symlink switched'); const namespace = getDeploymentName(config); - if (app.appType === 'backend' && namespace) { + if (app.appType === 'backend' && isWatt(app)) { + // Launchers and configs are per-release, so restarting the units re-reads + // the rolled-back release through the `current` symlink. + const units = await resolveWattUnits(executor, appPath, app, { colors: 'active' }); + for (const unit of units) await executor.execOrThrow(restartUnitCommand(unit)); + ui.success('systemd units restarted'); + } else if (app.appType === 'backend' && namespace) { const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion; const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; // Prefer reloading from the rolled-back release's ecosystem file (ADR-0001 — it @@ -263,14 +271,19 @@ async function rollbackBlueGreen( // The previous colour must still be online to serve traffic after the flip. // Parse pm2's JSON in-process rather than relying on `node` being on the // remote PATH at rollback time. - const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; - const jlist = await executor.exec(`${mise} && mise exec -- pm2 jlist`); let online = false; - try { - const entries = JSON.parse(jlist.stdout.trim()) as Array<{ name: string; pm2_env?: { status?: string } }>; - online = entries.some((e) => e.name === previousName && e.pm2_env?.status === 'online'); - } catch { - online = false; + if (isWatt(app)) { + const unit = wattUnitName(namespace, webApp.name, previous); + online = (await executor.exec(isActiveCommand(unit))).exitCode === 0; + } else { + const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; + const jlist = await executor.exec(`${mise} && mise exec -- pm2 jlist`); + try { + const entries = JSON.parse(jlist.stdout.trim()) as Array<{ name: string; pm2_env?: { status?: string } }>; + online = entries.some((e) => e.name === previousName && e.pm2_env?.status === 'online'); + } catch { + online = false; + } } if (!online) { throw new Error( diff --git a/src/cli/commands/setup.ts b/src/cli/commands/setup.ts index 503718f..2f05636 100644 --- a/src/cli/commands/setup.ts +++ b/src/cli/commands/setup.ts @@ -15,6 +15,7 @@ import { } from '../../infrastructure/provisioning/commands.js'; import { isFleet } from '../../domain/servers.js'; import { loadUsersYml, saveUsersYml, syncUsers, upsertUser } from './user.js'; +import { isWatt } from '../../domain/runtime/watt.js'; import { Pm2StartupError } from '../../shared/result-errors.js'; const DEPLOY_USER = 'deploy'; @@ -177,7 +178,7 @@ export function buildTasks(executor: RemoteExecutor, config: ShipnodeConfig, own }, ], { concurrent: false }), }] : []), - ...(hasApps && config.apps.some((app) => app.appType === 'backend' && app.pm2) ? [{ + ...(hasApps && config.apps.some((app) => app.appType === 'backend' && app.pm2 && !isWatt(app)) ? [{ title: 'PM2', task: (_ctx: object, task: any) => task.newListr([ { diff --git a/src/cli/commands/status.ts b/src/cli/commands/status.ts index 56cfb91..772fda3 100644 --- a/src/cli/commands/status.ts +++ b/src/cli/commands/status.ts @@ -1,147 +1,14 @@ -import { runRemoteCommandForTargets } from '../runner.js'; +import { loadConfig } from '../../config/loader.js'; +import { printObserveStatus, takeSnapshot } from '../observe.js'; import { ui } from '../ui.js'; -import { getDeploymentName, getPm2Name } from '../../domain/pm2/apps.js'; -import { isFleet } from '../../domain/servers.js'; -import { - assessConvergence, - describeConvergence, - type ReplicaObservation, -} from '../../domain/deploy/convergence.js'; export async function cmdStatus(cwd: string, options: { config?: string; app?: string; on?: string }): Promise { - // Per-app, per-replica observations, gathered as the fan-out visits each - // server. Release skew is invisible from inside one server, so the comparison - // happens after every replica has reported. - const observed = new Map(); - const fleetApps = new Set(); - - await runRemoteCommandForTargets( - cwd, - async ({ config, executor, serverName }) => { - const apps = options.app - ? config.apps.filter((app) => app.name === options.app) - : config.apps; - - if (apps.length === 0) return; - ui.heading(`Server: ${serverName} (${config.ssh.user}@${config.ssh.host})`); - - for (const app of apps) { - ui.heading(`Status: ${app.name} (${app.appType})`); - - if (app.appType === 'backend' && app.pm2) { - const pm2Result = await executor.exec( - `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH" && pm2 jlist`, - ); - - if (pm2Result.exitCode === 0) { - try { - const allApps = JSON.parse(pm2Result.stdout) as Array<{ - name: string; - pid?: number; - pm2_env?: { status?: string; pm_uptime?: number; restart_time?: number }; - monit?: { memory: number; cpu: number }; - }>; - const declared = app.pm2.apps; - const namespace = getDeploymentName(config) ?? ''; - const byName = new Map(allApps.map((a) => [a.name, a])); - for (const pm2App of declared) { - const pm2Name = getPm2Name(namespace, pm2App.name); - const running = byName.get(pm2Name); - if (!running) { - // A process pinned to the primary is absent everywhere else by - // design; saying "not found" would report the feature as a fault. - if (pm2App.placement === 'primary') { - ui.info(` App '${pm2App.name}' is pinned to the primary replica — not expected here`); - } else { - ui.warn(` App '${pm2App.name}' not found in PM2`); - } - continue; - } - ui.heading(` PM2: ${pm2App.name}`); - ui.section(' Status', [ - ['Status', running.pm2_env?.status ?? 'unknown'], - ['PID', String(running.pid ?? 'N/A')], - ['Uptime', running.pm2_env?.pm_uptime ? new Date(running.pm2_env.pm_uptime).toISOString() : 'N/A'], - ['Restarts', String(running.pm2_env?.restart_time ?? 0)], - ['Memory', running.monit ? `${running.monit.memory} MB` : 'N/A'], - ['CPU', running.monit ? `${running.monit.cpu}%` : 'N/A'], - ]); - } - } catch { - ui.warn(' Could not parse PM2 output'); - } - } else { - ui.warn(' PM2 is not running'); - } - } - - const appPath = `${config.remotePath}/${app.name}`; - const currentResult = await executor.exec(`readlink "${appPath}/current" 2>/dev/null || echo "no current symlink"`); - const hasRelease = currentResult.stdout !== 'no current symlink' && currentResult.stdout !== ''; - if (hasRelease) { - ui.success(` Current release: ${currentResult.stdout}`); - } else { - ui.warn(' No active release'); - } - - const entries = observed.get(app.name) ?? []; - entries.push({ - server: serverName, - // The symlink is absolute; only the release directory name is - // comparable between replicas. - release: hasRelease ? currentResult.stdout.split('/').pop() ?? null : null, - }); - observed.set(app.name, entries); - - if (isFleet(config, app)) fleetApps.add(app.name); - - const releasesResult = await executor.exec(`ls -1t "${appPath}/releases/" 2>/dev/null | head -5`); - if (releasesResult.stdout) { - const releases = releasesResult.stdout.split('\n').filter(Boolean); - ui.section(' Recent Releases', releases.map((r: string, i: number) => [`#${i + 1}`, r])); - } - } - }, - { configPath: options.config, serverName: options.on }, - ); - - reportFleetConvergence(observed, fleetApps, options.on !== undefined); -} - -/** - * The cross-replica view, which is the only place a half-rolled fleet shows up. - * - * Skipped when `--on` narrowed the run to one server: a single observation - * proves nothing about the others, and reporting "converged" from it would be - * worse than saying nothing. - */ -function reportFleetConvergence( - observed: Map, - fleetApps: Set, - narrowed: boolean, -): void { - if (narrowed) return; - - for (const [appName, observations] of observed) { - if (!fleetApps.has(appName) || observations.length < 2) continue; - - const convergence = assessConvergence(observations); - ui.heading(`Fleet: ${appName}`); - ui.section( - ' Replicas', - observations.map((o) => [ - o.server, - o.release ?? 'no release', - ]), - ); - - if (convergence.converged) { - ui.success(` All ${observations.length} replicas on ${convergence.releases[0]}`); - continue; - } - - for (const line of describeConvergence(appName, observations, convergence)) { - ui.warn(` ${line}`); - } + const config = await loadConfig(cwd, options.config); + const snapshot = await takeSnapshot(config, { app: options.app, on: options.on }); + if (snapshot.isErr()) { + ui.error(snapshot.error.message); + process.exit(1); + return; } + printObserveStatus(snapshot.value, { narrowed: options.on !== undefined }); } diff --git a/src/cli/commands/stop.ts b/src/cli/commands/stop.ts index 6efe7dd..2b3023e 100644 --- a/src/cli/commands/stop.ts +++ b/src/cli/commands/stop.ts @@ -1,5 +1,6 @@ import { runRemoteCommandForTargets } from '../runner.js'; import { ui } from '../ui.js'; +import { isWatt, resolveWattUnits, stopUnitCommand } from '../../domain/runtime/watt.js'; export async function cmdStop(cwd: string, options: { config?: string; process?: string; app?: string; on?: string }): Promise { await runRemoteCommandForTargets( @@ -24,6 +25,12 @@ export async function cmdStop(cwd: string, options: { config?: string; process?: const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; for (const app of apps) { + if (isWatt(app)) { + const units = await resolveWattUnits(executor, `${config.remotePath}/${app.name}`, app, { process: options.process, colors: 'all' }); + for (const unit of units) await executor.execOrThrow(stopUnitCommand(unit)); + ui.warn(`App '${app.name}' stopped`); + continue; + } const namespace = app.pm2!.apps[0].name; const target = options.process ? app.pm2!.apps.find((a) => a.name === options.process)?.name ?? namespace diff --git a/src/cli/index.ts b/src/cli/index.ts index 83c454d..7e33c20 100644 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -105,8 +105,9 @@ program .option('--steps ', 'Number of releases to go back', '1') .option('--app ', 'App to roll back (required)') .option('--on ', 'Roll back one replica of a fleet instead of all of them') + .option('--yes', 'Roll back without confirmation (for CI and scripts)') .option('--config ', 'Use a specific config file') - .action((opts) => cmdRollback(process.cwd(), { steps: parseInt(opts.steps, 10), app: opts.app, on: opts.on, config: opts.config })); + .action((opts) => cmdRollback(process.cwd(), { steps: parseInt(opts.steps, 10), app: opts.app, on: opts.on, yes: opts.yes, config: opts.config })); program .command('migrate') @@ -120,7 +121,7 @@ program .command('env') .description('Upload local .env file to the server') .option('--file ', 'Path to .env file to upload (default: .env from config)') - .option('--no-reload', 'Upload without reloading running PM2 processes') + .option('--no-reload', 'Upload without reloading running processes') .option('--app ', 'Target a specific app') .option('--config ', 'Use a specific config file') .option('--on ', 'Target a specific server') @@ -140,16 +141,16 @@ program .command('logs') .description('Show application logs') .option('--lines ', 'Number of log lines to show', '100') - .option('--process ', 'Target a specific PM2 process') + .option('--process ', 'Target a specific process (PM2 app or watt unit)') .option('--app ', 'Target a specific app') .option('--config ', 'Use a specific config file') .option('--on ', 'Target a specific server') - .action((opts) => cmdLogs(process.cwd(), { lines: parseInt(opts.lines, 10), process: opts.process, app: opts.app, config: opts.config })); + .action((opts) => cmdLogs(process.cwd(), { lines: parseInt(opts.lines, 10), process: opts.process, app: opts.app, config: opts.config, on: opts.on })); program .command('restart') .description('Restart the application') - .option('--process ', 'Target a specific PM2 process') + .option('--process ', 'Target a specific process (PM2 app or watt unit)') .option('--app ', 'Target a specific app') .option('--config ', 'Use a specific config file') .option('--on ', 'Target a specific server') @@ -158,7 +159,7 @@ program program .command('stop') .description('Stop the application') - .option('--process ', 'Target a specific PM2 process') + .option('--process ', 'Target a specific process (PM2 app or watt unit)') .option('--app ', 'Target a specific app') .option('--config ', 'Use a specific config file') .option('--on ', 'Target a specific server') @@ -166,16 +167,20 @@ program program .command('metrics') - .description('Open PM2 monitoring dashboard') + .description('Open PM2 monit (or systemctl status for watt apps) over SSH (escape hatch; shipnode monitor is namespaced)') .option('--app ', 'Target a specific app') + .option('--on ', 'Target a specific replica of a fleet') .option('--config ', 'Use a specific config file') .action((opts) => cmdMetrics(process.cwd(), opts)); program .command('monitor') - .description('Live TUI dashboard with PM2 stats, system metrics, and logs') + .description('Live TUI dashboard with process stats (PM2 or systemd), system metrics, and logs') .option('--interval ', 'Polling interval in seconds (default: 2)', '2') .option('--app ', 'Target a specific app') + .option('--on ', 'Watch one replica (required for a live TUI on a fleet)') + .option('--once', 'Collect one snapshot and exit') + .option('--json', 'Print one snapshot as JSON (implies --once)') .option('--config ', 'Use a specific config file') .action((opts) => cmdMonitor(process.cwd(), opts)); diff --git a/src/cli/monitor/App.tsx b/src/cli/monitor/App.tsx index d2d501c..09d422a 100644 --- a/src/cli/monitor/App.tsx +++ b/src/cli/monitor/App.tsx @@ -72,7 +72,7 @@ export function App({ executor, config, app: initialApp, apps, accessoryNames, t setOverlay('none'); if (selectedInfo === undefined) return; monitor.appendEvent(chalk.yellow(`Restarting ${selectedInfo.pm2Name}…`)); - const result = await restartProcess(executor, selectedInfo.pm2Name); + const result = await restartProcess(executor, selectedInfo.pm2Name, selectedInfo.supervisor); if (result.isOk()) { monitor.appendEvent(chalk.green(`Restarted ${chalk.bold(selectedInfo.pm2Name)}`)); } else { diff --git a/src/cli/monitor/actions.ts b/src/cli/monitor/actions.ts index 362fba2..30f1ad1 100644 --- a/src/cli/monitor/actions.ts +++ b/src/cli/monitor/actions.ts @@ -4,6 +4,7 @@ import type { ShipnodeApp, ShipnodeConfig } from '../../shared/types.js'; import { ProcessRestartError, ReleaseRollbackError } from '../../shared/result-errors.js'; import { getEcosystemPath } from '../../domain/pm2/apps.js'; import { MISE, shellQuote } from './poller.js'; +import { isWatt, resolveWattUnits, restartUnitCommand } from '../../domain/runtime/watt.js'; /** * Restart exactly one PM2 process by its full pm2 name — never a bare @@ -12,9 +13,12 @@ import { MISE, shellQuote } from './poller.js'; export async function restartProcess( executor: RemoteExecutor, pm2Name: string, + supervisor?: 'systemd', ): Promise> { const result = await executor.exec( - `${MISE} && pm2 restart ${shellQuote(pm2Name)} --update-env`, + supervisor === 'systemd' + ? restartUnitCommand(pm2Name) + : `${MISE} && pm2 restart ${shellQuote(pm2Name)} --update-env`, ); if (result.exitCode !== 0) { return Result.err(new ProcessRestartError({ @@ -54,7 +58,16 @@ export async function rollbackToRelease( } const namespace = app.pm2?.apps[0]?.name; - if (app.appType === 'backend' && namespace !== undefined) { + if (app.appType === 'backend' && isWatt(app)) { + // Launchers and configs are per-release; restarting the units re-reads them. + const units = await resolveWattUnits(executor, appPath, app, { colors: 'active' }); + for (const unit of units) { + const restarted = await executor.exec(restartUnitCommand(unit)); + if (restarted.exitCode !== 0) { + return fail(`symlink switched but restarting ${unit} failed: ${(restarted.stderr || restarted.stdout).trim() || `exit code ${restarted.exitCode}`}`); + } + } + } else if (app.appType === 'backend' && namespace !== undefined) { const nodeVersion = config.nodeVersion === 'lts' ? '24' : config.nodeVersion; const ecosystem = getEcosystemPath(config, app.name); // Prefer the rolled-back release's ecosystem file (ADR-0001); fall back to diff --git a/src/cli/monitor/hooks/use-live-logs.ts b/src/cli/monitor/hooks/use-live-logs.ts index 0612326..0d0d20c 100644 --- a/src/cli/monitor/hooks/use-live-logs.ts +++ b/src/cli/monitor/hooks/use-live-logs.ts @@ -2,6 +2,7 @@ import { useEffect, useRef, useState } from 'react'; import type { RemoteExecutor } from '../../../domain/remote/executor.js'; import type { ShipnodeApp } from '../../../shared/types.js'; import { collectLogs, collectCaddyLogs } from '../poller.js'; +import { isWatt } from '../../../domain/runtime/watt.js'; const MAX_BUFFER_LINES = 500; @@ -15,7 +16,7 @@ export function useLiveLogs( app: ShipnodeApp, liveMode: boolean, interval: number, - /** Exact pm2 process name to tail, or null for the whole app namespace. */ + /** Exact process (pm2 name or systemd unit) to tail, or null for the whole app namespace. */ filter: string | null = null, ): LiveLogsState { const [logBuffer, setLogBuffer] = useState(''); @@ -33,7 +34,7 @@ export function useLiveLogs( } const target = filter ?? app.pm2?.apps[0]?.name; if (target === undefined) return ''; - return collectLogs(executor, target, 20); + return collectLogs(executor, target, 20, isWatt(app) ? 'systemd' : undefined); }; const pollLogs = async (): Promise => { diff --git a/src/cli/monitor/monitor-session.ts b/src/cli/monitor/monitor-session.ts index 49df52e..fdd22c6 100644 --- a/src/cli/monitor/monitor-session.ts +++ b/src/cli/monitor/monitor-session.ts @@ -12,6 +12,7 @@ export interface MonitorSession { export function resolveMonitorSession( config: ShipnodeConfig, appName?: string, + serverName?: string, ): ResultType { const app = appName === undefined ? config.apps[0] @@ -19,9 +20,10 @@ export function resolveMonitorSession( if (app === undefined) return Result.err(new UnknownAppError({ name: appName ?? '(default)' })); - // The monitor holds one live connection, so a fleet app must be narrowed to - // one replica first (`monitor --on `). - const target = getServerTargetResult(config, app.on, `App '${app.name}'`); + // The live TUI holds one connection. `--once` / `--json` observe the whole + // fleet; this path still needs a replica. `--on` picks it; otherwise the + // app's `on` must already name a single server. + const target = getServerTargetResult(config, serverName ?? app.on, `App '${app.name}'`); if (target.isErr()) return Result.err(target.error); return Result.ok({ config, app, target: target.value }); diff --git a/src/cli/monitor/panels/Pm2Panel.tsx b/src/cli/monitor/panels/Pm2Panel.tsx index 8cc772f..d6a37d8 100644 --- a/src/cli/monitor/panels/Pm2Panel.tsx +++ b/src/cli/monitor/panels/Pm2Panel.tsx @@ -37,7 +37,7 @@ export function Pm2Panel({ processes, cpuHistory, memHistory, health, responseHi if (processes.length === 0) { return ( - PM2 Processes + Processes No PM2 processes (frontend app) ); @@ -45,7 +45,7 @@ export function Pm2Panel({ processes, cpuHistory, memHistory, health, responseHi return ( - PM2 Processes + Processes {health !== undefined && } {processes.map((p, index) => { const uptimeSeconds = p.uptime > 0 ? Math.floor((Date.now() - p.uptime) / 1000) : 0; @@ -60,7 +60,7 @@ export function Pm2Panel({ processes, cpuHistory, memHistory, health, responseHi pid:{p.pid ?? '—'} {p.status} {p.execMode !== 'unknown' && ( - {p.execMode === 'cluster' ? `cluster×${p.instances}` : 'fork'} + {p.execMode === 'cluster' ? `cluster×${p.instances}` : p.execMode === 'threads' ? `threads×${p.instances}` : 'fork'} )} {p.nodeVersion !== undefined && node v{p.nodeVersion}} {p.restarts > 0 && restarts:{p.restarts}} diff --git a/src/cli/monitor/poller.ts b/src/cli/monitor/poller.ts index 35dd369..92233d4 100644 --- a/src/cli/monitor/poller.ts +++ b/src/cli/monitor/poller.ts @@ -1,129 +1,18 @@ -import type { RemoteExecutor } from '../../domain/remote/executor.js'; -import type { ShipnodeConfig, ShipnodeApp } from '../../shared/types.js'; -import type { MetricsSnapshot } from './state.js'; -import { - parsePm2Jlist, - parseSystemStats, - parseReleaseRecords, - parseDeployLock, - parseHealthProbe, - parseAccessoryStatus, - parseCaddyInfo, - splitSections, -} from './state.js'; - -export const MISE = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; -const PM2_FAILED = '##SHIPNODE_PM2_FAILED##'; - -export function shellQuote(value: string): string { - return `'${value.replace(/'/g, "'\"'\"'")}'`; -} - -function sectionMarker(name: string): string { - return `echo "@@SHIPNODE:${name}@@"`; -} - -function buildSystemSection(): string { - return [ - `echo "mem:$(free -mb | awk '/^Mem:/{print $2, $3}')"`, - `echo "load:$(awk '{print $1, $2, $3}' /proc/loadavg)"`, - `echo "cores:$(nproc 2>/dev/null || echo 1)"`, - `echo "uptime:$(cat /proc/uptime | awk '{print $1}')"`, - `echo "disk:$(df -BG --output=size,used / 2>/dev/null | tail -1 | awk '{print $1+0, $2+0}')"`, - ].join('; '); -} - -function buildLockSection(lockPath: string): string { - // Directory lock (atomic mkdir) stores the timestamp in acquired; - // legacy file locks store it as the file contents. - return ( - `if [ -d "${lockPath}" ]; then ` + - `echo "$(cat "${lockPath}/acquired" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` + - `elif [ -f "${lockPath}" ]; then ` + - `echo "$(cat "${lockPath}" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` + - `else echo "none"; fi` - ); -} - -function buildHealthSection(port: number, path: string, maxTimeSeconds: number): string { - const url = `http://localhost:${port}${path}`; - return ( - `start=$(date +%s%N); ` + - `code=$(curl -s -o /dev/null -w "%{http_code}" --max-time ${maxTimeSeconds} "${url}" 2>/dev/null); ` + - `end=$(date +%s%N); ` + - `echo "$code $(( (end - start) / 1000000 ))"` - ); -} - -function buildAccessoriesSection(names: string[]): string { - const containers = names.map((name) => shellQuote(`shipnode-${name}`)).join(' '); - // `sudo -n` fails instantly without a NOPASSWD rule instead of hanging the poll. - return ( - `sudo -n docker inspect ` + - `--format '{{.Name}}|{{.State.Status}}|{{.State.Health.Status}}|{{.Config.Image}}' ` + - `${containers} 2>/dev/null || true` - ); -} - -export interface MonitorCommandOptions { - /** Upper bound in seconds for the HTTP health probe, so it never stretches a poll. */ - healthMaxTimeSeconds?: number; - /** Accessory names (without the shipnode- prefix) to sample docker state for on this poll. */ - accessoryNames?: string[]; -} - /** - * One shell script covering every metric the monitor needs, so a poll costs a - * single SSH round trip. Sections are delimited by `@@SHIPNODE:@@` - * marker lines and joined with `;` — each section carries its own fallback so - * one failing probe cannot blank out the rest of the snapshot. + * The existing TUI's adapter onto the observation layer. + * + * `MetricsCollector` observes a whole server; this flattens the single-app case + * back into the `MetricsSnapshot` the current Ink components expect. It goes + * away with them when the dashboard subscribes to an `ObserveSession` directly. */ -export function buildMonitorCommand( - app: ShipnodeApp, - config: ShipnodeConfig, - options: MonitorCommandOptions = {}, -): string { - const appPath = `${config.remotePath}/${app.name}`; - const lockFile = `${config.remotePath}/.shipnode/deploy.lock`; - - const pm2Command = - app.appType === 'backend' && app.pm2 - ? `pm2 jlist 2>/dev/null || echo "${PM2_FAILED}"` - : `echo "[]"`; - - const sections: Array<[string, string]> = [ - ['pm2', pm2Command], - ['sys', buildSystemSection()], - ['current', `readlink "${appPath}/current" 2>/dev/null || echo "none"`], - ['releases', `cat "${appPath}/.shipnode/releases.json" 2>/dev/null || echo "[]"`], - ['lock', buildLockSection(lockFile)], - ]; - - const webPort = app.pm2?.apps.find((pm2App) => pm2App.port !== undefined)?.port; - if (app.healthCheck.enabled && webPort !== undefined) { - const maxTime = Math.max( - 1, - Math.min(app.healthCheck.timeout, options.healthMaxTimeSeconds ?? app.healthCheck.timeout), - ); - sections.push(['health', buildHealthSection(webPort, app.healthCheck.path, maxTime)]); - } - - if (options.accessoryNames !== undefined && options.accessoryNames.length > 0) { - sections.push(['accessories', buildAccessoriesSection(options.accessoryNames)]); - } - - if (app.appType === 'frontend') { - const logFile = `/var/log/caddy/${app.name}.log`; - sections.push(['caddy-status', `systemctl is-active caddy 2>/dev/null || echo "unknown"`]); - sections.push([ - 'caddy-log', - `sudo -n tail -n 50 "${logFile}" 2>/dev/null || tail -n 50 "${logFile}" 2>/dev/null || true`, - ]); - } +import type { RemoteExecutor } from '../../domain/remote/executor.js'; +import type { ShipnodeApp, ShipnodeConfig } from '../../shared/types.js'; +import { MetricsCollector } from '../../domain/observe/collector.js'; +import { parseSystemStats } from '../../domain/observe/parse.js'; +import { MISE, shellQuote } from '../../domain/observe/script.js'; +import type { MetricsSnapshot } from './state.js'; - const script = sections.map(([name, command]) => `${sectionMarker(name)}; ${command}`); - return [MISE, ...script].join('; '); -} +export { MISE, shellQuote }; export interface CollectMetricsOptions { /** Poll interval in seconds; bounds both the SSH timeout and the health probe. */ @@ -138,56 +27,71 @@ export async function collectMetrics( config: ShipnodeConfig, options: CollectMetricsOptions = {}, ): Promise { - const timestamp = new Date().toISOString(); - const namespace = app.pm2?.apps[0]?.name ?? ''; - const intervalSeconds = options.intervalSeconds ?? 2; - - const command = buildMonitorCommand(app, config, { - healthMaxTimeSeconds: intervalSeconds, + const collector = new MetricsCollector(executor, config.ssh.host, config); + const server = await collector.collect({ + apps: [app], + intervalSeconds: options.intervalSeconds, accessoryNames: options.accessoryNames, }); - const result = await executor.exec(command, { timeout: intervalSeconds * 3000 }); - const sections = splitSections(result.stdout); - const pm2Section = sections.get('pm2') ?? ''; - const currentRaw = (sections.get('current') ?? 'none').trim(); - const healthSection = sections.get('health'); - const accessoriesSection = sections.get('accessories'); + const observed = server.apps[0]; + if (observed === undefined) { + return { + timestamp: server.timestamp, + processes: [], + system: server.system, + currentRelease: null, + releases: [], + deployLock: server.deployLock, + accessories: server.accessories, + error: server.error ?? 'Monitor poll returned no data', + }; + } return { - timestamp, - processes: parsePm2Jlist(pm2Section, namespace), - system: parseSystemStats(sections.get('sys') ?? ''), - currentRelease: currentRaw === 'none' || currentRaw === '' ? null : currentRaw, - releases: parseReleaseRecords(sections.get('releases') ?? ''), - deployLock: parseDeployLock(sections.get('lock') ?? ''), - health: healthSection === undefined ? undefined : parseHealthProbe(healthSection) ?? undefined, - accessories: accessoriesSection === undefined ? undefined : parseAccessoryStatus(accessoriesSection), - caddy: - app.appType === 'frontend' - ? parseCaddyInfo(sections.get('caddy-status') ?? '', sections.get('caddy-log') ?? '') - : undefined, - error: resolveSnapshotError(app, pm2Section, sections.size), + timestamp: server.timestamp, + processes: observed.processes, + system: server.system, + currentRelease: observed.currentRelease, + releases: observed.releases, + deployLock: server.deployLock, + health: observed.health, + accessories: server.accessories, + caddy: observed.caddy, + error: server.error ?? observed.error, }; } -function resolveSnapshotError( - app: ShipnodeApp, - pm2Section: string, - sectionCount: number, -): string | undefined { - if (sectionCount === 0) return 'Monitor poll returned no data'; - if (app.appType === 'backend' && app.pm2 && pm2Section.includes(PM2_FAILED)) { - return 'PM2 command failed'; - } - return undefined; +/** A snapshot for a server that could not be reached at all. */ +export function unreachableSnapshot(error: string): MetricsSnapshot { + return { + timestamp: new Date().toISOString(), + processes: [], + system: parseSystemStats(''), + currentRelease: null, + releases: [], + deployLock: null, + error, + }; } export async function collectLogs( executor: RemoteExecutor, namespace: string, lines: number = 20, + supervisor?: 'systemd', ): Promise { + if (supervisor === 'systemd') { + // `namespace` is a unit name when a single process is selected, or the + // deployment namespace when tailing everything (matches its units by glob). + const units = namespace.startsWith('shipnode-') + ? `-u ${shellQuote(namespace)}` + : `-u ${shellQuote(`shipnode-${namespace}`)} -u ${shellQuote(`shipnode-${namespace}-*`)}`; + const journal = await executor.exec( + `S=$([ "$(id -u)" = 0 ] || echo sudo); $S journalctl ${units} -n ${lines} --no-pager 2>&1 || echo "(no logs)"`, + ); + return journal.stdout; + } const result = await executor.exec( `${MISE} && pm2 logs ${shellQuote(namespace)} --lines ${lines} --nostream 2>&1 || echo "(no logs)"`, ); diff --git a/src/cli/monitor/state.ts b/src/cli/monitor/state.ts index d4108a5..ab4f94c 100644 --- a/src/cli/monitor/state.ts +++ b/src/cli/monitor/state.ts @@ -1,88 +1,32 @@ -export interface ProcessInfo { - name: string; - pm2Name: string; - pid: number | null; - status: string; - cpu: number; - memory: number; - uptime: number; - restarts: number; - execMode: 'cluster' | 'fork' | 'unknown'; - instances: number; - unstableRestarts: number; - nodeVersion?: string; - exitCode: number | null; -} - -export interface SystemInfo { - load1: number; - load5: number; - load15: number; - cores: number; - totalMem: number; - usedMem: number; - totalDisk: number; - usedDisk: number; - uptime: number; -} - -/** CPU utilisation as a 0–1 fraction: 1-minute load normalised by core count. */ -export function systemCpuPercent(system: SystemInfo): number { - const load = system.load1 / Math.max(system.cores, 1); - return Math.max(0, Math.min(1, load)); -} - -export interface DeployLockInfo { - lockedAt: string; - ageSeconds: number; -} - -export interface HealthInfo { - status: 'ok' | 'fail'; - httpCode: number; - responseMs: number; -} - /** - * Consecutive-failure streak for the HTTP health probe. A poll without probe - * data (health check disabled or probe produced nothing) leaves the streak - * untouched — only a real `ok` result clears it. + * The existing TUI's view of the observation layer. + * + * Everything here now lives under `domain/observe` and `services/observe`; this + * module survives as the seam the current Ink components still import, and goes + * away with them when the dashboard is rebuilt on `ObserveSession`. */ -export function nextHealthFailStreak(previous: number, health: HealthInfo | undefined): number { - if (health === undefined) return previous; - return health.status === 'fail' ? previous + 1 : 0; -} - -export interface AccessoryInfo { - name: string; - status: string; - health: string; - image: string; -} - -export interface CaddyRequest { - status: number; - method: string; - uri: string; - ms: number; -} - -export interface CaddyInfo { - serviceActive: boolean; - total: number; - ok2xx: number; - err4xx: number; - err5xx: number; - recent: CaddyRequest[]; -} - -export interface ReleaseRecord { - timestamp: string; - status: string; - duration: number; - gitCommit?: string; -} +export * from '../../domain/observe/types.js'; +export * from '../../domain/observe/parse.js'; +export { MetricsHistory } from '../../services/observe/history.js'; + +import type { + AccessoryInfo, + CaddyInfo, + DeployLockInfo, + HealthInfo, + ProcessInfo, + ReleaseRecord, + SystemInfo, +} from '../../domain/observe/types.js'; +/** + * One app on one server, flattened. + * + * The observation layer keeps host-level facts (`system`, `accessories`, + * `deployLock`) on the server and app-level facts on the app, because a box + * hosting three apps has one load average rather than three. This shape + * predates that split and is kept only for the components below it. + */ export interface MetricsSnapshot { timestamp: string; processes: ProcessInfo[]; @@ -98,273 +42,3 @@ export interface MetricsSnapshot { caddy?: CaddyInfo; error?: string; } - -export class MetricsHistory { - cpu: number[] = []; - memory: number[] = []; - responseMs: number[] = []; - - constructor(private maxEntries: number = 60) {} - - push(snapshot: MetricsSnapshot): void { - const avgCpu = snapshot.processes.reduce((s, p) => s + p.cpu, 0) / Math.max(snapshot.processes.length, 1); - const avgMem = snapshot.processes.reduce((s, p) => s + p.memory, 0) / Math.max(snapshot.processes.length, 1); - - this.cpu.push(avgCpu); - this.memory.push(avgMem); - if (snapshot.health !== undefined) this.responseMs.push(snapshot.health.responseMs); - - if (this.cpu.length > this.maxEntries) this.cpu.shift(); - if (this.memory.length > this.maxEntries) this.memory.shift(); - if (this.responseMs.length > this.maxEntries) this.responseMs.shift(); - } - - clear(): void { - this.cpu = []; - this.memory = []; - this.responseMs = []; - } -} - -function readRecord(value: unknown): Record | null { - if (typeof value !== 'object' || value === null || Array.isArray(value)) return null; - // SAFETY: Runtime checks above establish a non-null plain object shape for indexed boundary parsing. - return value as Record; -} - -function readNumber(value: unknown): number | undefined { - return typeof value === 'number' ? value : undefined; -} - -function readString(value: unknown): string | undefined { - return typeof value === 'string' ? value : undefined; -} - -export function parsePm2Jlist(stdout: string, namespace: string): ProcessInfo[] { - try { - const raw: unknown = JSON.parse(stdout.trim()); - if (!Array.isArray(raw)) return []; - return raw.flatMap((entry) => parsePm2Entry(entry, namespace)); - } catch { - return []; - } -} - -function parsePm2Entry(entry: unknown, namespace: string): ProcessInfo[] { - const item = readRecord(entry); - if (item === null) return []; - const name = readString(item.name); - if (name === undefined) return []; - if (!belongsToNamespace(name, namespace)) return []; - - const pm2Env = readRecord(item.pm2_env); - const monit = readRecord(item.monit); - const memory = readNumber(monit?.memory); - - return [{ - name: resolveShortName(name, namespace), - pm2Name: name, - pid: readNumber(item.pid) ?? null, - status: readString(pm2Env?.status) ?? 'unknown', - cpu: readNumber(monit?.cpu) ?? 0, - memory: memory === undefined ? 0 : Math.round(memory / (1024 * 1024)), - uptime: readNumber(pm2Env?.pm_uptime) ?? 0, - restarts: readNumber(pm2Env?.restart_time) ?? 0, - execMode: parseExecMode(readString(pm2Env?.exec_mode)), - instances: readNumber(pm2Env?.instances) ?? 1, - unstableRestarts: readNumber(pm2Env?.unstable_restarts) ?? 0, - nodeVersion: readString(pm2Env?.node_version), - exitCode: readNumber(pm2Env?.exit_code) ?? null, - }]; -} - -function parseExecMode(raw: string | undefined): ProcessInfo['execMode'] { - if (raw === 'cluster_mode') return 'cluster'; - if (raw === 'fork_mode') return 'fork'; - return 'unknown'; -} - -function belongsToNamespace(pm2Name: string, namespace: string): boolean { - if (namespace === '') return true; - return pm2Name === namespace || pm2Name.startsWith(`${namespace}-`); -} - -export function parseSystemStats(stdout: string): SystemInfo { - const lines = stdout.trim().split('\n').flatMap((line) => line ? [line] : []); - const result: SystemInfo = { - load1: 0, - load5: 0, - load15: 0, - cores: 1, - totalMem: 0, - usedMem: 0, - totalDisk: 0, - usedDisk: 0, - uptime: 0, - }; - - for (const line of lines) { - if (line.startsWith('mem:')) { - const parts = line.split(/[:\s]+/); - result.totalMem = parseInt(parts[1] ?? '0', 10) || 0; - result.usedMem = parseInt(parts[2] ?? '0', 10) || 0; - } else if (line.startsWith('load:')) { - const parts = line.split(/[:\s]+/); - result.load1 = parseFloat(parts[1] ?? '0') || 0; - result.load5 = parseFloat(parts[2] ?? '0') || 0; - result.load15 = parseFloat(parts[3] ?? '0') || 0; - } else if (line.startsWith('cores:')) { - result.cores = parseInt(line.split(/[:\s]+/)[1] ?? '1', 10) || 1; - } else if (line.startsWith('uptime:')) { - result.uptime = parseInt(line.split(/[:\s]+/)[1] ?? '0', 10) || 0; - } else if (line.startsWith('disk:')) { - const parts = line.split(/[:\s]+/); - result.totalDisk = parseInt(parts[1] ?? '0', 10) || 0; - result.usedDisk = parseInt(parts[2] ?? '0', 10) || 0; - } - } - - return result; -} - -const SECTION_MARKER = /^@@SHIPNODE:([a-z0-9-]+)@@$/; - -/** - * Split the combined monitor command output into named sections. Missing or - * empty sections simply do not appear in the map — callers treat absence as - * "no data", never as an error. - */ -export function splitSections(stdout: string): Map { - const sections = new Map(); - let current: string | null = null; - let buffer: string[] = []; - - const flush = (): void => { - if (current !== null) sections.set(current, buffer.join('\n').trim()); - }; - - for (const line of stdout.split('\n')) { - const match = SECTION_MARKER.exec(line.trim()); - if (match !== null) { - flush(); - current = match[1]; - buffer = []; - } else if (current !== null) { - buffer.push(line); - } - } - flush(); - return sections; -} - -export function parseDeployLock(section: string): DeployLockInfo | null { - const trimmed = section.trim(); - if (trimmed === '' || trimmed === 'none') return null; - - const parts = trimmed.split(/\s+/); - const age = parseInt(parts[parts.length - 1] ?? '', 10); - if (Number.isNaN(age)) return null; - - return { - lockedAt: parts.slice(0, -1).join(' '), - ageSeconds: Math.max(age, 0), - }; -} - -export function parseHealthProbe(section: string): HealthInfo | null { - const trimmed = section.trim(); - if (trimmed === '') return null; - - const parts = trimmed.split(/\s+/); - const httpCode = parseInt(parts[0] ?? '', 10); - const responseMs = parseInt(parts[1] ?? '', 10); - if (Number.isNaN(httpCode) || Number.isNaN(responseMs)) return null; - - return { - status: httpCode >= 200 && httpCode < 400 ? 'ok' : 'fail', - httpCode, - responseMs, - }; -} - -export function parseAccessoryStatus(section: string): AccessoryInfo[] { - return section.split('\n').flatMap((line) => { - const trimmed = line.trim(); - if (trimmed === '') return []; - - const parts = trimmed.split('|'); - if (parts.length < 4) return []; - - const [rawName, status, health, image] = parts; - return [{ - name: rawName.replace(/^\/?(?:shipnode-)?/, ''), - status, - health: health === '' || health === '' ? '-' : health, - image, - }]; - }); -} - -export function parseCaddyInfo(statusSection: string, logSection: string): CaddyInfo { - const info: CaddyInfo = { - serviceActive: statusSection.trim() === 'active', - total: 0, - ok2xx: 0, - err4xx: 0, - err5xx: 0, - recent: [], - }; - - for (const line of logSection.split('\n')) { - const request = parseCaddyLogLine(line); - if (request === null) continue; - info.total += 1; - if (request.status >= 500) info.err5xx += 1; - else if (request.status >= 400) info.err4xx += 1; - else info.ok2xx += 1; - info.recent.push(request); - } - - info.recent = info.recent.slice(-5); - return info; -} - -function parseCaddyLogLine(line: string): CaddyRequest | null { - const trimmed = line.trim(); - if (!trimmed.startsWith('{')) return null; - - try { - const raw: unknown = JSON.parse(trimmed); - const record = readRecord(raw); - if (record === null) return null; - - const status = readNumber(record.status); - if (status === undefined) return null; - - const request = readRecord(record.request); - const duration = readNumber(record.duration); - return { - status, - method: readString(request?.method) ?? '', - uri: readString(request?.uri) ?? '', - ms: duration === undefined ? 0 : Math.round(duration * 1000), - }; - } catch { - return null; - } -} - -export function parseReleaseRecords(stdout: string): ReleaseRecord[] { - try { - const records: ReleaseRecord[] = JSON.parse(stdout.trim()); - return records.reverse().slice(0, 10); - } catch { - return []; - } -} - -function resolveShortName(pm2Name: string, namespace: string): string { - if (pm2Name === namespace) return pm2Name; - if (pm2Name.startsWith(`${namespace}-`)) return pm2Name.slice(namespace.length + 1); - return pm2Name; -} diff --git a/src/cli/observe.ts b/src/cli/observe.ts new file mode 100644 index 0000000..58b9973 --- /dev/null +++ b/src/cli/observe.ts @@ -0,0 +1,235 @@ +import { Result, type Result as ResultType } from 'better-result'; +import { describeConvergence } from '../domain/deploy/convergence.js'; +import { MetricsCollector } from '../domain/observe/collector.js'; +import { parseSystemStats } from '../domain/observe/parse.js'; +import { releaseNameOf, type FleetView, type ServerSnapshot } from '../domain/observe/snapshot.js'; +import { configForAppResult, configForServer, getServerTargets, type ServerTarget } from '../domain/servers.js'; +import { SshConnection } from '../infrastructure/ssh/connection.js'; +import type { AppTargetError, ServerTargetError } from '../shared/result-errors.js'; +import { UnknownServerTargetError } from '../shared/result-errors.js'; +import type { ShipnodeApp, ShipnodeConfig, SshConfig } from '../shared/types.js'; +import type { ObserveState, ObserveTarget } from '../services/observe/session.js'; +import { ObserveSession } from '../services/observe/session.js'; +import { getAccessoriesForMonitorTarget, getAppsForMonitorTarget } from './monitor/monitor-session.js'; +import { ui } from './ui.js'; + +export interface ObserveHostPlan { + name: string; + ssh: SshConfig; + apps: ShipnodeApp[]; + accessoryNames: string[]; +} + +export interface ObserveFilter { + app?: string; + on?: string; + intervalSeconds?: number; +} + +/** + * One observation of the workspace: plan hosts, poll, close. + * + * Status, `--once`, and `--json` are the same use case with different printers. + */ +export async function takeSnapshot( + config: ShipnodeConfig, + filter: ObserveFilter = {}, +): Promise> { + const hosts = planObserveHosts(config, filter); + if (hosts.isErr()) return Result.err(hosts.error); + + const connections: SshConnection[] = []; + const targets: ObserveTarget[] = []; + try { + for (const host of hosts.value) { + targets.push(await connectHost(config, host, connections)); + } + const session = new ObserveSession({ + targets, + intervalSeconds: filter.intervalSeconds ?? 2, + }); + await session.tick(); + return Result.ok(session.getState()); + } finally { + for (const ssh of connections) ssh.disconnect(); + } +} + +export function planObserveHosts( + config: ShipnodeConfig, + filter: ObserveFilter = {}, +): ResultType { + const scoped = filter.app === undefined ? Result.ok(config) : configForAppResult(config, filter.app); + if (scoped.isErr()) return Result.err(scoped.error); + + const selected = selectServers(scoped.value, filter.on); + if (selected.isErr()) return Result.err(selected.error); + + const hosts: ObserveHostPlan[] = []; + for (const target of selected.value) { + const apps = getAppsForMonitorTarget(scoped.value, target.name); + if (apps.isErr()) return Result.err(apps.error); + const accessoryNames = getAccessoriesForMonitorTarget(scoped.value, target.name); + if (accessoryNames.isErr()) return Result.err(accessoryNames.error); + if (apps.value.length === 0 && accessoryNames.value.length === 0) continue; + hosts.push({ + name: target.name, + ssh: target.ssh, + apps: apps.value, + accessoryNames: accessoryNames.value, + }); + } + return Result.ok(hosts); +} + +function selectServers( + config: ShipnodeConfig, + on: string | undefined, +): ResultType { + const targets = getServerTargets(config); + if (on === undefined) return Result.ok(targets); + + const match = targets.filter((target) => target.name === on); + if (match.length === 0) { + const known = targets.map((target) => target.name).join(', ') || '(none)'; + return Result.err(new UnknownServerTargetError({ target: on, known })); + } + return Result.ok(match); +} + +async function connectHost( + config: ShipnodeConfig, + host: ObserveHostPlan, + connections: SshConnection[], +): Promise { + const ssh = new SshConnection(); + try { + await ssh.connect(host.ssh); + connections.push(ssh); + return { + observer: new MetricsCollector(ssh, host.name, configForServer(config, host.name)), + apps: host.apps, + accessoryNames: host.accessoryNames, + }; + } catch (cause: unknown) { + ssh.disconnect(); + const message = cause instanceof Error ? cause.message : String(cause); + return { + observer: unreachableObserver(host.name, `Failed to connect: ${message}`), + apps: host.apps, + accessoryNames: host.accessoryNames, + }; + } +} + +function unreachableObserver(serverName: string, message: string): ObserveTarget['observer'] { + return { + serverName, + async collect(): Promise { + return { + server: serverName, + timestamp: new Date().toISOString(), + system: parseSystemStats(''), + deployLock: null, + apps: [], + error: message, + }; + }, + }; +} + +export function printObserveStatus(state: ObserveState, options: { narrowed: boolean }): void { + for (const server of state.servers) printServer(server); + if (options.narrowed) return; + for (const fleet of state.fleets) { + if (fleet.replicas.length + fleet.unreachable.length < 2) continue; + printFleet(fleet); + } +} + +function printServer(server: ServerSnapshot): void { + ui.heading(`Server: ${server.server}`); + if (server.error !== undefined) { + ui.warn(` Unreachable: ${server.error}`); + return; + } + + ui.section(' System', [ + ['Load', `${server.system.load1.toFixed(2)} ${server.system.load5.toFixed(2)} ${server.system.load15.toFixed(2)}`], + ['Memory', `${server.system.usedMem} / ${server.system.totalMem} MB`], + ]); + if (server.deployLock) { + ui.warn(` Deploy lock held since ${server.deployLock.lockedAt} (${server.deployLock.ageSeconds}s)`); + } + + for (const app of server.apps) { + ui.heading(` App: ${app.app} (${app.appType})`); + if (app.error !== undefined) { + ui.warn(` ${app.error}`); + continue; + } + for (const process of app.processes) { + ui.section(` ${process.supervisor === 'systemd' ? 'systemd' : 'PM2'}: ${process.name}`, [ + ['Status', process.status], + ['PID', String(process.pid ?? 'N/A')], + ['Restarts', String(process.restarts)], + ['Memory', `${process.memory} MB`], + ['CPU', `${process.cpu}%`], + ]); + } + const release = releaseNameOf(app); + if (release) ui.success(` Current release: ${release}`); + else ui.warn(' No active release'); + if (app.releases.length > 0) { + ui.section( + ' Recent Releases', + app.releases.slice(0, 5).map((record, index) => [`#${index + 1}`, record.timestamp]), + ); + } + if (app.health) { + ui.section(' Health', [ + ['Status', app.health.status], + ['HTTP', String(app.health.httpCode)], + ['Ms', String(app.health.responseMs)], + ]); + } + } +} + +function printFleet(fleet: FleetView): void { + ui.heading(`Fleet: ${fleet.app}`); + ui.section( + ' Replicas', + fleet.replicas.map((replica) => [ + replica.server, + replica.reachable ? (releaseNameOf(replica.snapshot) ?? 'no release') : 'unreachable', + ]), + ); + if (fleet.unreachable.length > 0) { + ui.warn(` Unreachable: ${fleet.unreachable.join(', ')}`); + } + if (fleet.convergence.converged && fleet.unreachable.length === 0) { + ui.success(` All ${fleet.replicas.length} replicas on ${fleet.convergence.releases[0] ?? 'no release'}`); + return; + } + const observations = fleet.replicas.map((replica) => ({ + server: replica.server, + release: releaseNameOf(replica.snapshot), + })); + for (const line of describeConvergence(fleet.app, observations, fleet.convergence)) { + ui.warn(` ${line}`); + } +} + +export function observeStateJson(state: ObserveState): string { + return `${JSON.stringify( + { + servers: state.servers, + fleets: state.fleets, + events: state.events, + lastUpdate: state.lastUpdate, + }, + null, + 2, + )}\n`; +} diff --git a/src/config/assembly.ts b/src/config/assembly.ts index 12ce312..2ac19d6 100644 --- a/src/config/assembly.ts +++ b/src/config/assembly.ts @@ -37,6 +37,8 @@ type AssembleInput = { domain?: string; caddy?: ShipnodeApp['caddy']; pm2?: LegacyPm2Input | Pm2Config; + runtime?: 'pm2' | 'watt'; + watt?: { main: string; module?: string; maxHeapUsed?: string }; backend?: { port?: number }; healthCheck?: unknown; envFile?: string; diff --git a/src/config/builder.ts b/src/config/builder.ts index 0c97405..3602e92 100644 --- a/src/config/builder.ts +++ b/src/config/builder.ts @@ -14,6 +14,8 @@ import type { HookFn, HooksConfig, PkgManager, + AppRuntime, + WattConfig, } from '../shared/types.js'; import { assembleConfig } from './assembly.js'; @@ -35,6 +37,8 @@ type BuilderState = { app?: string; on?: string | string[]; pm2?: { apps: Pm2App[] }; + runtime?: AppRuntime; + watt?: WattConfig; domain?: string; keepReleases?: number; healthCheck?: Partial; @@ -128,6 +132,19 @@ export class ShipnodeBuilder { return this; } + /** + * Opt in to the wattpm runtime: the web app runs as worker threads sharing the + * port via SO_REUSEPORT (its `instances` becomes the thread count) instead of + * PM2 cluster/fork processes. `pm2` stays the default. + */ + runtime(kind: 'watt', opts: WattConfig): this; + runtime(kind: 'pm2'): this; + runtime(kind: AppRuntime, opts?: WattConfig): this { + this.config.runtime = kind; + this.config.watt = kind === 'watt' ? opts : undefined; + return this; + } + port(n: number): this { this.firstApp().port = n; return this; @@ -355,6 +372,19 @@ export class ShipnodeAppBuilder { return this; } + /** + * Opt in to the wattpm runtime: the web app runs as worker threads sharing the + * port via SO_REUSEPORT (its `instances` becomes the thread count) instead of + * PM2 cluster/fork processes. `pm2` stays the default. + */ + runtime(kind: 'watt', opts: WattConfig): this; + runtime(kind: 'pm2'): this; + runtime(kind: AppRuntime, opts?: WattConfig): this { + this.state.runtime = kind; + this.state.watt = kind === 'watt' ? opts : undefined; + return this; + } + port(n: number): this { this.firstPm2App().port = n; return this; diff --git a/src/config/schema.ts b/src/config/schema.ts index 00a5c62..51a3f07 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -174,6 +174,12 @@ export const Pm2ConfigSchema = z.object({ { message: 'pm2.apps entries must have unique names' }, ); +export const WattConfigSchema = z.object({ + main: z.string().min(1, 'watt.main is required (the file each worker thread loads)'), + module: z.string().min(1).optional(), + maxHeapUsed: z.string().regex(/^\d+[KMG]?$/i, 'maxHeapUsed must look like 512M or 1G').optional(), +}); + export const HealthCheckConfigSchema = z.object({ enabled: z.boolean().default(true), path: z.string().default('/health'), @@ -264,6 +270,8 @@ export const ShipnodeAppSchema = z.object({ append: z.string().optional(), }).optional(), pm2: Pm2ConfigSchema.optional(), + runtime: z.enum(['pm2', 'watt']).optional(), + watt: WattConfigSchema.optional(), healthCheck: HealthCheckConfigSchema, envFile: z.string().default('.env'), keepReleases: z.number().int().min(1).default(5), @@ -300,6 +308,18 @@ export const ShipnodeAppSchema = z.object({ return webPort === undefined || cfg.altPort !== webPort; }, { message: 'altPort must differ from the web app port (blue and green need distinct ports)', path: ['altPort'] }, +).refine( + (cfg) => cfg.runtime !== 'watt' || cfg.watt !== undefined, + { message: "runtime 'watt' requires a watt config: set watt.main (the entry file each worker thread loads)", path: ['watt'] }, +).refine( + (cfg) => cfg.runtime === 'watt' || cfg.watt === undefined, + { message: "watt settings require runtime 'watt'", path: ['runtime'] }, +).refine( + (cfg) => cfg.runtime !== 'watt' || (cfg.pm2?.apps.some((a) => a.port !== undefined) ?? false), + { message: "runtime 'watt' requires a web app: one process must declare a port", path: ['runtime'] }, +).refine( + (cfg) => cfg.runtime !== 'watt' || cfg.appType === 'backend', + { message: "runtime 'watt' is for backend apps", path: ['runtime'] }, ); // A z.preprocess wrapper synthesizes `apps[0]` from the legacy top-level fields when @@ -527,6 +547,8 @@ export const ShipnodeConfigSchema = z.preprocess( domain: obj.domain, caddy: obj.caddy, pm2: obj.pm2, + runtime: obj.runtime, + watt: obj.watt, healthCheck: obj.healthCheck, envFile: obj.envFile, keepReleases: obj.keepReleases, diff --git a/src/domain/deploy/backend-strategy.ts b/src/domain/deploy/backend-strategy.ts index 52dc66a..c02f744 100644 --- a/src/domain/deploy/backend-strategy.ts +++ b/src/domain/deploy/backend-strategy.ts @@ -11,6 +11,12 @@ import { DeployError } from '../../shared/errors.js'; import type { DeploymentStrategy, StrategyContext } from './strategy.js'; import { runWithDotenv } from './dotenv.js'; import { envSymlinkCommand } from './env-links.js'; +import { + WATT_APP_FILE, WATT_RUNTIME_FILE, WATT_START_COMMAND, + installUnitCommand, isWatt, pm2DeleteCommand, portFreeGuard, removeUnitCommand, renderAppConfig, renderRunScript, + renderRuntimeConfig, renderUnit, restartUnitCommand, runScriptName, stopUnitCommand, wattEnsureInstalledCommand, wattInstalledGuard, wattUnitName, +} from '../runtime/watt.js'; +import type { DeployColor } from './blue-green.js'; function escapeSingleQuotes(s: string): string { return s.replace(/'/g, "\\'"); @@ -143,6 +149,11 @@ export class BackendStrategy implements DeploymentStrategy { const cdPath = `${this.appPath}/current`; const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; + if (isWatt(this.app)) { + await this.startWatt(ctx, pkgManager, cdPath, mise); + return; + } + if (this.app.zeroDowntime && ctx.deployTarget) { await this.startBlueGreen(ctx, pkgManager, cdPath, mise); return; @@ -258,6 +269,17 @@ export class BackendStrategy implements DeploymentStrategy { async afterHealthy(ctx: StrategyContext): Promise { if (!this.app.zeroDowntime || !ctx.deployTarget || !this.app.pm2) return; + if (isWatt(this.app)) { + // Workers are systemd units written (not started) by startWatt; start them + // now that the new release passed health. + const namespace = this.app.pm2.apps[0].name; + const workers = this.placedApps(ctx).filter((a) => a.port === undefined); + for (const worker of workers) { + await ctx.executor.execOrThrow(restartUnitCommand(wattUnitName(namespace, worker.name))); + } + return; + } + const workers = this.placedApps(ctx).filter((a) => a.port === undefined); if (workers.length === 0) return; @@ -276,6 +298,11 @@ export class BackendStrategy implements DeploymentStrategy { async afterTrafficSwitch(ctx: StrategyContext): Promise { if (!this.app.zeroDowntime || !ctx.deployTarget || !this.app.pm2) return; + if (isWatt(this.app)) { + await this.reapPreviousWattColor(ctx); + return; + } + const namespace = this.app.pm2.apps[0].name; const webApp = this.app.pm2.apps.find((app) => app.port !== undefined); if (!webApp) return; @@ -304,6 +331,151 @@ export class BackendStrategy implements DeploymentStrategy { ); } + // ------------------------------------------------------------------------- + // watt runtime (opt-in). The web app runs under wattpm as worker threads; + // workers are plain systemd units. See ../runtime/watt.ts and ADR-0009. + // ------------------------------------------------------------------------- + + private get wattWebRoot(): string { + return this.app.appRoot ? `${this.appPath}/current/${this.app.appRoot}` : `${this.appPath}/current`; + } + + /** Write one file into the release being staged. `printf` keeps content byte-exact. */ + private async writeReleaseFile(ctx: StrategyContext, path: string, content: string): Promise { + await ctx.executor.execOrThrow(`printf '%s' ${shellSingleQuote(content)} > "${path}"`); + } + + /** + * Render and install everything one declared process needs: its launcher + * script (per release) and its systemd unit. Returns the unit name. Does not + * start it — callers decide when, so blue-green can hold workers back. + */ + private async provisionWattProcess( + ctx: StrategyContext, + proc: Pm2App, + pkgManager: string, + color?: DeployColor, + portOverride?: number, + ): Promise { + const namespace = this.app.pm2!.apps[0].name; + const isWeb = proc.port !== undefined; + const port = portOverride ?? proc.port; + const env: Record = { NODE_ENV: 'production' }; + if (port !== undefined) env.PORT = port; + for (const [k, v] of Object.entries(proc.env ?? {})) env[k] = v; + + const command = isWeb + ? WATT_START_COMMAND + : (proc.command ?? `${pkgManager} start`); + + const script = renderRunScript({ + cwd: this.wattWebRoot, + command, + envFile: this.app.envFile ? `${this.appPath}/shared/${this.app.envFile}` : undefined, + env, + }); + const scriptName = runScriptName(proc.name, color); + await this.writeReleaseFile(ctx, `${ctx.workDir}/${scriptName}`, script); + + const unit = wattUnitName(namespace, proc.name, color); + await ctx.executor.execOrThrow(installUnitCommand(unit, renderUnit({ + description: `shipnode ${namespace}/${proc.name}${color ? ` (${color})` : ''}`, + user: this.workspace.ssh.user, + workingDirectory: `${this.appPath}/current`, + script: `${this.appPath}/current/${scriptName}`, + }))); + return unit; + } + + /** Write wattpm's runtime + capability config into the release's app root. */ + private async writeWattConfigs(ctx: StrategyContext, web: Pm2App): Promise { + const watt = this.app.watt!; + const root = this.app.appRoot ? `${ctx.workDir}/${this.app.appRoot}` : ctx.workDir; + await this.writeReleaseFile(ctx, `${root}/${WATT_RUNTIME_FILE}`, renderRuntimeConfig(web, watt)); + await this.writeReleaseFile(ctx, `${root}/${WATT_APP_FILE}`, renderAppConfig(watt)); + } + + private async startWatt( + ctx: StrategyContext, + pkgManager: PkgManager, + cdPath: string, + mise: string, + ): Promise { + const pm2 = this.app.pm2!; + const watt = this.app.watt!; + const namespace = pm2.apps[0].name; + const web = pm2.apps.find((a) => a.port !== undefined)!; + const placed = this.placedApps(ctx); + const workers = placed.filter((a) => a.port === undefined); + const target = ctx.deployTarget; + const blueGreen = this.app.zeroDowntime && target !== undefined; + + await this.writeWattConfigs(ctx, web); + await this.relinkPackages(ctx, pkgManager, cdPath, mise); + const ensured = await ctx.executor.exec( + `${mise} && ${wattEnsureInstalledCommand(this.wattWebRoot, watt, pkgManager)}`, + ); + if (ensured.exitCode !== 0) { + throw new DeployError((ensured.stderr || ensured.stdout).trim() || 'Installing wattpm failed', 'start'); + } + await ctx.executor.execOrThrow(wattInstalledGuard(this.wattWebRoot, watt)); + + // Web: the colour being booted under blue-green, or the single recreate unit. + const color = blueGreen ? target.color : undefined; + const webUnit = await this.provisionWattProcess(ctx, web, pkgManager, color, blueGreen ? target.port : undefined); + const workerUnits: string[] = []; + for (const worker of workers) { + workerUnits.push(await this.provisionWattProcess(ctx, worker, pkgManager)); + } + + const port = blueGreen ? target.port : web.port!; + // Reap a stale same-colour instance, then guard the port against a foreign + // process — but never the unit we are about to (re)start in recreate mode. + const reap = blueGreen ? `${stopUnitCommand(webUnit)} && ` : ''; + // The idle colour may still be a resident PM2 process (previous release of + // an app adopting watt); it is not serving, so retire it before binding its port. + const retirePm2 = blueGreen + ? `${mise} && ${pm2DeleteCommand(coloredWebName(namespace, web.name, target.color))} && ` + : ''; + await ctx.executor.execOrThrow( + reap + + retirePm2 + + (blueGreen ? `${portFreeGuard(port)} && ` : '') + + restartUnitCommand(webUnit), + ); + + // Recreate mode restarts workers with the web app. Blue-green holds them + // until the new colour is healthy (afterHealthy). + if (!blueGreen) { + for (const unit of workerUnits) await ctx.executor.execOrThrow(restartUnitCommand(unit)); + // Adopting watt on a host that ran this app under PM2: retire the old + // process so it stops holding the port. A no-op when PM2 was never used. + await ctx.executor.execOrThrow( + `${mise} && { mise exec -- pm2 delete "${namespace}" 2>/dev/null || true; }`, + ); + } + } + + /** After the Caddy flip: stop what is no longer serving, per retention. */ + private async reapPreviousWattColor(ctx: StrategyContext): Promise { + const target = ctx.deployTarget!; + const namespace = this.app.pm2!.apps[0].name; + const web = this.app.pm2!.apps.find((a) => a.port !== undefined); + if (!web) return; + + if (target.previousColor === null) { + // First blue-green deploy: retire the pre-blue-green process — an + // uncoloured watt unit, or a PM2 process when migrating from PM2. + const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; + await ctx.executor.execOrThrow(removeUnitCommand(wattUnitName(namespace, web.name))); + await ctx.executor.execOrThrow(`${mise} && { mise exec -- pm2 delete "${namespace}" 2>/dev/null || true; }`); + return; + } + if (this.app.blueGreenRetention === 'none') { + await ctx.executor.execOrThrow(stopUnitCommand(wattUnitName(namespace, web.name, target.previousColor))); + } + } + private async writeEcosystem(ctx: StrategyContext, writePath: string, content: string): Promise { const escaped = content.replace(/'/g, "'\"'\"'"); await ctx.executor.execOrThrow(`echo '${escaped}' > "${writePath}"`); diff --git a/src/domain/deploy/hot-sync.ts b/src/domain/deploy/hot-sync.ts index 9fbca30..0410c59 100644 --- a/src/domain/deploy/hot-sync.ts +++ b/src/domain/deploy/hot-sync.ts @@ -12,6 +12,7 @@ import { DeployError } from '../../shared/errors.js'; import { readDeployState, portFor } from './blue-green.js'; import { runWithDotenv } from './dotenv.js'; import { envSymlinkCommand } from './env-links.js'; +import { isWatt, resolveWattUnits, restartUnitCommand } from '../runtime/watt.js'; /** * The `deploy --watch` inner loop: patch the *live* release in place and reload. @@ -337,6 +338,18 @@ export class HotSync { private async reload(): Promise { if (!this.app.pm2) return false; + if (isWatt(this.app)) { + // Restart the serving colour's unit and the workers in place. Launchers + // and configs live in the live release, so a restart re-reads them. + const units = await resolveWattUnits(this.executor, this.appPath, this.app, { colors: 'active' }); + try { + for (const unit of units) await this.executor.execOrThrow(restartUnitCommand(unit)); + } catch (error) { + throw new DeployError(`Reload failed: ${error instanceof Error ? error.message : String(error)}`, 'start'); + } + return true; + } + const files = this.app.zeroDowntime ? [`${this.liveDir}/ecosystem.web.config.cjs`, `${this.liveDir}/ecosystem.workers.config.cjs`] : [`${this.liveDir}/ecosystem.config.cjs`]; diff --git a/src/domain/observe/collector.ts b/src/domain/observe/collector.ts new file mode 100644 index 0000000..eef24ed --- /dev/null +++ b/src/domain/observe/collector.ts @@ -0,0 +1,123 @@ +import type { RemoteExecutor } from '../remote/executor.js'; +import type { ShipnodeApp, ShipnodeConfig } from '../../shared/types.js'; +import { + parseAccessoryStatus, + parseCaddyInfo, + parseDeployLock, + parseHealthProbe, + parsePm2Jlist, + parseSystemdUnits, + parseReleaseRecords, + parseSystemStats, + splitSections, +} from './parse.js'; +import { appSectionName, buildObserveScript, PM2_FAILED } from './script.js'; +import { isWatt, wattUnitName } from '../runtime/watt.js'; +import type { AppSnapshot, ServerSnapshot } from './snapshot.js'; + +export interface CollectRequest { + apps: ShipnodeApp[]; + /** Poll interval in seconds; bounds both the SSH timeout and the health probe. */ + intervalSeconds?: number; + /** Accessory names to sample this poll; omit to skip the (heavier) accessories section. */ + accessoryNames?: string[]; +} + +/** + * One server's observer. + * + * Stateless by design: it holds no history, no timers, and no notion of a + * previous tick. Scheduling and memory belong to the session above it, which + * is what lets the same collector serve a live dashboard, a one-shot + * `--json`, and `status` without any of them knowing about the others. + */ +export class MetricsCollector { + constructor( + private readonly executor: RemoteExecutor, + private readonly server: string, + private readonly config: ShipnodeConfig, + ) {} + + get serverName(): string { + return this.server; + } + + async collect(request: CollectRequest): Promise { + const timestamp = new Date().toISOString(); + const intervalSeconds = request.intervalSeconds ?? 2; + const command = buildObserveScript(this.config, { + apps: request.apps, + healthMaxTimeSeconds: intervalSeconds, + accessoryNames: request.accessoryNames, + }); + + let stdout: string; + try { + const result = await this.executor.exec(command, { timeout: intervalSeconds * 3000 }); + stdout = result.stdout; + } catch (cause: unknown) { + // An unreachable host is a state the fleet view has to render, not an + // exception to unwind — the same stance `rollFleet` takes on a replica + // that fails mid-roll. + return this.unreachable(timestamp, cause instanceof Error ? cause.message : String(cause)); + } + + const sections = splitSections(stdout); + if (sections.size === 0) { + return this.unreachable(timestamp, 'Observe poll returned no data'); + } + + const accessoriesSection = sections.get('accessories'); + return { + server: this.server, + timestamp, + system: parseSystemStats(sections.get('sys') ?? ''), + accessories: accessoriesSection === undefined ? undefined : parseAccessoryStatus(accessoriesSection), + deployLock: parseDeployLock(sections.get('lock') ?? ''), + apps: request.apps.map((app, index) => readApp(app, index, sections)), + }; + } + + private unreachable(timestamp: string, error: string): ServerSnapshot { + return { + server: this.server, + timestamp, + system: parseSystemStats(''), + deployLock: null, + apps: [], + error, + }; + } +} + +function readApp(app: ShipnodeApp, index: number, sections: Map): AppSnapshot { + const at = (section: string): string | undefined => sections.get(appSectionName(index, section)); + const pm2Section = at('pm2') ?? ''; + const currentRaw = (at('current') ?? 'none').trim(); + const healthSection = at('health'); + const namespace = app.pm2?.apps[0]?.name ?? ''; + + return { + app: app.name, + appType: app.appType, + processes: isWatt(app) + ? parseSystemdUnits(at('units') ?? '', namespace, wattWeb(app, namespace)) + : parsePm2Jlist(pm2Section, namespace), + currentRelease: currentRaw === 'none' || currentRaw === '' ? null : currentRaw, + releases: parseReleaseRecords(at('releases') ?? ''), + health: healthSection === undefined ? undefined : parseHealthProbe(healthSection) ?? undefined, + caddy: + app.appType === 'frontend' + ? parseCaddyInfo(at('caddy-status') ?? '', at('caddy-log') ?? '') + : undefined, + error: + app.appType === 'backend' && app.pm2 && !isWatt(app) && pm2Section.includes(PM2_FAILED) + ? 'PM2 command failed' + : undefined, + }; +} + +function wattWeb(app: ShipnodeApp, namespace: string): { unitBase: string; instances: number } | undefined { + const web = app.pm2?.apps.find((p) => p.port !== undefined); + return web === undefined ? undefined : { unitBase: wattUnitName(namespace, web.name), instances: web.instances ?? 1 }; +} diff --git a/src/domain/observe/parse.ts b/src/domain/observe/parse.ts new file mode 100644 index 0000000..63d3cb2 --- /dev/null +++ b/src/domain/observe/parse.ts @@ -0,0 +1,344 @@ +/** + * Parsers for the sections of one observe script's output. + * + * Every parser is total: malformed or missing input yields an empty/neutral + * value rather than throwing, so one failing probe on the remote host cannot + * blank out the rest of a snapshot. + */ +import type { + AccessoryInfo, + CaddyInfo, + CaddyRequest, + DeployLockInfo, + HealthInfo, + ProcessInfo, + ReleaseRecord, + SystemInfo, +} from './types.js'; + +function readRecord(value: unknown): Record | null { + if (typeof value !== 'object' || value === null || Array.isArray(value)) return null; + // SAFETY: Runtime checks above establish a non-null plain object shape for indexed boundary parsing. + return value as Record; +} + +function readNumber(value: unknown): number | undefined { + return typeof value === 'number' ? value : undefined; +} + +function readString(value: unknown): string | undefined { + return typeof value === 'string' ? value : undefined; +} + +export function parsePm2Jlist(stdout: string, namespace: string): ProcessInfo[] { + try { + const raw: unknown = JSON.parse(stdout.trim()); + if (!Array.isArray(raw)) return []; + return raw.flatMap((entry) => parsePm2Entry(entry, namespace)); + } catch { + return []; + } +} + +function parsePm2Entry(entry: unknown, namespace: string): ProcessInfo[] { + const item = readRecord(entry); + if (item === null) return []; + const name = readString(item.name); + if (name === undefined) return []; + if (!belongsToNamespace(name, namespace)) return []; + + const pm2Env = readRecord(item.pm2_env); + const monit = readRecord(item.monit); + const memory = readNumber(monit?.memory); + + return [{ + name: resolveShortName(name, namespace), + pm2Name: name, + pid: readNumber(item.pid) ?? null, + status: readString(pm2Env?.status) ?? 'unknown', + cpu: readNumber(monit?.cpu) ?? 0, + memory: memory === undefined ? 0 : Math.round(memory / (1024 * 1024)), + uptime: readNumber(pm2Env?.pm_uptime) ?? 0, + restarts: readNumber(pm2Env?.restart_time) ?? 0, + execMode: parseExecMode(readString(pm2Env?.exec_mode)), + instances: readNumber(pm2Env?.instances) ?? 1, + unstableRestarts: readNumber(pm2Env?.unstable_restarts) ?? 0, + nodeVersion: readString(pm2Env?.node_version), + exitCode: readNumber(pm2Env?.exit_code) ?? null, + }]; +} + +function parseExecMode(raw: string | undefined): ProcessInfo['execMode'] { + if (raw === 'cluster_mode') return 'cluster'; + if (raw === 'fork_mode') return 'fork'; + return 'unknown'; +} + +function belongsToNamespace(pm2Name: string, namespace: string): boolean { + if (namespace === '') return true; + return pm2Name === namespace || pm2Name.startsWith(`${namespace}-`); +} + +const UNIT_PREFIX = 'shipnode-'; +const NS_UNSET = 18446744073709551615n; + +function bigOrNull(raw: string | undefined): bigint | null { + if (raw === undefined || !/^\d+$/.test(raw.trim())) return null; + return BigInt(raw.trim()); +} + +/** Map systemd's ActiveState onto the PM2 vocabulary the dashboard already colours. */ +function unitStatus(active: string, sub: string): string { + if (active === 'active') return 'online'; + if (active === 'failed') return 'errored'; + if (active === 'inactive') return 'stopped'; + if (active === 'activating' && sub === 'auto-restart') return 'errored'; + if (active === 'deactivating') return 'stopping'; + return active === '' ? 'unknown' : active; +} + +/** + * Parse the watt `units` observe section. `web` identifies the wattpm unit(s) + * (any colour) so they can be labelled as threads; every other unit is a + * plain worker process. + */ +export function parseSystemdUnits( + stdout: string, + namespace: string, + web?: { unitBase: string; instances: number }, +): ProcessInfo[] { + const props = new Map>(); + const cpuAfter = new Map(); + let wallNs = 0n; + let current: Record | null = null; + + for (const line of stdout.split('\n')) { + if (line.startsWith('@unit ')) { + current = {}; + props.set(line.slice(6).trim(), current); + } else if (line.startsWith('@wall ')) { + wallNs = bigOrNull(line.slice(6)) ?? 0n; + current = null; + } else if (line.startsWith('@cpu ')) { + const [unit, value] = line.slice(5).trim().split(/\s+/); + const parsed = bigOrNull(value); + if (unit !== undefined && parsed !== null) cpuAfter.set(unit, parsed); + } else if (current !== null) { + const eq = line.indexOf('='); + if (eq > 0) current[line.slice(0, eq)] = line.slice(eq + 1).trim(); + } + } + + const result: ProcessInfo[] = []; + for (const [unit, p] of props) { + if (p.LoadState !== 'loaded') continue; + const before = bigOrNull(p.CPUUsageNSec); + const after = cpuAfter.get(unit); + const cpu = + before !== null && before !== NS_UNSET && after !== undefined && after >= before && wallNs > 0n + ? Math.round(Number(((after - before) * 1000n) / wallNs)) / 10 + : 0; + const mem = bigOrNull(p.MemoryCurrent); + const pid = Number(p.MainPID); + const started = Number(p.started); + const isWeb = web !== undefined && (unit === web.unitBase || unit === `${web.unitBase}-blue` || unit === `${web.unitBase}-green`); + const exit = Number(p.ExecMainStatus); + + result.push({ + name: resolveShortName(unit.startsWith(UNIT_PREFIX) ? unit.slice(UNIT_PREFIX.length) : unit, namespace), + pm2Name: unit, + supervisor: 'systemd', + pid: Number.isFinite(pid) && pid > 0 ? pid : null, + status: unitStatus(p.ActiveState ?? '', p.SubState ?? ''), + cpu, + memory: mem === null || mem === NS_UNSET ? 0 : Math.round(Number(mem) / (1024 * 1024)), + uptime: Number.isFinite(started) && started > 0 && p.ActiveState === 'active' ? started * 1000 : 0, + restarts: Number(p.NRestarts) || 0, + execMode: isWeb ? 'threads' : 'fork', + instances: isWeb ? web.instances : 1, + unstableRestarts: 0, + exitCode: Number.isFinite(exit) ? exit : null, + }); + } + return result; +} + +export function parseSystemStats(stdout: string): SystemInfo { + const lines = stdout.trim().split('\n').flatMap((line) => line ? [line] : []); + const result: SystemInfo = { + load1: 0, + load5: 0, + load15: 0, + cores: 1, + totalMem: 0, + usedMem: 0, + totalDisk: 0, + usedDisk: 0, + uptime: 0, + }; + + for (const line of lines) { + if (line.startsWith('mem:')) { + const parts = line.split(/[:\s]+/); + result.totalMem = parseInt(parts[1] ?? '0', 10) || 0; + result.usedMem = parseInt(parts[2] ?? '0', 10) || 0; + } else if (line.startsWith('load:')) { + const parts = line.split(/[:\s]+/); + result.load1 = parseFloat(parts[1] ?? '0') || 0; + result.load5 = parseFloat(parts[2] ?? '0') || 0; + result.load15 = parseFloat(parts[3] ?? '0') || 0; + } else if (line.startsWith('cores:')) { + result.cores = parseInt(line.split(/[:\s]+/)[1] ?? '1', 10) || 1; + } else if (line.startsWith('uptime:')) { + result.uptime = parseInt(line.split(/[:\s]+/)[1] ?? '0', 10) || 0; + } else if (line.startsWith('disk:')) { + const parts = line.split(/[:\s]+/); + result.totalDisk = parseInt(parts[1] ?? '0', 10) || 0; + result.usedDisk = parseInt(parts[2] ?? '0', 10) || 0; + } + } + + return result; +} + +const SECTION_MARKER = /^@@SHIPNODE:([a-z0-9-]+)@@$/; + +/** + * Split the combined monitor command output into named sections. Missing or + * empty sections simply do not appear in the map — callers treat absence as + * "no data", never as an error. + */ +export function splitSections(stdout: string): Map { + const sections = new Map(); + let current: string | null = null; + let buffer: string[] = []; + + const flush = (): void => { + if (current !== null) sections.set(current, buffer.join('\n').trim()); + }; + + for (const line of stdout.split('\n')) { + const match = SECTION_MARKER.exec(line.trim()); + if (match !== null) { + flush(); + current = match[1]; + buffer = []; + } else if (current !== null) { + buffer.push(line); + } + } + flush(); + return sections; +} + +export function parseDeployLock(section: string): DeployLockInfo | null { + const trimmed = section.trim(); + if (trimmed === '' || trimmed === 'none') return null; + + const parts = trimmed.split(/\s+/); + const age = parseInt(parts[parts.length - 1] ?? '', 10); + if (Number.isNaN(age)) return null; + + return { + lockedAt: parts.slice(0, -1).join(' '), + ageSeconds: Math.max(age, 0), + }; +} + +export function parseHealthProbe(section: string): HealthInfo | null { + const trimmed = section.trim(); + if (trimmed === '') return null; + + const parts = trimmed.split(/\s+/); + const httpCode = parseInt(parts[0] ?? '', 10); + const responseMs = parseInt(parts[1] ?? '', 10); + if (Number.isNaN(httpCode) || Number.isNaN(responseMs)) return null; + + return { + status: httpCode >= 200 && httpCode < 400 ? 'ok' : 'fail', + httpCode, + responseMs, + }; +} + +export function parseAccessoryStatus(section: string): AccessoryInfo[] { + return section.split('\n').flatMap((line) => { + const trimmed = line.trim(); + if (trimmed === '') return []; + + const parts = trimmed.split('|'); + if (parts.length < 4) return []; + + const [rawName, status, health, image] = parts; + return [{ + name: rawName.replace(/^\/?(?:shipnode-)?/, ''), + status, + health: health === '' || health === '' ? '-' : health, + image, + }]; + }); +} + +export function parseCaddyInfo(statusSection: string, logSection: string): CaddyInfo { + const info: CaddyInfo = { + serviceActive: statusSection.trim() === 'active', + total: 0, + ok2xx: 0, + err4xx: 0, + err5xx: 0, + recent: [], + }; + + for (const line of logSection.split('\n')) { + const request = parseCaddyLogLine(line); + if (request === null) continue; + info.total += 1; + if (request.status >= 500) info.err5xx += 1; + else if (request.status >= 400) info.err4xx += 1; + else info.ok2xx += 1; + info.recent.push(request); + } + + info.recent = info.recent.slice(-5); + return info; +} + +function parseCaddyLogLine(line: string): CaddyRequest | null { + const trimmed = line.trim(); + if (!trimmed.startsWith('{')) return null; + + try { + const raw: unknown = JSON.parse(trimmed); + const record = readRecord(raw); + if (record === null) return null; + + const status = readNumber(record.status); + if (status === undefined) return null; + + const request = readRecord(record.request); + const duration = readNumber(record.duration); + return { + status, + method: readString(request?.method) ?? '', + uri: readString(request?.uri) ?? '', + ms: duration === undefined ? 0 : Math.round(duration * 1000), + }; + } catch { + return null; + } +} + +export function parseReleaseRecords(stdout: string): ReleaseRecord[] { + try { + const records: ReleaseRecord[] = JSON.parse(stdout.trim()); + return records.reverse().slice(0, 10); + } catch { + return []; + } +} + +function resolveShortName(pm2Name: string, namespace: string): string { + if (pm2Name === namespace) return pm2Name; + if (pm2Name.startsWith(`${namespace}-`)) return pm2Name.slice(namespace.length + 1); + return pm2Name; +} diff --git a/src/domain/observe/pivot.ts b/src/domain/observe/pivot.ts new file mode 100644 index 0000000..a13ba7a --- /dev/null +++ b/src/domain/observe/pivot.ts @@ -0,0 +1,59 @@ +import { assessConvergence, type ReplicaObservation } from '../deploy/convergence.js'; +import { releaseNameOf, type AppSnapshot, type FleetView, type ReplicaView, type ServerSnapshot } from './snapshot.js'; + +/** + * Turn server-shaped observations into app-shaped views. + * + * Collection is server-shaped — one round trip per host — but release skew + * lives *between* hosts, so it is invisible until the snapshots are pivoted. + * This is a pure function over already-collected data: no I/O, no config + * lookups, so `status`, the live dashboard, and `--json` can all derive the + * fleet picture from the same tick without polling again. + */ +export function pivotByApp(snapshots: ServerSnapshot[]): FleetView[] { + const byApp = new Map(); + const order: string[] = []; + + for (const server of snapshots) { + for (const app of server.apps) { + const replicas = byApp.get(app.app); + if (replicas === undefined) { + byApp.set(app.app, [toReplica(server, app)]); + order.push(app.app); + } else { + replicas.push(toReplica(server, app)); + } + } + } + + // A server whose poll failed reports no apps at all, so it cannot say which + // apps it was meant to be running. It is attributed to every app another + // replica proves exists — enough to stop a partial observation from reading + // as a converged fleet, without inventing apps for a host we never reached. + const unreachable = snapshots.flatMap((server) => (server.error === undefined ? [] : [server.server])); + + return order.map((appName) => { + const replicas = byApp.get(appName) ?? []; + const observations: ReplicaObservation[] = replicas.map((replica) => ({ + server: replica.server, + release: releaseNameOf(replica.snapshot), + })); + + return { + app: appName, + appType: replicas[0]?.snapshot.appType ?? 'backend', + replicas, + convergence: assessConvergence(observations), + unreachable, + }; + }); +} + +function toReplica(server: ServerSnapshot, snapshot: AppSnapshot): ReplicaView { + return { + server: server.server, + snapshot, + system: server.system, + reachable: server.error === undefined, + }; +} diff --git a/src/domain/observe/script.ts b/src/domain/observe/script.ts new file mode 100644 index 0000000..f77f26e --- /dev/null +++ b/src/domain/observe/script.ts @@ -0,0 +1,171 @@ +import type { ShipnodeApp, ShipnodeConfig } from '../../shared/types.js'; +import { isWatt, wattUnitName } from '../runtime/watt.js'; + +export const MISE = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`; +export const PM2_FAILED = '##SHIPNODE_PM2_FAILED##'; + +export function shellQuote(value: string): string { + return `'${value.replace(/'/g, "'\"'\"'")}'`; +} + +/** + * Sections are addressed by the app's *index* in the request, not its name. + * App names are user-supplied and would have to be escaped into the marker + * grammar; an index needs no escaping and the caller already holds the array + * that resolves it. + */ +export function appSectionName(index: number, section: string): string { + return `a${index}-${section}`; +} + +function sectionMarker(name: string): string { + return `echo "@@SHIPNODE:${name}@@"`; +} + +function buildSystemSection(): string { + return [ + `echo "mem:$(free -m | awk '/^Mem:/{print $2, $3}')"`, + `echo "load:$(awk '{print $1, $2, $3}' /proc/loadavg)"`, + `echo "cores:$(nproc 2>/dev/null || echo 1)"`, + `echo "uptime:$(cat /proc/uptime | awk '{print $1}')"`, + `echo "disk:$(df -BG --output=size,used / 2>/dev/null | tail -1 | awk '{print $1+0, $2+0}')"`, + ].join('; '); +} + +function buildLockSection(lockPath: string): string { + // Directory lock (atomic mkdir) stores the timestamp in acquired; + // legacy file locks store it as the file contents. + return ( + `if [ -d "${lockPath}" ]; then ` + + `echo "$(cat "${lockPath}/acquired" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` + + `elif [ -f "${lockPath}" ]; then ` + + `echo "$(cat "${lockPath}" 2>/dev/null) $(( $(date +%s) - $(stat -c %Y "${lockPath}" 2>/dev/null || date +%s) ))"; ` + + `else echo "none"; fi` + ); +} + +function buildHealthSection(port: number, path: string, maxTimeSeconds: number): string { + const url = `http://localhost:${port}${path}`; + return ( + `start=$(date +%s%N); ` + + `code=$(curl -s -o /dev/null -w "%{http_code}" --max-time ${maxTimeSeconds} "${url}" 2>/dev/null); ` + + `end=$(date +%s%N); ` + + `echo "$code $(( (end - start) / 1000000 ))"` + ); +} + +/** + * Every unit that could exist for a watt app: the web app in each colour plus + * the uncoloured form (recreate mode, or pre-blue-green), and each worker. + * Units that do not exist report `LoadState=not-found` and are dropped by the + * parser, so the script needs no state file lookup. + */ +export function wattCandidateUnits(app: ShipnodeApp): string[] { + const procs = app.pm2?.apps ?? []; + const namespace = procs[0]?.name ?? ''; + return procs.flatMap((proc) => + proc.port === undefined + ? [wattUnitName(namespace, proc.name)] + : [ + wattUnitName(namespace, proc.name), + wattUnitName(namespace, proc.name, 'blue'), + wattUnitName(namespace, proc.name, 'green'), + ], + ); +} + +/** + * systemd's answer to `pm2 jlist`. CPU is a rate, and systemd only exposes a + * cumulative counter, so the section samples it twice ~200ms apart within the + * one script — a stateless collector cannot diff against a previous tick. + */ +function buildUnitsSection(units: string[]): string { + const list = units.map(shellQuote).join(' '); + const props = '-p LoadState -p ActiveState -p SubState -p MainPID -p MemoryCurrent -p NRestarts -p ExecMainStatus -p CPUUsageNSec'; + return ( + `t0=$(date +%s%N); ` + + `for u in ${list}; do echo "@unit $u"; systemctl show "$u" ${props} 2>/dev/null; ` + + `s=$(systemctl show "$u" -p ActiveEnterTimestamp --value 2>/dev/null); echo "started=$(date -d "$s" +%s 2>/dev/null)"; done; ` + + `sleep 0.2; t1=$(date +%s%N); echo "@wall $((t1-t0))"; ` + + `for u in ${list}; do echo "@cpu $u $(systemctl show "$u" -p CPUUsageNSec --value 2>/dev/null)"; done` + ); +} + +function buildAccessoriesSection(names: string[]): string { + const containers = names.map((name) => shellQuote(`shipnode-${name}`)).join(' '); + // `sudo -n` fails instantly without a NOPASSWD rule instead of hanging the poll. + return ( + `sudo -n docker inspect ` + + `--format '{{.Name}}|{{.State.Status}}|{{.State.Health.Status}}|{{.Config.Image}}' ` + + `${containers} 2>/dev/null || true` + ); +} + +export interface ObserveRequest { + /** Apps to sample on this server, in the order their sections are addressed. */ + apps: ShipnodeApp[]; + /** Upper bound in seconds for each HTTP health probe, so it never stretches a poll. */ + healthMaxTimeSeconds?: number; + /** Accessory names (without the `shipnode-` prefix) to sample docker state for; omit to skip. */ + accessoryNames?: string[]; +} + +/** + * One shell script covering every metric a poll needs from one server, so a + * tick costs a single SSH round trip per host regardless of how many apps live + * there. Sections are delimited by `@@SHIPNODE:@@` marker lines and + * joined with `;` — each section carries its own fallback so one failing probe + * cannot blank out the rest of the snapshot. + * + * Host-level sections (system, deploy lock, accessories) are emitted once, not + * once per app: they describe the box, and repeating them per app would both + * waste the round trip and invite three different answers to the same question. + */ +export function buildObserveScript(config: ShipnodeConfig, request: ObserveRequest): string { + const lockFile = `${config.remotePath}/.shipnode/deploy.lock`; + const sections: Array<[string, string]> = [ + ['sys', buildSystemSection()], + ['lock', buildLockSection(lockFile)], + ]; + + if (request.accessoryNames !== undefined && request.accessoryNames.length > 0) { + sections.push(['accessories', buildAccessoriesSection(request.accessoryNames)]); + } + + request.apps.forEach((app, index) => { + const appPath = `${config.remotePath}/${app.name}`; + const named = (section: string): string => appSectionName(index, section); + + if (app.appType === 'backend' && app.pm2 && isWatt(app)) { + sections.push([named('units'), buildUnitsSection(wattCandidateUnits(app))]); + } else { + sections.push([ + named('pm2'), + app.appType === 'backend' && app.pm2 ? `pm2 jlist 2>/dev/null || echo "${PM2_FAILED}"` : `echo "[]"`, + ]); + } + sections.push([named('current'), `readlink "${appPath}/current" 2>/dev/null || echo "none"`]); + sections.push([named('releases'), `cat "${appPath}/.shipnode/releases.json" 2>/dev/null || echo "[]"`]); + + const webPort = app.pm2?.apps.find((pm2App) => pm2App.port !== undefined)?.port; + if (app.healthCheck.enabled && webPort !== undefined) { + const maxTime = Math.max( + 1, + Math.min(app.healthCheck.timeout, request.healthMaxTimeSeconds ?? app.healthCheck.timeout), + ); + sections.push([named('health'), buildHealthSection(webPort, app.healthCheck.path, maxTime)]); + } + + if (app.appType === 'frontend') { + const logFile = `/var/log/caddy/${app.name}.log`; + sections.push([named('caddy-status'), `systemctl is-active caddy 2>/dev/null || echo "unknown"`]); + sections.push([ + named('caddy-log'), + `sudo -n tail -n 50 "${logFile}" 2>/dev/null || tail -n 50 "${logFile}" 2>/dev/null || true`, + ]); + } + }); + + const script = sections.map(([name, command]) => `${sectionMarker(name)}; ${command}`); + return [MISE, ...script].join('; '); +} diff --git a/src/domain/observe/snapshot.ts b/src/domain/observe/snapshot.ts new file mode 100644 index 0000000..84fc17a --- /dev/null +++ b/src/domain/observe/snapshot.ts @@ -0,0 +1,75 @@ +import type { + AccessoryInfo, + CaddyInfo, + DeployLockInfo, + HealthInfo, + ProcessInfo, + ReleaseRecord, + SystemInfo, +} from './types.js'; +import type { FleetConvergence } from '../deploy/convergence.js'; + +/** + * What one poll of one server established. + * + * Collection is server-shaped because that is the shape of the round trip: one + * SSH connection, one script, one set of host-level facts. An app-shaped + * snapshot would report the same load average once per app on the box. + */ +export interface ServerSnapshot { + /** The host string — the server's identity everywhere, per ADR-0008. */ + server: string; + timestamp: string; + system: SystemInfo; + /** Absent on polls that skipped the (heavier) accessories section — carry the previous value forward. */ + accessories?: AccessoryInfo[]; + /** Server-scoped: the lock lives at `{remotePath}/.shipnode/deploy.lock`, not under an app. */ + deployLock: DeployLockInfo | null; + apps: AppSnapshot[]; + /** Whole-server failure — unreachable, timed out, script produced nothing. */ + error?: string; +} + +/** The app-scoped slice of one server's poll. */ +export interface AppSnapshot { + app: string; + appType: 'backend' | 'frontend'; + processes: ProcessInfo[]; + currentRelease: string | null; + releases: ReleaseRecord[]; + /** Absent when the app has no enabled HTTP health check or the probe produced no data. */ + health?: HealthInfo; + /** Present only for frontend apps served by Caddy. */ + caddy?: CaddyInfo; + /** Per-app failure — PM2 down for this namespace — leaving the rest of the server readable. */ + error?: string; +} + +/** One app seen across every server it runs on. Derived, never collected. */ +export interface FleetView { + app: string; + appType: 'backend' | 'frontend'; + replicas: ReplicaView[]; + /** + * Convergence over the replicas that actually answered. A partial + * observation cannot prove convergence, so `unreachable` must be read + * alongside it — the same rule `status` already applies when `--on` + * narrows a run to one server. + */ + convergence: FleetConvergence; + /** Servers that should be running this app but could not be reached this tick. */ + unreachable: string[]; +} + +export interface ReplicaView { + server: string; + snapshot: AppSnapshot; + system: SystemInfo; + /** False when the server-level poll failed; `snapshot` is then a placeholder. */ + reachable: boolean; +} + +/** The release directory name, which is what is comparable between replicas. */ +export function releaseNameOf(snapshot: AppSnapshot): string | null { + return snapshot.currentRelease?.split('/').pop() ?? null; +} diff --git a/src/domain/observe/types.ts b/src/domain/observe/types.ts new file mode 100644 index 0000000..28082f2 --- /dev/null +++ b/src/domain/observe/types.ts @@ -0,0 +1,95 @@ +/** + * The vocabulary of a single observation. + * + * These are the facts a poll can establish about a host and the apps on it. + * They carry no scope of their own — `snapshot.ts` decides which belong to a + * server and which to an app. + */ + +export interface ProcessInfo { + name: string; + /** Supervisor-level name: the PM2 process name, or the systemd unit for watt. */ + pm2Name: string; + /** Absent means PM2. `systemd` marks a unit of the watt runtime. */ + supervisor?: 'systemd'; + pid: number | null; + status: string; + cpu: number; + memory: number; + uptime: number; + restarts: number; + execMode: 'cluster' | 'fork' | 'threads' | 'unknown'; + instances: number; + unstableRestarts: number; + nodeVersion?: string; + exitCode: number | null; +} + +export interface SystemInfo { + load1: number; + load5: number; + load15: number; + cores: number; + totalMem: number; + usedMem: number; + totalDisk: number; + usedDisk: number; + uptime: number; +} + +/** CPU utilisation as a 0–1 fraction: 1-minute load normalised by core count. */ +export function systemCpuPercent(system: SystemInfo): number { + const load = system.load1 / Math.max(system.cores, 1); + return Math.max(0, Math.min(1, load)); +} + +export interface DeployLockInfo { + lockedAt: string; + ageSeconds: number; +} + +export interface HealthInfo { + status: 'ok' | 'fail'; + httpCode: number; + responseMs: number; +} + +/** + * Consecutive-failure streak for the HTTP health probe. A poll without probe + * data (health check disabled or probe produced nothing) leaves the streak + * untouched — only a real `ok` result clears it. + */ +export function nextHealthFailStreak(previous: number, health: HealthInfo | undefined): number { + if (health === undefined) return previous; + return health.status === 'fail' ? previous + 1 : 0; +} + +export interface AccessoryInfo { + name: string; + status: string; + health: string; + image: string; +} + +export interface CaddyRequest { + status: number; + method: string; + uri: string; + ms: number; +} + +export interface CaddyInfo { + serviceActive: boolean; + total: number; + ok2xx: number; + err4xx: number; + err5xx: number; + recent: CaddyRequest[]; +} + +export interface ReleaseRecord { + timestamp: string; + status: string; + duration: number; + gitCommit?: string; +} diff --git a/src/domain/runtime/watt.ts b/src/domain/runtime/watt.ts new file mode 100644 index 0000000..cf7a8fd --- /dev/null +++ b/src/domain/runtime/watt.ts @@ -0,0 +1,282 @@ +import type { Pm2App, PkgManager, ShipnodeApp, WattConfig } from '../../shared/types.js'; +import { getPm2Name } from '../pm2/apps.js'; +import { runWithDotenv } from '../deploy/dotenv.js'; +import { readDeployState, otherColor, type DeployColor } from '../deploy/blue-green.js'; +import type { RemoteExecutor } from '../remote/executor.js'; + +/** + * The opt-in `watt` runtime (wattpm). + * + * The web app runs under wattpm: N worker threads in one process, each + * accepting straight from the kernel via SO_REUSEPORT — no supervisor in the + * request path. Every other declared process (workers) keeps its own OS process + * as a plain systemd unit, so it is supervised, restarted and logged the same + * way. wattpm supplies the threading; systemd supplies the supervision that + * PM2 would otherwise provide. + * + * Everything here is a pure renderer or command builder so it can be tested + * without a host. See docs/adr/0009-watt-runtime.md. + */ + +/** Per-release file names. They live next to the app so `current/` resolves them (ADR-0001). */ +export const WATT_RUNTIME_FILE = 'shipnode.watt.json'; +export const WATT_APP_FILE = 'shipnode.platformatic.json'; + +const DEFAULT_MODULE = '@platformatic/node'; + +/** + * The wattpm version shipnode's rendered configs target. It is the schema + * version below and the version installed when an app does not list wattpm + * itself, so the two cannot drift apart. + */ +export const WATT_VERSION = '3.71.0'; +const MISE_PATH = 'export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"'; + +export function isWatt(app: Pick): boolean { + return app.runtime === 'watt'; +} + +/** systemd unit name for a declared process, optionally suffixed with a blue-green colour. */ +export function wattUnitName(namespace: string, appName: string, color?: DeployColor): string { + return `shipnode-${getPm2Name(namespace, appName)}${color ? `-${color}` : ''}`; +} + +/** File name of the per-release launcher for a process, per colour when blue-green. */ +export function runScriptName(appName: string, color?: DeployColor): string { + return `shipnode-run-${appName}${color ? `-${color}` : ''}.sh`; +} + +/** `512M` / `1G` / `2048K` / `1048576` → bytes. Returns undefined for anything else. */ +export function parseSize(size: string | undefined): number | undefined { + if (!size) return undefined; + const match = /^(\d+)([KMG]?)$/i.exec(size.trim()); + if (!match) return undefined; + const unit = { '': 1, K: 1024, M: 1024 ** 2, G: 1024 ** 3 }[match[2].toUpperCase() as '' | 'K' | 'M' | 'G']; + return Number(match[1]) * unit; +} + +/** + * The wattpm runtime config for the web app. The port is `{PORT}` so one file + * serves both blue-green colours; each colour's launcher sets PORT. + */ +export function renderRuntimeConfig(web: Pm2App, watt: WattConfig): string { + const heap = parseSize(watt.maxHeapUsed ?? web.maxMemory); + const config = { + $schema: `https://schemas.platformatic.dev/wattpm/${WATT_VERSION}.json`, + entrypoint: 'web', + workers: { static: web.instances ?? 1 }, + server: { hostname: '0.0.0.0', port: '{PORT}' }, + applications: [{ id: 'web', path: '.', config: WATT_APP_FILE }], + restartOnError: true, + health: { enabled: true, ...(heap !== undefined ? { maxHeapUsed: heap } : {}) }, + }; + return `${JSON.stringify(config, null, 2)}\n`; +} + +/** The capability config that tells wattpm how to load the app inside each worker thread. */ +export function renderAppConfig(watt: WattConfig): string { + const config = { + $schema: `https://schemas.platformatic.dev/@platformatic/node/${WATT_VERSION}.json`, + module: watt.module ?? DEFAULT_MODULE, + node: { main: watt.main }, + }; + return `${JSON.stringify(config, null, 2)}\n`; +} + +export interface RunScriptInput { + /** Absolute directory the process starts from (the release's app root via `current`). */ + cwd: string; + /** Command line to exec, e.g. `./node_modules/.bin/wattpm start -c shipnode.watt.json`. */ + command: string; + envFile?: string; + env: Record; +} + +/** Quote one ExecStart argument: spaces stay in one word, and `\`, `"` and `%` are not interpreted by systemd. */ +function systemdQuote(value: string): string { + return `"${value.replace(/[\\"]/g, '\\$&').replace(/%/g, '%%')}"`; +} + +/** systemd reads a unit line by line: a CR/LF/NUL in a value would add directives or corrupt the unit. */ +function assertUnitSafe(label: string, value: string): void { + if (/[\r\n\0]/.test(value)) throw new Error(`Cannot render systemd unit: ${label} contains a control character`); +} + +/** + * Launcher script the systemd unit executes. A script file (rather than an + * inline ExecStart) sidesteps systemd's own quoting and `%`/`$` expansion, and + * keeps the dotenv handling identical to the PM2 path (ADR-0003): the env file + * is parsed as data, never sourced. + */ +export function renderRunScript(input: RunScriptInput): string { + const exec = `exec mise exec -- ${input.command}`; + const body = runWithDotenv(input.envFile, exec, input.env); + const exports = input.envFile + ? '' + : Object.entries(input.env).map(([k, v]) => `export ${k}='${String(v).replace(/'/g, `'"'"'`)}'\n`).join(''); + return `#!/usr/bin/env bash\nset -e\n${MISE_PATH}\ncd "${input.cwd}"\n${exports}${body}\n`; +} + +export interface UnitInput { + description: string; + user: string; + workingDirectory: string; + script: string; +} + +export function renderUnit(input: UnitInput): string { + assertUnitSafe('description', input.description); + assertUnitSafe('user', input.user); + assertUnitSafe('working directory', input.workingDirectory); + assertUnitSafe('script path', input.script); + return `[Unit] +Description=${input.description} +After=network.target + +[Service] +Type=simple +User=${input.user} +WorkingDirectory=${input.workingDirectory} +ExecStart=/usr/bin/env bash ${systemdQuote(input.script)} +Restart=always +RestartSec=2 +KillSignal=SIGTERM +TimeoutStopSec=30 +LimitNOFILE=65535 + +[Install] +WantedBy=multi-user.target +`; +} + +// --------------------------------------------------------------------------- +// Remote command builders. Each is a self-contained shell snippet: it elevates +// only when not already root, so it works for both `root` and a NOPASSWD deploy +// user (the identity `shipnode setup` provisions). +// --------------------------------------------------------------------------- + +const SUDO = 'S=$([ "$(id -u)" = 0 ] || echo sudo)'; + +function quote(value: string): string { + return `'${value.replace(/'/g, `'"'"'`)}'`; +} + +export function unitPath(unit: string): string { + return `/etc/systemd/system/${unit}.service`; +} + +/** Write the unit file, reload systemd and enable it at boot. Does not start it. */ +export function installUnitCommand(unit: string, content: string): string { + return `${SUDO}; printf '%s' ${quote(content)} | $S tee ${unitPath(unit)} >/dev/null && ` + + `$S systemctl daemon-reload && $S systemctl enable ${unit}`; +} + +/** Start the unit, or restart it onto the newly linked release. */ +export function restartUnitCommand(unit: string): string { + return `${SUDO}; $S systemctl restart ${unit}`; +} + +export function stopUnitCommand(unit: string): string { + return `${SUDO}; $S systemctl stop ${unit} 2>/dev/null || true`; +} + +/** Stop, disable and delete a unit; a missing unit is not an error. */ +export function removeUnitCommand(unit: string): string { + return `${SUDO}; $S systemctl disable --now ${unit} 2>/dev/null || true; ` + + `$S rm -f ${unitPath(unit)}; $S systemctl daemon-reload`; +} + +export function logsCommand(unit: string, opts: { follow?: boolean; lines?: number } = {}): string { + return `${SUDO}; $S journalctl -u ${unit} -n ${opts.lines ?? 100} --no-pager${opts.follow ? ' -f' : ''}`; +} + +/** Fails (non-zero) when a port is already bound by something else. */ +export function portFreeGuard(port: number): string { + return `{ if ss -tlnp | grep -q ":${port} "; then echo "Port ${port} is already in use by another process" >&2; false; else true; fi; }`; +} + +/** + * Best-effort removal of a PM2 process by exact name. Adopting watt on a host + * whose app ran blue-green under PM2 leaves the idle colour's PM2 process + * resident on the port the watt unit is about to bind. + */ +export function pm2DeleteCommand(name: string): string { + return `{ mise exec -- pm2 delete "${name}" 2>/dev/null || true; }`; +} + +/** `add` command per package manager, run in the app root of the release. */ +const ADD_COMMANDS: Record = { + npm: 'npm install --no-audit --no-fund', + pnpm: 'pnpm add', + yarn: 'yarn add', + bun: 'bun add', +}; + +/** + * Installs wattpm (and the default capability module) at the version shipnode + * targets when the release does not already have them. Packages the app lists + * itself are left alone, so an app that pins its own versions keeps them. Runs + * after the release's normal install and relink, so nothing prunes the result. + * A custom `module` is never auto-installed; the guard below reports it. + */ +export function wattEnsureInstalledCommand(cwd: string, watt: WattConfig, pkgManager: PkgManager): string { + const missing = [ + `[ -x "${cwd}/node_modules/.bin/wattpm" ] || pkgs="$pkgs wattpm@${WATT_VERSION}"`, + ...(watt.module === undefined + ? [`[ -d "${cwd}/node_modules/${DEFAULT_MODULE}" ] || pkgs="$pkgs ${DEFAULT_MODULE}@${WATT_VERSION}"`] + : []), + ]; + return `{ pkgs=""; ${missing.join('; ')}; ` + + `if [ -n "$pkgs" ]; then echo "wattpm not in dependencies, installing:$pkgs"; cd "${cwd}" && ${ADD_COMMANDS[pkgManager]} $pkgs; fi; }`; +} + +/** Fails with an actionable message when wattpm is still not installed in the release. */ +export function wattInstalledGuard(cwd: string, watt: WattConfig): string { + const module = watt.module ?? DEFAULT_MODULE; + return `{ [ -x "${cwd}/node_modules/.bin/wattpm" ] && [ -d "${cwd}/node_modules/${module}" ] || ` + + `{ echo "runtime 'watt' needs wattpm and ${module} in your dependencies: npm i wattpm ${module}" >&2; false; }; }`; +} + +/** The command line the web launcher execs. */ +export const WATT_START_COMMAND = `./node_modules/.bin/wattpm start -c ${WATT_RUNTIME_FILE}`; + +/** + * The systemd units a CLI verb should act on. + * + * Under blue-green the web unit carries a colour that only the host knows (it + * is persisted in deploy-state.json), so it is resolved by asking the host. + * `restart`/`logs` want the serving colour; `stop` wants every colour that may + * still be resident so nothing keeps holding the port. + */ +export async function resolveWattUnits( + executor: RemoteExecutor, + appPath: string, + app: ShipnodeApp, + opts: { process?: string; colors: 'active' | 'all' }, +): Promise { + const procs = app.pm2?.apps ?? []; + if (procs.length === 0) return []; + const namespace = procs[0].name; + const selected = opts.process ? procs.filter((p) => p.name === opts.process) : procs; + if (opts.process && selected.length === 0) { + throw new Error(`No process named '${opts.process}' in this deployment. Known: ${procs.map((p) => p.name).join(', ')}`); + } + + const state = app.zeroDowntime ? await readDeployState(executor, appPath) : null; + const units: string[] = []; + for (const proc of selected) { + if (proc.port === undefined || !state) { + units.push(wattUnitName(namespace, proc.name)); + } else if (opts.colors === 'active') { + units.push(wattUnitName(namespace, proc.name, state.activeColor)); + } else { + units.push(wattUnitName(namespace, proc.name, state.activeColor), wattUnitName(namespace, proc.name, otherColor(state.activeColor))); + } + } + return units; +} + +/** Exit 0 when the unit is active. */ +export function isActiveCommand(unit: string): string { + return `systemctl is-active --quiet ${unit}`; +} diff --git a/src/services/health.service.ts b/src/services/health.service.ts index f91f52b..a74706c 100644 --- a/src/services/health.service.ts +++ b/src/services/health.service.ts @@ -2,6 +2,7 @@ import type { ShipnodeConfig, ShipnodeApp, Pm2App } from '../shared/types.js'; import type { RemoteExecutor } from '../domain/remote/executor.js'; import { HealthCheckError } from '../shared/errors.js'; import { getPm2Name } from '../domain/pm2/apps.js'; +import { isWatt } from '../domain/runtime/watt.js'; /** Exponential retry pacing for the HTTP probe. */ export interface RetryBackoff { @@ -65,14 +66,24 @@ export class HealthCheckService { let responseMs = 0; if (webApp) { - const result = await this.performHttpCheck(webApp, app.healthCheck, opts?.httpPort, opts?.backoff); + const result = await this.performHttpCheck( + webApp, + app.healthCheck, + opts?.httpPort, + opts?.backoff, + isWatt(app) ? unitFor(app, webApp, opts?.resolvePm2Name) : undefined, + ); attempts = result.attempts; responseMs = result.responseMs; } const pm2Apps = opts?.pm2Apps ?? app.pm2?.apps; if (pm2Apps?.length) { - await this.performPm2StatusCheck(app, pm2Apps, opts?.resolvePm2Name); + if (isWatt(app)) { + await this.performUnitStatusCheck(app, pm2Apps, opts?.resolvePm2Name); + } else { + await this.performPm2StatusCheck(app, pm2Apps, opts?.resolvePm2Name); + } } return { attempts, responseMs }; @@ -83,6 +94,7 @@ export class HealthCheckService { healthCheck: ShipnodeApp['healthCheck'], portOverride?: number, backoff?: RetryBackoff, + unit?: string, ): Promise<{ attempts: number; responseMs: number }> { const { path, timeout, retries } = healthCheck; const port = portOverride ?? webApp.port; @@ -112,7 +124,7 @@ export class HealthCheckService { } } - const diagnostics = await this.collectPm2Logs(webApp.name); + const diagnostics = unit ? await this.collectUnitLogs(unit) : await this.collectPm2Logs(webApp.name); throw new HealthCheckError( `Health check failed after ${retries} attempts. Last status: ${lastStatus}` + diagnostics, retries, @@ -182,6 +194,53 @@ export class HealthCheckService { ); } + /** + * The watt runtime's counterpart to the PM2 status check: every required + * unit must be `active` and must not have been restarted by systemd since it + * was started (a crash loop under `Restart=always` otherwise looks healthy + * between crashes). + */ + private async performUnitStatusCheck( + app: ShipnodeApp, + procs: Pm2App[], + resolvePm2Name?: (a: Pm2App) => string, + ): Promise { + const failures: string[] = []; + const bad: string[] = []; + for (const proc of procs) { + const unit = unitFor(app, proc, resolvePm2Name); + const result = await this.executor.exec( + `systemctl show ${unit} -p ActiveState -p NRestarts --value 2>/dev/null | paste -sd' '`, + ); + const [state = 'unknown', restarts = '0'] = result.stdout.trim().split(/\s+/); + if (state !== 'active') { + failures.push(`${unit}: state=${state}`); + bad.push(unit); + } else if (Number(restarts) > 0) { + failures.push(`${unit}: crashed during startup (NRestarts=${restarts})`); + bad.push(unit); + } + } + + if (failures.length === 0) return; + + let diagnostics = ''; + for (const unit of bad) diagnostics += await this.collectUnitLogs(unit); + throw new HealthCheckError( + `systemd unit(s) failed health check:\n - ${failures.join('\n - ')}${diagnostics}`, + 0, + 0, + ); + } + + private async collectUnitLogs(unit: string): Promise { + const logResult = await this.executor.exec( + `{ S=$([ "$(id -u)" = 0 ] || echo sudo); $S journalctl -u ${unit} -n 15 --no-pager 2>/dev/null; } || true`, + ).catch(() => ({ stdout: '', stderr: '' })); + const logs = logResult.stdout.trim(); + return logs ? `\n\nsystemd logs (${unit}):\n${logs}` : ''; + } + private async collectPm2Logs(name: string): Promise { const logResult = await this.executor.exec( `{ tail -15 ~/.pm2/logs/${name}-error.log 2>/dev/null; tail -15 ~/.pm2/logs/${name}-out.log 2>/dev/null; } || true`, @@ -194,3 +253,9 @@ export class HealthCheckService { return new Promise((resolve) => setTimeout(resolve, ms)); } } + +/** systemd unit for a declared process, honouring blue-green's coloured web name. */ +function unitFor(app: ShipnodeApp, proc: Pm2App, resolvePm2Name?: (a: Pm2App) => string): string { + const namespace = app.pm2?.apps[0]?.name ?? ''; + return `shipnode-${(resolvePm2Name ?? ((a: Pm2App) => getPm2Name(namespace, a.name)))(proc)}`; +} diff --git a/src/services/observe/events.ts b/src/services/observe/events.ts new file mode 100644 index 0000000..4bd87c5 --- /dev/null +++ b/src/services/observe/events.ts @@ -0,0 +1,14 @@ +/** + * What a tick noticed, as values rather than rendered strings. + * + * The old monitor built `chalk.red(...)` strings inside its polling hook, + * which put a presentation decision in the data layer and left `--json` with + * nothing but ANSI escapes to emit. Colour and wording are chosen at render. + */ +export type ObserveEvent = + | { kind: 'health-failing'; at: string; server: string; app: string; streak: number } + | { kind: 'health-recovered'; at: string; server: string; app: string } + | { kind: 'server-unreachable'; at: string; server: string; message: string } + | { kind: 'server-recovered'; at: string; server: string } + | { kind: 'app-error'; at: string; server: string; app: string; message: string } + | { kind: 'notice'; at: string; message: string }; diff --git a/src/services/observe/history.ts b/src/services/observe/history.ts new file mode 100644 index 0000000..2936a00 --- /dev/null +++ b/src/services/observe/history.ts @@ -0,0 +1,37 @@ +import type { AppSnapshot } from '../../domain/observe/snapshot.js'; + +/** + * A rolling window of one app's numbers on one server, for sparklines. + * + * Averaged across processes rather than kept per-process: the window exists to + * show a trend at a glance, and a per-process series would need a legend to be + * readable at sparkline size. + */ +export class MetricsHistory { + cpu: number[] = []; + memory: number[] = []; + responseMs: number[] = []; + + constructor(private maxEntries: number = 60) {} + + push(snapshot: Pick): void { + const count = Math.max(snapshot.processes.length, 1); + this.cpu.push(snapshot.processes.reduce((sum, p) => sum + p.cpu, 0) / count); + this.memory.push(snapshot.processes.reduce((sum, p) => sum + p.memory, 0) / count); + if (snapshot.health !== undefined) this.responseMs.push(snapshot.health.responseMs); + + this.trim(this.cpu); + this.trim(this.memory); + this.trim(this.responseMs); + } + + private trim(series: number[]): void { + if (series.length > this.maxEntries) series.shift(); + } + + clear(): void { + this.cpu = []; + this.memory = []; + this.responseMs = []; + } +} diff --git a/src/services/observe/session.ts b/src/services/observe/session.ts new file mode 100644 index 0000000..521ed78 --- /dev/null +++ b/src/services/observe/session.ts @@ -0,0 +1,254 @@ +import type { ShipnodeApp } from '../../shared/types.js'; +import type { CollectRequest } from '../../domain/observe/collector.js'; +import { pivotByApp } from '../../domain/observe/pivot.js'; +import type { FleetView, ServerSnapshot } from '../../domain/observe/snapshot.js'; +import { nextHealthFailStreak, type HealthInfo } from '../../domain/observe/types.js'; +import type { ObserveEvent } from './events.js'; +import { MetricsHistory } from './history.js'; + +/** Sample docker accessory state roughly every this many seconds, not every tick. */ +export const ACCESSORY_SAMPLE_SECONDS = 10; + +/** Consecutive failed health probes before the session raises an alert. */ +export const HEALTH_ALERT_THRESHOLD = 3; + +/** SSH connections opened at once. A twelve-host fleet must not open twelve. */ +export const MAX_CONCURRENT_POLLS = 4; + +/** The collector seam, structurally satisfied by `MetricsCollector`. */ +export interface ServerObserver { + readonly serverName: string; + collect(request: CollectRequest): Promise; +} + +/** One server to poll, with what lives on it. Resolved by the caller from config. */ +export interface ObserveTarget { + observer: ServerObserver; + apps: ShipnodeApp[]; + accessoryNames: string[]; +} + +export interface ObserveState { + /** The collected truth, one entry per target that has reported, in target order. */ + servers: ServerSnapshot[]; + /** The same tick pivoted by app, where release skew becomes visible. */ + fleets: FleetView[]; + events: ObserveEvent[]; + lastUpdate: string | null; + polling: boolean; +} + +export interface ObserveSessionOptions { + targets: ObserveTarget[]; + intervalSeconds: number; + maxEvents?: number; + /** Injected for tests; defaults to the real clock. */ + now?: () => Date; +} + +type Subscriber = (state: ObserveState) => void; + +/** + * Everything stateful about watching a set of servers: scheduling, history, + * health streaks, and the event log. + * + * The collectors below it are stateless and the renderers above it are pure, + * so this is the only place that has to reason about "since last tick" - which + * is why the same session can drive a live dashboard, a one-shot snapshot, and + * `--json` without any of them re-polling. + */ +export class ObserveSession { + private readonly targets: ObserveTarget[]; + private readonly intervalSeconds: number; + private readonly maxEvents: number; + private readonly now: () => Date; + + private readonly histories = new Map(); + private readonly healthStreaks = new Map(); + private readonly unreachable = new Set(); + private readonly subscribers = new Set(); + + private snapshots = new Map(); + private events: ObserveEvent[] = []; + private lastUpdate: string | null = null; + private polling = false; + private tickCount = 0; + private timer: ReturnType | null = null; + + constructor(options: ObserveSessionOptions) { + this.targets = options.targets; + this.intervalSeconds = Math.max(1, options.intervalSeconds); + this.maxEvents = options.maxEvents ?? 100; + this.now = options.now ?? (() => new Date()); + } + + /** Sparkline window for one app on one server, created on first sight. */ + history(server: string, app: string): MetricsHistory { + const key = historyKey(server, app); + const existing = this.histories.get(key); + if (existing !== undefined) return existing; + const created = new MetricsHistory(); + this.histories.set(key, created); + return created; + } + + getState(): ObserveState { + const servers = this.targets.flatMap((target) => { + const snapshot = this.snapshots.get(target.observer.serverName); + return snapshot === undefined ? [] : [snapshot]; + }); + return { + servers, + fleets: pivotByApp(servers), + events: this.events, + lastUpdate: this.lastUpdate, + polling: this.polling, + }; + } + + subscribe(subscriber: Subscriber): () => void { + this.subscribers.add(subscriber); + return () => { + this.subscribers.delete(subscriber); + }; + } + + /** Record something the user did, so it lands in the same log as observations. */ + notice(message: string): void { + this.append({ kind: 'notice', at: this.now().toISOString(), message }); + this.publish(); + } + + start(): void { + if (this.timer !== null) return; + void this.tick(); + this.timer = setInterval(() => { + void this.tick(); + }, this.intervalSeconds * 1000); + } + + stop(): void { + if (this.timer === null) return; + clearInterval(this.timer); + this.timer = null; + } + + /** + * One pass over every target. + * + * A tick still running when the next is due is skipped rather than + * overlapped: a slow host would otherwise stack connections until the box + * refuses them, and a stale reading beats a queue. + */ + async tick(): Promise { + if (this.polling) return; + this.polling = true; + this.publish(); + + const sampleAccessories = this.tickCount % this.accessoryCadence() === 0; + this.tickCount += 1; + + try { + await this.forEachTarget(async (target) => { + const snapshot = await target.observer.collect({ + apps: target.apps, + intervalSeconds: this.intervalSeconds, + accessoryNames: + sampleAccessories && target.accessoryNames.length > 0 ? target.accessoryNames : undefined, + }); + this.absorb(snapshot); + }); + this.lastUpdate = this.now().toISOString(); + } finally { + this.polling = false; + this.publish(); + } + } + + private accessoryCadence(): number { + return Math.max(1, Math.ceil(ACCESSORY_SAMPLE_SECONDS / this.intervalSeconds)); + } + + /** Bounded-parallel fan-out: at most MAX_CONCURRENT_POLLS connections live at once. */ + private async forEachTarget(visit: (target: ObserveTarget) => Promise): Promise { + const queue = [...this.targets]; + const width = Math.min(MAX_CONCURRENT_POLLS, queue.length); + const workers = Array.from({ length: width }, async () => { + for (let target = queue.shift(); target !== undefined; target = queue.shift()) { + await visit(target); + } + }); + await Promise.all(workers); + } + + private absorb(snapshot: ServerSnapshot): void { + const at = this.now().toISOString(); + const server = snapshot.server; + + // A skipped accessories section means "not sampled this tick", not "none + // left" - carrying the previous value forward keeps the panel from blinking. + const previous = this.snapshots.get(server); + const merged = + snapshot.accessories === undefined && previous?.accessories !== undefined + ? { ...snapshot, accessories: previous.accessories } + : snapshot; + this.snapshots.set(server, merged); + + if (merged.error !== undefined) { + // Report the crossing, not every tick the host stays down. + if (!this.unreachable.has(server)) { + this.unreachable.add(server); + this.append({ kind: 'server-unreachable', at, server, message: merged.error }); + } + return; + } + if (this.unreachable.delete(server)) { + this.append({ kind: 'server-recovered', at, server }); + } + + for (const app of merged.apps) { + this.history(server, app.app).push(app); + if (app.error !== undefined) { + this.append({ kind: 'app-error', at, server, app: app.app, message: app.error }); + } + this.trackHealth(at, server, app.app, app.health); + } + } + + private trackHealth(at: string, server: string, app: string, health: HealthInfo | undefined): void { + const key = historyKey(server, app); + const previous = this.healthStreaks.get(key) ?? 0; + const streak = nextHealthFailStreak(previous, health); + this.healthStreaks.set(key, streak); + + if (streak === HEALTH_ALERT_THRESHOLD && previous < HEALTH_ALERT_THRESHOLD) { + this.append({ kind: 'health-failing', at, server, app, streak }); + } + if (streak === 0 && previous >= HEALTH_ALERT_THRESHOLD) { + this.append({ kind: 'health-recovered', at, server, app }); + } + } + + /** Consecutive failed probes for one app on one server. */ + healthFailStreak(server: string, app: string): number { + return this.healthStreaks.get(historyKey(server, app)) ?? 0; + } + + private append(event: ObserveEvent): void { + this.events = [...this.events.slice(-(this.maxEvents - 1)), event]; + } + + private publish(): void { + const state = this.getState(); + for (const subscriber of this.subscribers) subscriber(state); + } +} + +/** + * App names are free-form, so the two halves are joined on NUL - a printable + * separator would let ("a b", "c") and ("a", "b c") collide into one history + * window. + */ +function historyKey(server: string, app: string): string { + return `${server}${app}`; +} diff --git a/src/shared/types.ts b/src/shared/types.ts index 4efd9df..64d3c38 100644 --- a/src/shared/types.ts +++ b/src/shared/types.ts @@ -91,6 +91,24 @@ export interface Pm2Config { apps: Pm2App[]; } +/** Process supervisor for a backend app. `pm2` is the default. */ +export type AppRuntime = 'pm2' | 'watt'; + +/** + * Settings for the opt-in `watt` runtime (wattpm — worker threads sharing the + * web port via SO_REUSEPORT). The web app runs under wattpm; every other + * declared process (workers) runs as its own systemd unit. `instances` on the + * web app becomes the worker-thread count. + */ +export interface WattConfig { + /** Entry file the web app loads inside each worker thread, e.g. `dist/server.js`. It must listen on `process.env.PORT`. */ + main: string; + /** wattpm capability module. Defaults to `@platformatic/node`. */ + module?: string; + /** Recycle a worker whose heap exceeds this (`512M`, `1G`). Defaults to the web app's `maxMemory`. */ + maxHeapUsed?: string; +} + export interface DockerConfig { image?: string; dockerfile: string; @@ -235,6 +253,9 @@ export interface ShipnodeApp { append?: string; }; pm2?: Pm2Config; + /** Process supervisor. Defaults to `pm2`; `watt` is opt-in and needs `watt`. */ + runtime?: AppRuntime; + watt?: WattConfig; docker?: DockerConfig; healthCheck: HealthCheckConfig; envFile: string; diff --git a/tests/unit/hot-sync.test.ts b/tests/unit/hot-sync.test.ts index fe7c0ad..47a2f53 100644 --- a/tests/unit/hot-sync.test.ts +++ b/tests/unit/hot-sync.test.ts @@ -231,6 +231,30 @@ describe('HotSync — remote install and build', () => { }); }); +describe('HotSync — reload on the watt runtime', () => { + it('restarts the serving colour and workers via systemd, never PM2', async () => { + const config = makeConfig({ + pm2: { apps: [{ name: 'api', port: 3000 }, { name: 'mailer', command: 'node mailer.js' }] }, + runtime: 'watt', + watt: { main: 'dist/server.js' }, + zeroDowntime: true, + domain: 'example.com', + }); + const executor = healthyExecutor().when( + (cmd) => cmd.includes('deploy-state.json'), + { stdout: JSON.stringify({ activeColor: 'green', bluePort: 3000, greenPort: 13000 }), stderr: '', exitCode: 0 }, + ); + + const result = await makeHotSync(executor, config).run(['src/index.ts']); + + const commands = executor.getHistory().map((entry) => entry.command); + expect(commands.some((c) => c.includes('systemctl restart shipnode-api-green'))).toBe(true); + expect(commands.some((c) => c.includes('systemctl restart shipnode-api-mailer'))).toBe(true); + expect(commands.some((c) => c.includes('pm2 reload'))).toBe(false); + expect(result.reloaded).toBe(true); + }); +}); + describe('HotSync — reload', () => { it('reloads the recreate ecosystem file for a non-blue-green app', async () => { const executor = healthyExecutor(); diff --git a/tests/unit/monitor.test.ts b/tests/unit/monitor.test.ts index 4b8c0cf..65694af 100644 --- a/tests/unit/monitor.test.ts +++ b/tests/unit/monitor.test.ts @@ -2,7 +2,8 @@ import { describe, it, expect } from 'vitest'; import { FakeRemoteExecutor } from '../testing/fake-executor.js'; import { buildSparkline, buildGauge, thresholdColor, statusColor, formatUptime, formatBytes } from '../../src/cli/monitor/charts.js'; import { parsePm2Jlist, parseSystemStats, parseReleaseRecords, parseDeployLock, parseHealthProbe, parseAccessoryStatus, parseCaddyInfo, splitSections, systemCpuPercent, nextHealthFailStreak, MetricsHistory, type ProcessInfo, type SystemInfo } from '../../src/cli/monitor/state.js'; -import { buildMonitorCommand, collectMetrics, collectLogs, collectCaddyLogs } from '../../src/cli/monitor/poller.js'; +import { collectMetrics, collectLogs, collectCaddyLogs } from '../../src/cli/monitor/poller.js'; +import { appSectionName } from '../../src/domain/observe/script.js'; import { restartProcess, rollbackToRelease } from '../../src/cli/monitor/actions.js'; import { logLineColor } from '../../src/cli/monitor/panels/LogPanel.js'; import { assembleConfig } from '../../src/config/assembly.js'; @@ -481,7 +482,7 @@ describe('MetricsHistory', () => { ], system: emptySystem, currentRelease: null, - releases: [], + [appSectionName(0, 'releases')]: [], deployLock: null, }); expect(history.cpu).toHaveLength(1); @@ -497,7 +498,7 @@ describe('MetricsHistory', () => { processes: [makeProcess({ cpu: i, memory: i * 10 })], system: emptySystem, currentRelease: null, - releases: [], + [appSectionName(0, 'releases')]: [], deployLock: null, }); } @@ -532,114 +533,18 @@ function sectioned(sections: Record): string { .join('\n'); } -describe('buildMonitorCommand', () => { - it('emits every section marker', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - for (const name of ['pm2', 'sys', 'current', 'releases', 'lock']) { - expect(command).toContain(`@@SHIPNODE:${name}@@`); - } - }); - - it('joins sections with ; so one failure cannot blank the rest', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - expect(command).not.toContain('&&'); - }); - - it('gives pm2 a failure sentinel fallback for backend apps', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - expect(command).toContain('pm2 jlist 2>/dev/null || echo "##SHIPNODE_PM2_FAILED##"'); - }); - - it('skips pm2 for frontend apps', () => { - const frontendConfig = assembleConfig({ - app: 'frontend', - ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, - remotePath: '/var/www/app', - }); - const command = buildMonitorCommand(frontendConfig.apps[0], frontendConfig); - expect(command).not.toContain('pm2 jlist'); - }); - - it('reads the workspace-level deploy lock', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - expect(command).toContain('/var/www/app/.shipnode/deploy.lock'); - }); - - it('collects core count and all three load averages', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - expect(command).toContain('nproc'); - expect(command).toContain(`awk '{print $1, $2, $3}' /proc/loadavg`); - }); - - it('probes the health endpoint capped to the poll interval', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig, { healthMaxTimeSeconds: 2 }); - expect(command).toContain('@@SHIPNODE:health@@'); - expect(command).toContain('http://localhost:3000/health'); - expect(command).toContain('--max-time 2'); - }); - - it('omits the health probe when the health check is disabled', () => { - const config = assembleConfig({ - app: 'backend', - ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, - remotePath: '/var/www/app', - pm2: { apps: [{ name: 'api', port: 3000 }] }, - healthCheck: { enabled: false }, - }); - expect(buildMonitorCommand(config.apps[0], config)).not.toContain('@@SHIPNODE:health@@'); - }); - - it('omits the health probe when no pm2 app declares a port', () => { - const config = assembleConfig({ - app: 'backend', - ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, - remotePath: '/var/www/app', - pm2: { apps: [{ name: 'worker' }] }, - }); - expect(buildMonitorCommand(config.apps[0], config)).not.toContain('@@SHIPNODE:health@@'); - }); - - it('samples accessories with sudo -n docker inspect', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig, { accessoryNames: ['postgres', 'redis'] }); - expect(command).toContain('@@SHIPNODE:accessories@@'); - expect(command).toContain('sudo -n docker inspect'); - expect(command).not.toContain('sudo docker inspect'); - expect(command).toContain(`'shipnode-postgres' 'shipnode-redis'`); - }); - - it('omits the accessories section when no names are given', () => { - const command = buildMonitorCommand(testConfig.apps[0], testConfig); - expect(command).not.toContain('@@SHIPNODE:accessories@@'); - }); - - it('adds caddy sections for frontend apps only', () => { - const frontendConfig = assembleConfig({ - app: 'frontend', - ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, - remotePath: '/var/www/app', - }); - const frontendApp = frontendConfig.apps[0]; - const command = buildMonitorCommand(frontendApp, frontendConfig); - expect(command).toContain('@@SHIPNODE:caddy-status@@'); - expect(command).toContain('systemctl is-active caddy'); - expect(command).toContain(`/var/log/caddy/${frontendApp.name}.log`); - - expect(buildMonitorCommand(testConfig.apps[0], testConfig)).not.toContain('caddy'); - }); -}); - describe('collectMetrics', () => { const fullStdout = sectioned({ - pm2: JSON.stringify([{ name: 'api', pid: 123, pm2_env: { status: 'online', pm_uptime: Date.now(), restart_time: 0 }, monit: { cpu: 2.5, memory: 128 * 1024 * 1024 } }]), + [appSectionName(0, 'pm2')]: JSON.stringify([{ name: 'api', pid: 123, pm2_env: { status: 'online', pm_uptime: Date.now(), restart_time: 0 }, monit: { cpu: 2.5, memory: 128 * 1024 * 1024 } }]), sys: ['mem:16000 8000', 'load:0.5 0.4 0.3', 'cores:2', 'uptime:86400', 'disk:100 45'].join('\n'), - current: '/var/www/app/releases/2026-01-01T00-00-00', - releases: JSON.stringify([{ timestamp: '2026-01-01', status: 'success', duration: 10 }]), + [appSectionName(0, 'current')]: '/var/www/app/releases/2026-01-01T00-00-00', + [appSectionName(0, 'releases')]: JSON.stringify([{ timestamp: '2026-01-01', status: 'success', duration: 10 }]), lock: 'none', }); it('collects the full snapshot in a single SSH round trip', async () => { const executor = new FakeRemoteExecutor(); - executor.when((c) => c.includes('@@SHIPNODE:pm2@@'), { stdout: fullStdout, stderr: '', exitCode: 0 }); + executor.when((c) => c.includes('@@SHIPNODE:a0-pm2@@'), { stdout: fullStdout, stderr: '', exitCode: 0 }); const result = await collectMetrics(executor, testConfig.apps[0], testConfig); @@ -670,9 +575,9 @@ describe('collectMetrics', () => { const executor = new FakeRemoteExecutor(); executor.when(() => true, { stdout: sectioned({ - pm2: '[]', + [appSectionName(0, 'pm2')]: '[]', lock: 'none', - health: '200 34', + [appSectionName(0, 'health')]: '200 34', accessories: '/shipnode-postgres|running|healthy|postgres:16', }), stderr: '', @@ -698,7 +603,7 @@ describe('collectMetrics', () => { it('reports an active deploy lock', async () => { const executor = new FakeRemoteExecutor(); executor.when(() => true, { - stdout: sectioned({ pm2: '[]', lock: '2026-07-11T10:00:00Z 42' }), + stdout: sectioned({ [appSectionName(0, 'pm2')]: '[]', lock: '2026-07-11T10:00:00Z 42' }), stderr: '', exitCode: 0, }); @@ -710,7 +615,7 @@ describe('collectMetrics', () => { it('sets error flag when pm2 jlist fails for backend app', async () => { const executor = new FakeRemoteExecutor(); executor.when(() => true, { - stdout: sectioned({ pm2: '##SHIPNODE_PM2_FAILED##', lock: 'none' }), + stdout: sectioned({ [appSectionName(0, 'pm2')]: '##SHIPNODE_PM2_FAILED##', lock: 'none' }), stderr: '', exitCode: 0, }); @@ -725,7 +630,7 @@ describe('collectMetrics', () => { executor.when(() => true, { stdout: '', stderr: '', exitCode: 1 }); const result = await collectMetrics(executor, testConfig.apps[0], testConfig); - expect(result.error).toBe('Monitor poll returned no data'); + expect(result.error).toBe('Observe poll returned no data'); }); it('handles frontend app without PM2 gracefully', async () => { @@ -737,7 +642,7 @@ describe('collectMetrics', () => { const executor = new FakeRemoteExecutor(); executor.when(() => true, { - stdout: sectioned({ pm2: '[]', lock: 'none' }), + stdout: sectioned({ [appSectionName(0, 'pm2')]: '[]', lock: 'none' }), stderr: '', exitCode: 0, }); @@ -808,6 +713,23 @@ describe('monitor session', () => { } }); + it('picks a fleet replica with --on', () => { + const config = assembleConfig({ + servers: { + a: { host: '1.1.1.1', user: 'deploy', port: 22 }, + b: { host: '2.2.2.2', user: 'deploy', port: 22 }, + }, + remotePath: '/var/www/app', + apps: [ + { name: 'api', appType: 'backend', on: ['a', 'b'], healthCheck: { enabled: true } }, + ], + }); + + const session = resolveMonitorSession(config, 'api', 'b'); + expect(session.isOk()).toBe(true); + if (session.isOk()) expect(session.value.target.name).toBe('b'); + }); + it('limits selectable apps to the connected server target', () => { const config = assembleConfig({ servers: { diff --git a/tests/unit/observe-collector.test.ts b/tests/unit/observe-collector.test.ts new file mode 100644 index 0000000..5b7f75c --- /dev/null +++ b/tests/unit/observe-collector.test.ts @@ -0,0 +1,207 @@ +import { describe, it, expect } from 'vitest'; +import { FakeRemoteExecutor } from '../testing/fake-executor.js'; +import { assembleConfig } from '../../src/config/assembly.js'; +import { buildObserveScript, appSectionName } from '../../src/domain/observe/script.js'; +import { MetricsCollector } from '../../src/domain/observe/collector.js'; + +const config = assembleConfig({ + app: 'backend', + ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, + remotePath: '/var/www/app', + pm2: { apps: [{ name: 'api', port: 3000 }] }, +}); + +const frontendConfig = assembleConfig({ + app: 'frontend', + ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, + remotePath: '/var/www/app', +}); + +function sectioned(sections: Record): string { + return Object.entries(sections) + .map(([name, body]) => `@@SHIPNODE:${name}@@\n${body}`) + .join('\n'); +} + +const SYSTEM = 'mem:2048 1024\nload:0.5 0.4 0.3\ncores:2\nuptime:1000\ndisk:40 10'; + +describe('buildObserveScript', () => { + it('emits host-level sections once, not once per app', () => { + const two = { ...config, apps: [config.apps[0], { ...config.apps[0], name: 'worker' }] }; + const command = buildObserveScript(two, { apps: two.apps }); + + expect(command.match(/@@SHIPNODE:sys@@/g)).toHaveLength(1); + expect(command.match(/@@SHIPNODE:lock@@/g)).toHaveLength(1); + }); + + it('addresses each app by index so app names never enter the marker grammar', () => { + const two = { ...config, apps: [config.apps[0], { ...config.apps[0], name: 'has spaces & pipes' }] }; + const command = buildObserveScript(two, { apps: two.apps }); + + expect(command).toContain('@@SHIPNODE:a0-pm2@@'); + expect(command).toContain('@@SHIPNODE:a1-pm2@@'); + }); + + it('joins sections with ; so one failure cannot blank the rest', () => { + expect(buildObserveScript(config, { apps: config.apps })).not.toContain('&&'); + }); + + it('gives pm2 a failure sentinel fallback for backend apps', () => { + expect(buildObserveScript(config, { apps: config.apps })).toContain( + 'pm2 jlist 2>/dev/null || echo "##SHIPNODE_PM2_FAILED##"', + ); + }); + + it('skips pm2 and adds caddy sections for frontend apps', () => { + const command = buildObserveScript(frontendConfig, { apps: frontendConfig.apps }); + expect(command).not.toContain('pm2 jlist'); + expect(command).toContain('@@SHIPNODE:a0-caddy-status@@'); + expect(command).toContain('@@SHIPNODE:a0-caddy-log@@'); + }); + + it('reads the workspace-level deploy lock', () => { + expect(buildObserveScript(config, { apps: config.apps })).toContain('/var/www/app/.shipnode/deploy.lock'); + }); + + it('omits the accessories section when none are requested', () => { + expect(buildObserveScript(config, { apps: config.apps })).not.toContain('@@SHIPNODE:accessories@@'); + expect(buildObserveScript(config, { apps: config.apps, accessoryNames: ['postgres'] })).toContain( + '@@SHIPNODE:accessories@@', + ); + }); + + it('bounds the health probe by the poll interval', () => { + const command = buildObserveScript(config, { apps: config.apps, healthMaxTimeSeconds: 2 }); + expect(command).toContain('--max-time 2'); + }); + + it('emits no app sections for an empty app list', () => { + const command = buildObserveScript(config, { apps: [] }); + expect(command).toContain('@@SHIPNODE:sys@@'); + expect(command).not.toContain('a0-'); + }); +}); + +describe('MetricsCollector', () => { + it('parses a full server snapshot', async () => { + const executor = new FakeRemoteExecutor().when( + () => true, + { + stdout: sectioned({ + sys: SYSTEM, + lock: 'none', + [appSectionName(0, 'pm2')]: JSON.stringify([ + { name: 'api', pid: 42, pm2_env: { status: 'online' }, monit: { cpu: 5, memory: 104857600 } }, + ]), + [appSectionName(0, 'current')]: '/var/www/app/backend/releases/20260830010101', + [appSectionName(0, 'releases')]: '[]', + [appSectionName(0, 'health')]: '200 12', + }), + stderr: '', + exitCode: 0, + }, + ); + + const snapshot = await new MetricsCollector(executor, 'a.example.com', config).collect({ apps: config.apps }); + + expect(snapshot.server).toBe('a.example.com'); + expect(snapshot.error).toBeUndefined(); + expect(snapshot.system.cores).toBe(2); + expect(snapshot.deployLock).toBeNull(); + expect(snapshot.apps).toHaveLength(1); + expect(snapshot.apps[0].app).toBe('api'); + expect(snapshot.apps[0].processes[0].pm2Name).toBe('api'); + expect(snapshot.apps[0].currentRelease).toBe('/var/www/app/backend/releases/20260830010101'); + expect(snapshot.apps[0].health).toEqual({ status: 'ok', httpCode: 200, responseMs: 12 }); + }); + + it('degrades one failing section without blanking the rest', async () => { + const executor = new FakeRemoteExecutor().when(() => true, { + stdout: sectioned({ + sys: SYSTEM, + lock: 'none', + [appSectionName(0, 'pm2')]: '##SHIPNODE_PM2_FAILED##', + [appSectionName(0, 'current')]: '/var/www/app/backend/releases/20260830010101', + [appSectionName(0, 'releases')]: '[]', + }), + stderr: '', + exitCode: 0, + }); + + const snapshot = await new MetricsCollector(executor, 'a', config).collect({ apps: config.apps }); + + expect(snapshot.error).toBeUndefined(); + expect(snapshot.system.cores).toBe(2); + expect(snapshot.apps[0].error).toBe('PM2 command failed'); + expect(snapshot.apps[0].currentRelease).not.toBeNull(); + }); + + it('reports an unreachable host as a snapshot, not a throw', async () => { + const executor = new (class extends FakeRemoteExecutor { + override async exec(): Promise { + throw new Error('connect ETIMEDOUT'); + } + })(); + + const snapshot = await new MetricsCollector(executor, 'b', config).collect({ apps: config.apps }); + + expect(snapshot.error).toBe('connect ETIMEDOUT'); + expect(snapshot.apps).toEqual([]); + expect(snapshot.server).toBe('b'); + }); + + it('reports empty output as an error rather than an empty snapshot', async () => { + const executor = new FakeRemoteExecutor().when(() => true, { stdout: '', stderr: '', exitCode: 0 }); + const snapshot = await new MetricsCollector(executor, 'c', config).collect({ apps: config.apps }); + expect(snapshot.error).toBe('Observe poll returned no data'); + }); + + it('keeps apps addressable independently when several share a server', async () => { + const two = { ...config, apps: [config.apps[0], { ...config.apps[0], name: 'worker' }] }; + const executor = new FakeRemoteExecutor().when(() => true, { + stdout: sectioned({ + sys: SYSTEM, + lock: 'none', + [appSectionName(0, 'pm2')]: '[]', + [appSectionName(0, 'current')]: '/var/www/app/backend/releases/1', + [appSectionName(0, 'releases')]: '[]', + [appSectionName(1, 'pm2')]: '[]', + [appSectionName(1, 'current')]: 'none', + [appSectionName(1, 'releases')]: '[]', + }), + stderr: '', + exitCode: 0, + }); + + const snapshot = await new MetricsCollector(executor, 'a', two).collect({ apps: two.apps }); + + expect(snapshot.apps.map((a) => a.app)).toEqual(['api', 'worker']); + expect(snapshot.apps[0].currentRelease).toBe('/var/www/app/backend/releases/1'); + expect(snapshot.apps[1].currentRelease).toBeNull(); + }); + + it('carries the accessories section only when it was requested', async () => { + const stdout = sectioned({ + sys: SYSTEM, + lock: 'none', + [appSectionName(0, 'pm2')]: '[]', + [appSectionName(0, 'current')]: 'none', + [appSectionName(0, 'releases')]: '[]', + }); + const executor = new FakeRemoteExecutor().when(() => true, { stdout, stderr: '', exitCode: 0 }); + + const snapshot = await new MetricsCollector(executor, 'a', config).collect({ apps: config.apps }); + expect(snapshot.accessories).toBeUndefined(); + }); + + it('bounds the SSH timeout by the poll interval', async () => { + const executor = new FakeRemoteExecutor().when(() => true, { + stdout: sectioned({ sys: SYSTEM, lock: 'none' }), + stderr: '', + exitCode: 0, + }); + + await new MetricsCollector(executor, 'a', config).collect({ apps: [], intervalSeconds: 3 }); + expect(executor.getLastCommand()?.options?.timeout).toBe(9000); + }); +}); diff --git a/tests/unit/observe-pivot.test.ts b/tests/unit/observe-pivot.test.ts new file mode 100644 index 0000000..95f9e91 --- /dev/null +++ b/tests/unit/observe-pivot.test.ts @@ -0,0 +1,113 @@ +import { describe, it, expect } from 'vitest'; +import { pivotByApp } from '../../src/domain/observe/pivot.js'; +import type { AppSnapshot, ServerSnapshot } from '../../src/domain/observe/snapshot.js'; +import type { SystemInfo } from '../../src/domain/observe/types.js'; + +const system: SystemInfo = { + load1: 0.5, load5: 0.4, load15: 0.3, cores: 2, + totalMem: 2048, usedMem: 1024, totalDisk: 40, usedDisk: 10, uptime: 100, +}; + +function app(name: string, release: string | null, extra: Partial = {}): AppSnapshot { + return { + app: name, + appType: 'backend', + processes: [], + currentRelease: release === null ? null : `/srv/${name}/releases/${release}`, + releases: [], + ...extra, + }; +} + +function server(name: string, apps: AppSnapshot[], extra: Partial = {}): ServerSnapshot { + return { + server: name, + timestamp: '2026-08-30T00:00:00.000Z', + system, + deployLock: null, + apps, + ...extra, + }; +} + +describe('pivotByApp', () => { + it('returns nothing for no snapshots', () => { + expect(pivotByApp([])).toEqual([]); + }); + + it('groups one app across its replicas', () => { + const views = pivotByApp([ + server('a.example.com', [app('api', '20260830010101')]), + server('b.example.com', [app('api', '20260830010101')]), + ]); + + expect(views).toHaveLength(1); + expect(views[0].app).toBe('api'); + expect(views[0].replicas.map((r) => r.server)).toEqual(['a.example.com', 'b.example.com']); + expect(views[0].convergence.converged).toBe(true); + expect(views[0].convergence.releases).toEqual(['20260830010101']); + }); + + it('compares the release directory name, not the absolute symlink', () => { + const views = pivotByApp([ + server('a', [{ ...app('api', null), currentRelease: '/srv/api/releases/2026' }]), + server('b', [{ ...app('api', null), currentRelease: '/opt/other/api/releases/2026' }]), + ]); + expect(views[0].convergence.converged).toBe(true); + }); + + it('reports skew when replicas hold different releases', () => { + const views = pivotByApp([ + server('a', [app('api', '20260830010101')]), + server('b', [app('api', '20260829090909')]), + ]); + + expect(views[0].convergence.converged).toBe(false); + expect(views[0].convergence.releases).toHaveLength(2); + }); + + it('separates distinct apps and preserves per-server order', () => { + const views = pivotByApp([ + server('a', [app('api', '1'), app('web', '1')]), + server('b', [app('api', '1')]), + ]); + + expect(views.map((v) => v.app)).toEqual(['api', 'web']); + expect(views[0].replicas).toHaveLength(2); + expect(views[1].replicas).toHaveLength(1); + }); + + it('names an unreachable server instead of counting it as converged', () => { + const views = pivotByApp([ + server('a', [app('api', '20260830010101')]), + server('b', [], { error: 'connect ETIMEDOUT', apps: [] }), + ]); + + expect(views[0].unreachable).toEqual(['b']); + expect(views[0].convergence.releases).toEqual(['20260830010101']); + }); + + it('carries an unreachable replica through only when the app is known to run there', () => { + // A server that failed reports no apps, so it can only be attributed to an + // app another replica proves exists. + const views = pivotByApp([server('b', [], { error: 'down' })]); + expect(views).toEqual([]); + }); + + it('keeps a per-app error visible on the replica', () => { + const views = pivotByApp([ + server('a', [app('api', '1', { error: 'PM2 command failed' })]), + ]); + expect(views[0].replicas[0].snapshot.error).toBe('PM2 command failed'); + expect(views[0].replicas[0].reachable).toBe(true); + }); + + it('treats a replica with no release as undeployed', () => { + const views = pivotByApp([ + server('a', [app('api', '1')]), + server('b', [app('api', null)]), + ]); + expect(views[0].convergence.undeployed).toEqual(['b']); + expect(views[0].convergence.converged).toBe(false); + }); +}); diff --git a/tests/unit/observe-plan.test.ts b/tests/unit/observe-plan.test.ts new file mode 100644 index 0000000..5f962e3 --- /dev/null +++ b/tests/unit/observe-plan.test.ts @@ -0,0 +1,70 @@ +import { describe, it, expect } from 'vitest'; +import { assembleConfig } from '../../src/config/assembly.js'; +import { observeStateJson, planObserveHosts } from '../../src/cli/observe.js'; + +const fleet = assembleConfig({ + servers: { + a: { host: '1.1.1.1', user: 'deploy', port: 22 }, + b: { host: '2.2.2.2', user: 'deploy', port: 22 }, + data: { host: '3.3.3.3', user: 'deploy', port: 22 }, + }, + remotePath: '/var/www/app', + apps: [ + { name: 'api', appType: 'backend', on: ['a', 'b'], healthCheck: { enabled: true } }, + { name: 'web', appType: 'frontend', on: 'a', healthCheck: { enabled: false } }, + ], + accessories: { + postgres: { image: 'postgres:16', on: 'data' }, + }, +}); + +describe('planObserveHosts', () => { + it('groups apps and accessories by the servers they run on', () => { + const plan = planObserveHosts(fleet); + expect(plan.isOk()).toBe(true); + if (!plan.isOk()) return; + expect(plan.value.map((host) => host.name)).toEqual(['a', 'b', 'data']); + expect(plan.value[0]?.apps.map((app) => app.name)).toEqual(['api', 'web']); + expect(plan.value[1]?.apps.map((app) => app.name)).toEqual(['api']); + expect(plan.value[2]?.accessoryNames).toEqual(['postgres']); + }); + + it('narrows to --on without dropping that server\'s other apps', () => { + const plan = planObserveHosts(fleet, { on: 'a' }); + expect(plan.isOk()).toBe(true); + if (!plan.isOk()) return; + expect(plan.value.map((host) => host.name)).toEqual(['a']); + expect(plan.value[0]?.apps.map((app) => app.name)).toEqual(['api', 'web']); + }); + + it('follows an app onto every replica it runs on', () => { + const plan = planObserveHosts(fleet, { app: 'api' }); + expect(plan.isOk()).toBe(true); + if (!plan.isOk()) return; + expect(plan.value.map((host) => host.name)).toEqual(['a', 'b']); + expect(plan.value.every((host) => host.apps.map((app) => app.name).join() === 'api')).toBe(true); + }); + + it('rejects an unknown app or server', () => { + expect(planObserveHosts(fleet, { app: 'nope' }).isErr()).toBe(true); + expect(planObserveHosts(fleet, { on: 'nope' }).isErr()).toBe(true); + }); +}); + +describe('observeStateJson', () => { + it('emits servers and fleets from one tick', () => { + const json = observeStateJson({ + servers: [], + fleets: [], + events: [], + lastUpdate: '2026-09-18T00:00:00.000Z', + polling: false, + }); + expect(JSON.parse(json)).toEqual({ + servers: [], + fleets: [], + events: [], + lastUpdate: '2026-09-18T00:00:00.000Z', + }); + }); +}); diff --git a/tests/unit/observe-script.test.ts b/tests/unit/observe-script.test.ts new file mode 100644 index 0000000..c5a61d3 --- /dev/null +++ b/tests/unit/observe-script.test.ts @@ -0,0 +1,31 @@ +import { describe, it, expect } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import { assembleConfig } from '../../src/config/assembly.js'; +import { buildObserveScript } from '../../src/domain/observe/script.js'; + +const config = assembleConfig({ + app: 'backend', + ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, + remotePath: '/var/www/app', + pm2: { apps: [{ name: 'api', port: 3000 }] }, +} as never); + +describe('buildObserveScript system section', () => { + const script = buildObserveScript(config, { apps: [] }); + + it('reads memory in megabytes with a single unit flag', () => { + // procps `free` rejects `-m` combined with `-b` ("Multiple unit options"), + // which left the memory line empty and the monitor showing 0 / 0 MB. + expect(script).toContain('free -m |'); + expect(script).not.toMatch(/free -\w*m\w*b|free -\w*b\w*m/); + }); + + it('emits a well-formed mem line wherever `free -m` is available', () => { + const line = /echo "(mem:\$\(free -m \| awk '[^']*'\))"/.exec(script)?.[1]; + expect(line).toBeDefined(); + const probe = spawnSync('bash', ['-c', 'command -v free'], { encoding: 'utf8' }); + if (probe.status !== 0) return; // not Linux; the flag check above still guards the regression + const out = spawnSync('bash', ['-c', `echo "${line}"`], { encoding: 'utf8' }).stdout.trim(); + expect(out).toMatch(/^mem:\d+ \d+$/); + }); +}); diff --git a/tests/unit/observe-session.test.ts b/tests/unit/observe-session.test.ts new file mode 100644 index 0000000..c9d084d --- /dev/null +++ b/tests/unit/observe-session.test.ts @@ -0,0 +1,291 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { + ObserveSession, + HEALTH_ALERT_THRESHOLD, + MAX_CONCURRENT_POLLS, + type ObserveTarget, + type ServerObserver, +} from '../../src/services/observe/session.js'; +import type { CollectRequest } from '../../src/domain/observe/collector.js'; +import type { AppSnapshot, ServerSnapshot } from '../../src/domain/observe/snapshot.js'; +import type { HealthInfo, SystemInfo } from '../../src/domain/observe/types.js'; + +const system: SystemInfo = { + load1: 0.5, load5: 0.4, load15: 0.3, cores: 2, + totalMem: 2048, usedMem: 1024, totalDisk: 40, usedDisk: 10, uptime: 100, +}; + +function appSnapshot(name: string, extra: Partial = {}): AppSnapshot { + return { app: name, appType: 'backend', processes: [], currentRelease: null, releases: [], ...extra }; +} + +/** A collector stand-in whose answers the test dictates tick by tick. */ +class StubObserver implements ServerObserver { + requests: CollectRequest[] = []; + private queue: ServerSnapshot[] = []; + + constructor( + readonly serverName: string, + private fallback: ServerSnapshot, + private onCollect?: () => Promise, + ) {} + + queueUp(...snapshots: ServerSnapshot[]): this { + this.queue.push(...snapshots); + return this; + } + + async collect(request: CollectRequest): Promise { + this.requests.push(request); + if (this.onCollect) await this.onCollect(); + return this.queue.shift() ?? this.fallback; + } +} + +function serverSnapshot(server: string, apps: AppSnapshot[], extra: Partial = {}): ServerSnapshot { + return { server, timestamp: '2026-08-30T00:00:00.000Z', system, deployLock: null, apps, ...extra }; +} + +function target(observer: ServerObserver, apps: string[] = ['api'], accessoryNames: string[] = []): ObserveTarget { + return { + observer, + // Only the name is read by the session; the collector owns the rest. + apps: apps.map((name) => ({ name }) as ObserveTarget['apps'][number]), + accessoryNames, + }; +} + +const healthy: HealthInfo = { status: 'ok', httpCode: 200, responseMs: 10 }; +const failing: HealthInfo = { status: 'fail', httpCode: 502, responseMs: 5 }; + +describe('ObserveSession', () => { + it('publishes both the server-shaped and app-shaped views of one tick', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api', { currentRelease: '/x/releases/1' })])); + const b = new StubObserver('b', serverSnapshot('b', [appSnapshot('api', { currentRelease: '/x/releases/1' })])); + const session = new ObserveSession({ targets: [target(a), target(b)], intervalSeconds: 2 }); + + await session.tick(); + const state = session.getState(); + + expect(state.servers.map((s) => s.server)).toEqual(['a', 'b']); + expect(state.fleets).toHaveLength(1); + expect(state.fleets[0].convergence.converged).toBe(true); + expect(state.lastUpdate).not.toBeNull(); + }); + + it('surfaces release skew across replicas without a second poll', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api', { currentRelease: '/x/releases/2' })])); + const b = new StubObserver('b', serverSnapshot('b', [appSnapshot('api', { currentRelease: '/x/releases/1' })])); + const session = new ObserveSession({ targets: [target(a), target(b)], intervalSeconds: 2 }); + + await session.tick(); + + expect(session.getState().fleets[0].convergence.converged).toBe(false); + expect(a.requests).toHaveLength(1); + expect(b.requests).toHaveLength(1); + }); + + it('keeps reporting the other servers when one is unreachable', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')])); + const b = new StubObserver('b', serverSnapshot('b', [], { error: 'connect ETIMEDOUT' })); + const session = new ObserveSession({ targets: [target(a), target(b)], intervalSeconds: 2 }); + + await session.tick(); + const state = session.getState(); + + expect(state.servers).toHaveLength(2); + expect(state.fleets[0].unreachable).toEqual(['b']); + expect(state.events.filter((e) => e.kind === 'server-unreachable')).toHaveLength(1); + }); + + it('reports an unreachable host once, not once per tick', async () => { + const b = new StubObserver('b', serverSnapshot('b', [], { error: 'down' })); + const session = new ObserveSession({ targets: [target(b)], intervalSeconds: 2 }); + + await session.tick(); + await session.tick(); + await session.tick(); + + expect(session.getState().events.filter((e) => e.kind === 'server-unreachable')).toHaveLength(1); + }); + + it('announces recovery when a host answers again', async () => { + const healthySnapshot = serverSnapshot('b', [appSnapshot('api')]); + const b = new StubObserver('b', healthySnapshot).queueUp(serverSnapshot('b', [], { error: 'down' })); + const session = new ObserveSession({ targets: [target(b)], intervalSeconds: 2 }); + + await session.tick(); + await session.tick(); + + expect(session.getState().events.map((e) => e.kind)).toEqual(['server-unreachable', 'server-recovered']); + }); + + it('raises a health alert on the threshold crossing only', async () => { + const down = serverSnapshot('a', [appSnapshot('api', { health: failing })]); + const a = new StubObserver('a', down); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + + for (let i = 0; i < HEALTH_ALERT_THRESHOLD + 2; i += 1) await session.tick(); + + const alerts = session.getState().events.filter((e) => e.kind === 'health-failing'); + expect(alerts).toHaveLength(1); + expect(session.healthFailStreak('a', 'api')).toBe(HEALTH_ALERT_THRESHOLD + 2); + }); + + it('announces health recovery after an alert', async () => { + const up = serverSnapshot('a', [appSnapshot('api', { health: healthy })]); + const down = serverSnapshot('a', [appSnapshot('api', { health: failing })]); + const a = new StubObserver('a', up).queueUp(down, down, down); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + + for (let i = 0; i < 4; i += 1) await session.tick(); + + expect(session.getState().events.map((e) => e.kind)).toEqual(['health-failing', 'health-recovered']); + expect(session.healthFailStreak('a', 'api')).toBe(0); + }); + + it('tracks health streaks per app, not per server', async () => { + const a = new StubObserver( + 'a', + serverSnapshot('a', [ + appSnapshot('api', { health: failing }), + appSnapshot('web', { health: healthy }), + ]), + ); + const session = new ObserveSession({ targets: [target(a, ['api', 'web'])], intervalSeconds: 2 }); + + for (let i = 0; i < HEALTH_ALERT_THRESHOLD; i += 1) await session.tick(); + + expect(session.healthFailStreak('a', 'api')).toBe(HEALTH_ALERT_THRESHOLD); + expect(session.healthFailStreak('a', 'web')).toBe(0); + }); + + it('samples accessories on a slower cadence than the poll', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')])); + const session = new ObserveSession({ targets: [target(a, ['api'], ['postgres'])], intervalSeconds: 2 }); + + // 10s cadence over a 2s interval means one sample every fifth tick. + for (let i = 0; i < 6; i += 1) await session.tick(); + + const sampled = a.requests.filter((r) => r.accessoryNames !== undefined); + expect(sampled).toHaveLength(2); + expect(a.requests[0].accessoryNames).toEqual(['postgres']); + expect(a.requests[1].accessoryNames).toBeUndefined(); + }); + + it('carries the last accessory reading through ticks that skipped it', async () => { + const withAccessories = serverSnapshot('a', [appSnapshot('api')], { + accessories: [{ name: 'postgres', status: 'running', health: 'healthy', image: 'postgres:16' }], + }); + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')])).queueUp(withAccessories); + const session = new ObserveSession({ targets: [target(a, ['api'], ['postgres'])], intervalSeconds: 2 }); + + await session.tick(); + await session.tick(); + + expect(session.getState().servers[0].accessories).toHaveLength(1); + }); + + it('skips a tick that arrives while the previous one is still running', async () => { + let release = (): void => {}; + const blocked = new Promise((resolve) => { + release = resolve; + }); + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')]), () => blocked); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + + const first = session.tick(); + await session.tick(); + release(); + await first; + + expect(a.requests).toHaveLength(1); + }); + + it('caps how many servers are polled at once', async () => { + let live = 0; + let peak = 0; + const observers = Array.from({ length: MAX_CONCURRENT_POLLS + 3 }, (_, i) => { + const name = `s${i}`; + return new StubObserver(name, serverSnapshot(name, [appSnapshot('api')]), async () => { + live += 1; + peak = Math.max(peak, live); + await Promise.resolve(); + live -= 1; + }); + }); + const session = new ObserveSession({ targets: observers.map((o) => target(o)), intervalSeconds: 2 }); + + await session.tick(); + + expect(peak).toBe(MAX_CONCURRENT_POLLS); + expect(observers.every((o) => o.requests.length === 1)).toBe(true); + }); + + it('records history per server and app', async () => { + const a = new StubObserver( + 'a', + serverSnapshot('a', [ + appSnapshot('api', { + processes: [ + { name: 'api', pm2Name: 'api', pid: 1, status: 'online', cpu: 10, memory: 100, uptime: 0, restarts: 0, execMode: 'fork', instances: 1, unstableRestarts: 0, exitCode: null }, + ], + health: healthy, + }), + ]), + ); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + + await session.tick(); + await session.tick(); + + expect(session.history('a', 'api').cpu).toEqual([10, 10]); + expect(session.history('a', 'api').responseMs).toEqual([10, 10]); + expect(session.history('a', 'other').cpu).toEqual([]); + }); + + it('notifies subscribers and stops on unsubscribe', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')])); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + const seen: number[] = []; + const unsubscribe = session.subscribe((state) => seen.push(state.servers.length)); + + await session.tick(); + expect(seen.length).toBeGreaterThan(0); + + unsubscribe(); + const before = seen.length; + await session.tick(); + expect(seen).toHaveLength(before); + }); + + it('bounds the event log', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api', { error: 'PM2 command failed' })])); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2, maxEvents: 5 }); + + for (let i = 0; i < 12; i += 1) await session.tick(); + + expect(session.getState().events).toHaveLength(5); + }); +}); + +describe('ObserveSession scheduling', () => { + beforeEach(() => vi.useFakeTimers()); + afterEach(() => vi.useRealTimers()); + + it('polls immediately on start and then on the interval', async () => { + const a = new StubObserver('a', serverSnapshot('a', [appSnapshot('api')])); + const session = new ObserveSession({ targets: [target(a)], intervalSeconds: 2 }); + + session.start(); + await vi.advanceTimersByTimeAsync(0); + expect(a.requests).toHaveLength(1); + + await vi.advanceTimersByTimeAsync(4000); + expect(a.requests).toHaveLength(3); + + session.stop(); + await vi.advanceTimersByTimeAsync(10000); + expect(a.requests).toHaveLength(3); + }); +}); diff --git a/tests/unit/rollback-fleet.test.ts b/tests/unit/rollback-fleet.test.ts index e9252b1..8095b17 100644 --- a/tests/unit/rollback-fleet.test.ts +++ b/tests/unit/rollback-fleet.test.ts @@ -74,6 +74,7 @@ function commandsOn(host: string): string[] { beforeEach(() => { executors.clear(); vi.mocked(loadConfig).mockResolvedValue(fleetConfig()); + vi.mocked(confirm).mockClear(); vi.mocked(confirm).mockResolvedValue(true); }); @@ -111,6 +112,14 @@ describe('rolling back a fleet', () => { expect(commandsOn('10.0.0.12').some((cmd) => cmd.includes('mv -Tf'))).toBe(true); }); + it('skips the confirmation with --yes and still rolls every replica back', async () => { + await cmdRollback('/project', { app: 'api', yes: true }); + + expect(confirm).not.toHaveBeenCalled(); + expect(commandsOn('10.0.0.11').some((cmd) => cmd.includes('mv -Tf'))).toBe(true); + expect(commandsOn('10.0.0.12').some((cmd) => cmd.includes('mv -Tf'))).toBe(true); + }); + it('touches nothing when the confirmation is declined', async () => { vi.mocked(confirm).mockResolvedValue(false); diff --git a/tests/unit/status-fleet.test.ts b/tests/unit/status-fleet.test.ts index 0b887ab..76b6746 100644 --- a/tests/unit/status-fleet.test.ts +++ b/tests/unit/status-fleet.test.ts @@ -19,14 +19,20 @@ vi.mock('../../src/infrastructure/ssh/connection.js', () => ({ async connect(ssh: { host: string }): Promise { const release = releaseByHost.get(ssh.host) ?? null; - this.delegate - .when((cmd) => cmd.includes('pm2 jlist'), { stdout: '[]', stderr: '', exitCode: 0 }) - .when((cmd) => cmd.includes('readlink'), { - stdout: release === null ? 'no current symlink' : `/var/www/app/api/releases/${release}`, - stderr: '', - exitCode: 0, - }) - .when((cmd) => cmd.includes('ls -1t'), { stdout: '', stderr: '', exitCode: 0 }); + const current = release === null ? 'none' : `/var/www/app/api/releases/${release}`; + const stdout = [ + '@@SHIPNODE:sys@@', + 'mem:2048 1024\nload:0.5 0.4 0.3\ncores:2\nuptime:1000\ndisk:40 10', + '@@SHIPNODE:lock@@', + '', + '@@SHIPNODE:a0-pm2@@', + '[]', + '@@SHIPNODE:a0-current@@', + current, + '@@SHIPNODE:a0-releases@@', + '', + ].join('\n'); + this.delegate.when(() => true, { stdout, stderr: '', exitCode: 0 }); } disconnect(): void {} diff --git a/tests/unit/watt-runtime.test.ts b/tests/unit/watt-runtime.test.ts new file mode 100644 index 0000000..9d89ee0 --- /dev/null +++ b/tests/unit/watt-runtime.test.ts @@ -0,0 +1,376 @@ +import { describe, it, expect, vi } from 'vitest'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { BackendStrategy } from '../../src/domain/deploy/backend-strategy.js'; +import { FakeRemoteExecutor } from '../testing/fake-executor.js'; +import { assembleConfig } from '../../src/config/assembly.js'; +import { shipnode } from '../../src/config/builder.js'; +import type { StrategyContext } from '../../src/domain/deploy/strategy.js'; +import { + parseSize, renderAppConfig, renderRunScript, renderRuntimeConfig, renderUnit, resolveWattUnits, wattUnitName, + wattEnsureInstalledCommand, WATT_VERSION, portFreeGuard, pm2DeleteCommand, +} from '../../src/domain/runtime/watt.js'; + +vi.mock('execa', () => ({ execa: vi.fn().mockResolvedValue({ stdout: '', stderr: '', exitCode: 0 }) })); +vi.mock('fs-extra', () => ({ pathExists: vi.fn().mockResolvedValue(false) })); + +const watt = { main: 'dist/server.js' }; + +function config(overrides: Record = {}) { + return assembleConfig({ + app: 'backend', + ssh: { host: '1.2.3.4', user: 'deploy', port: 22 }, + remotePath: '/var/www/app', + pm2: { apps: [{ name: 'api', port: 3000, instances: 4, maxMemory: '512M' }, { name: 'mailer', command: 'node dist/mailer.js' }] }, + runtime: 'watt', + watt, + nodeVersion: 'lts', + pkgManager: 'npm', + ...overrides, + } as never); +} + +function ctx(executor: FakeRemoteExecutor, cfg = config(), extra: Partial = {}): StrategyContext { + return { config: cfg, app: cfg.apps[0], executor, workDir: '/var/www/app/api/releases/1', cwd: '/local', skipBuild: false, ...extra } as StrategyContext; +} + +const strategy = (cfg = config()) => new BackendStrategy(cfg, cfg.apps[0], '/local'); +const cmds = (e: FakeRemoteExecutor) => e.getHistory().map((h) => h.command); + +describe('watt config validation', () => { + it('requires watt.main when runtime is watt', () => { + expect(() => config({ watt: undefined })).toThrow(/watt\.main/); + }); + it('rejects watt settings without the watt runtime', () => { + expect(() => config({ runtime: undefined })).toThrow(/require runtime 'watt'/); + }); + it('requires a web app', () => { + expect(() => config({ pm2: { apps: [{ name: 'w', command: 'node w.js' }] } })).toThrow(/web app/); + }); + it('defaults to pm2 (runtime unset) so existing configs are unchanged', () => { + const cfg = config({ runtime: undefined, watt: undefined }); + expect(cfg.apps[0].runtime).toBeUndefined(); + }); + it('builds through the fluent builder', () => { + const cfg = shipnode.backend().ssh({ host: '1.2.3.4', user: 'deploy' }).deployTo('/var/www/x') + .pm2('x', { instances: 3 }).port(3000).runtime('watt', watt).build(); + expect(cfg.apps[0].runtime).toBe('watt'); + expect(cfg.apps[0].watt?.main).toBe('dist/server.js'); + }); +}); + +describe('watt renderers', () => { + it('maps instances to worker threads and uses a per-colour PORT placeholder', () => { + const json = JSON.parse(renderRuntimeConfig({ name: 'api', port: 3000, instances: 4, maxMemory: '512M' }, watt)); + expect(json.workers).toEqual({ static: 4 }); + expect(json.server.port).toBe('{PORT}'); + expect(json.entrypoint).toBe('web'); + expect(json.health.maxHeapUsed).toBe(512 * 1024 ** 2); + }); + it('points the capability at the configured entry file', () => { + const json = JSON.parse(renderAppConfig({ main: 'dist/server.js', module: '@platformatic/node' })); + expect(json.node.main).toBe('dist/server.js'); + expect(json.module).toBe('@platformatic/node'); + }); + it('parses sizes', () => { + expect(parseSize('1G')).toBe(1024 ** 3); + expect(parseSize('nope')).toBeUndefined(); + }); + it('names units with the colour suffix and PM2-style namespacing', () => { + expect(wattUnitName('api', 'api', 'green')).toBe('shipnode-api-green'); + expect(wattUnitName('api', 'mailer')).toBe('shipnode-api-mailer'); + }); + it('renders a launcher that never sources the env file', () => { + const script = renderRunScript({ cwd: '/c', command: 'node w.js', envFile: '/s/.env', env: { PORT: 1 } }); + expect(script).not.toContain('source '); + expect(script).toContain('exec mise exec -- node w.js'); + }); + it('renders a restarting systemd unit', () => { + const unit = renderUnit({ description: 'd', user: 'deploy', workingDirectory: '/w', script: '/w/run.sh' }); + expect(unit).toContain('Restart=always'); + expect(unit).toContain('User=deploy'); + }); +}); + +describe('BackendStrategy watt — recreate', () => { + it('writes configs, installs a unit per process, and restarts them', async () => { + const cfg = config({ zeroDowntime: false }); + const e = new FakeRemoteExecutor(); + await strategy(cfg).startApp!(ctx(e, cfg)); + const all = cmds(e); + expect(all.some((c) => c.includes('shipnode.watt.json'))).toBe(true); + expect(all.some((c) => c.includes('/etc/systemd/system/shipnode-api.service'))).toBe(true); + expect(all.some((c) => c.includes('/etc/systemd/system/shipnode-api-mailer.service'))).toBe(true); + expect(all.some((c) => c.includes('systemctl restart shipnode-api'))).toBe(true); + expect(all.some((c) => c.includes('pm2 start'))).toBe(false); + expect(all.some((c) => c.includes('wattpm'))).toBe(true); + }); + + it('fails fast with an actionable message when wattpm is not a dependency', async () => { + const cfg = config({ zeroDowntime: false }); + const e = new FakeRemoteExecutor().when((c) => c.includes('bin/wattpm" ]'), { stdout: '', stderr: 'x', exitCode: 1 }); + await expect(strategy(cfg).startApp!(ctx(e, cfg))).rejects.toThrow(); + }); +}); + +describe('BackendStrategy watt — blue-green', () => { + const target = (o: Record = {}) => ({ color: 'green', port: 13000, previousColor: 'blue', previousPort: 3000, bluePort: 3000, greenPort: 13000, ...o }) as never; + const bg = () => config({ domain: 'api.example.com', zeroDowntime: true }); + + it('boots the idle colour on its own port and holds workers back', async () => { + const cfg = bg(); + const e = new FakeRemoteExecutor(); + await strategy(cfg).startApp!(ctx(e, cfg, { deployTarget: target() })); + const all = cmds(e); + const script = all.find((c) => c.includes('shipnode-run-api-green.sh') && c.includes('printf')); + expect(script).toBeDefined(); + expect(all.some((c) => c.includes('systemctl restart shipnode-api-green'))).toBe(true); + expect(all.some((c) => c.includes('13000'))).toBe(true); + expect(all.some((c) => c.includes('systemctl restart shipnode-api-mailer'))).toBe(false); + }); + + it('starts workers only after the new colour is healthy', async () => { + const cfg = bg(); + const e = new FakeRemoteExecutor(); + await strategy(cfg).afterHealthy!(ctx(e, cfg, { deployTarget: target() })); + expect(cmds(e).some((c) => c.includes('systemctl restart shipnode-api-mailer'))).toBe(true); + }); + + it('keeps the previous colour when retention is rollback', async () => { + const cfg = bg(); + const e = new FakeRemoteExecutor(); + await strategy(cfg).afterTrafficSwitch!(ctx(e, cfg, { deployTarget: target() })); + expect(cmds(e)).toHaveLength(0); + }); + + it('stops the previous colour when retention is none', async () => { + const cfg = config({ domain: 'api.example.com', zeroDowntime: true, blueGreenRetention: 'none' }); + const e = new FakeRemoteExecutor(); + await strategy(cfg).afterTrafficSwitch!(ctx(e, cfg, { deployTarget: target() })); + expect(cmds(e).some((c) => c.includes('systemctl stop shipnode-api-blue'))).toBe(true); + }); + + it('retires the pre-blue-green unit and any PM2 process on the first flip', async () => { + const cfg = bg(); + const e = new FakeRemoteExecutor(); + await strategy(cfg).afterTrafficSwitch!(ctx(e, cfg, { deployTarget: target({ previousColor: null }) })); + const all = cmds(e); + expect(all.some((c) => c.includes('disable --now shipnode-api'))).toBe(true); + expect(all.some((c) => c.includes('pm2 delete "api"'))).toBe(true); + }); +}); + +describe('resolveWattUnits', () => { + it('targets the active colour for the web unit and plain names for workers', async () => { + const cfg = config({ domain: 'api.example.com', zeroDowntime: true }); + const e = new FakeRemoteExecutor().when((c) => c.includes('deploy-state.json'), { + stdout: JSON.stringify({ activeColor: 'green', bluePort: 3000, greenPort: 13000 }), stderr: '', exitCode: 0, + }); + const units = await resolveWattUnits(e, '/var/www/app/api', cfg.apps[0], { colors: 'active' }); + expect(units).toEqual(['shipnode-api-green', 'shipnode-api-mailer']); + const all = await resolveWattUnits(e, '/var/www/app/api', cfg.apps[0], { colors: 'all', process: 'api' }); + expect(all).toEqual(['shipnode-api-green', 'shipnode-api-blue']); + }); + it('rejects an unknown process', async () => { + const cfg = config(); + await expect(resolveWattUnits(new FakeRemoteExecutor(), '/x', cfg.apps[0], { colors: 'active', process: 'nope' })).rejects.toThrow(/No process named/); + }); +}); + +// ── observe / health / monitor on watt ─────────────────────────────────────── + +import { parseSystemdUnits } from '../../src/domain/observe/parse.js'; +import { buildObserveScript, wattCandidateUnits } from '../../src/domain/observe/script.js'; +import { HealthCheckService } from '../../src/services/health.service.js'; +import { restartProcess } from '../../src/cli/monitor/actions.js'; +import { collectLogs } from '../../src/cli/monitor/poller.js'; + +const UNITS_OUTPUT = `@unit shipnode-api-green +LoadState=loaded +ActiveState=active +SubState=running +MainPID=4242 +MemoryCurrent=268435456 +NRestarts=0 +ExecMainStatus=0 +CPUUsageNSec=1000000000 +started=1790000000 +@unit shipnode-api-blue +LoadState=not-found +ActiveState=inactive +SubState=dead +MainPID=0 +MemoryCurrent=[not set] +NRestarts=0 +ExecMainStatus=0 +CPUUsageNSec=[not set] +started= +@unit shipnode-api-mailer +LoadState=loaded +ActiveState=activating +SubState=auto-restart +MainPID=0 +MemoryCurrent=[not set] +NRestarts=3 +ExecMainStatus=1 +CPUUsageNSec=5 +started= +@wall 200000000 +@cpu shipnode-api-green 1100000000 +@cpu shipnode-api-mailer 5 +`; + +describe('parseSystemdUnits', () => { + const procs = parseSystemdUnits(UNITS_OUTPUT, 'api', { unitBase: 'shipnode-api', instances: 4 }); + + it('drops units that do not exist', () => { + expect(procs.map((p) => p.pm2Name)).toEqual(['shipnode-api-green', 'shipnode-api-mailer']); + }); + it('maps state, memory, pid and uptime onto the dashboard vocabulary', () => { + const web = procs[0]; + expect(web).toMatchObject({ status: 'online', pid: 4242, memory: 256, supervisor: 'systemd', execMode: 'threads', instances: 4, uptime: 1790000000_000 }); + }); + it('derives CPU% from two samples of the cumulative counter', () => { + // 100ms of CPU over a 200ms window = 50% + expect(procs[0].cpu).toBe(50); + }); + it('reports a crash-looping worker as errored with its restart count', () => { + expect(procs[1]).toMatchObject({ status: 'errored', restarts: 3, pid: null, execMode: 'fork', exitCode: 1 }); + }); + it('tolerates empty output', () => { + expect(parseSystemdUnits('', 'api')).toEqual([]); + }); +}); + +describe('observe script for watt apps', () => { + it('samples systemd units instead of pm2', () => { + const cfg = config(); + const script = buildObserveScript(cfg, { apps: [cfg.apps[0]] }); + expect(script).toContain('systemctl show'); + expect(script).not.toContain('pm2 jlist'); + expect(wattCandidateUnits(cfg.apps[0])).toEqual([ + 'shipnode-api', 'shipnode-api-blue', 'shipnode-api-green', 'shipnode-api-mailer', + ]); + }); +}); + +describe('deploy health check on watt', () => { + const app = () => config({ zeroDowntime: false }).apps[0]; + const svc = (e: FakeRemoteExecutor) => new HealthCheckService(e, config()); + const ok = (e: FakeRemoteExecutor) => e.when((c) => c.includes('curl'), { stdout: '200 5', stderr: '', exitCode: 0 }); + + it('passes when every unit is active with no restarts', async () => { + const e = ok(new FakeRemoteExecutor()).when((c) => c.includes('systemctl show'), { stdout: 'active 0', stderr: '', exitCode: 0 }); + await expect(svc(e).perform({ ...app(), healthCheck: { ...app().healthCheck, startupDelay: 0 } })).resolves.toBeDefined(); + expect(e.getHistory().some((h) => h.command.includes('pm2'))).toBe(false); + }); + it('fails on an inactive unit and includes journal output', async () => { + const e = ok(new FakeRemoteExecutor()) + .when((c) => c.includes('journalctl'), { stdout: 'boom: cannot find module', stderr: '', exitCode: 0 }) + .when((c) => c.includes('systemctl show'), { stdout: 'failed 0', stderr: '', exitCode: 0 }); + await expect(svc(e).perform({ ...app(), healthCheck: { ...app().healthCheck, startupDelay: 0 } })).rejects.toThrow(/state=failed[\s\S]*cannot find module/); + }); + it('fails a unit that systemd already restarted (crash loop)', async () => { + const e = ok(new FakeRemoteExecutor()).when((c) => c.includes('systemctl show'), { stdout: 'active 2', stderr: '', exitCode: 0 }); + await expect(svc(e).perform({ ...app(), healthCheck: { ...app().healthCheck, startupDelay: 0 } })).rejects.toThrow(/NRestarts=2/); + }); + it('checks the coloured unit during blue-green', async () => { + const e = ok(new FakeRemoteExecutor()).when((c) => c.includes('systemctl show'), { stdout: 'active 0', stderr: '', exitCode: 0 }); + const a = app(); + await svc(e).perform({ ...a, healthCheck: { ...a.healthCheck, startupDelay: 0 } }, { + httpPort: 13000, + pm2Apps: [a.pm2!.apps[0]], + resolvePm2Name: (p) => `${p.name}-green`, + }); + expect(e.getHistory().some((h) => h.command.includes('shipnode-api-green'))).toBe(true); + }); +}); + +describe('monitor actions on watt', () => { + it('restarts a single unit through systemd', async () => { + const e = new FakeRemoteExecutor(); + const result = await restartProcess(e, 'shipnode-api-green', 'systemd'); + expect(result.isOk()).toBe(true); + expect(e.getLastCommand()?.command).toContain('systemctl restart shipnode-api-green'); + expect(e.getLastCommand()?.command).not.toContain('pm2'); + }); + it('tails the journal for a whole deployment by glob', async () => { + const e = new FakeRemoteExecutor(); + await collectLogs(e, 'api', 20, 'systemd'); + const cmd = e.getLastCommand()!.command; + expect(cmd).toContain("journalctl -u 'shipnode-api' -u 'shipnode-api-*'"); + }); +}); + +describe('wattEnsureInstalledCommand', () => { + it('installs wattpm and the default module at the pinned version when missing', () => { + const cmd = wattEnsureInstalledCommand('/srv/app/current', watt, 'pnpm'); + expect(cmd).toContain(`wattpm@${WATT_VERSION}`); + expect(cmd).toContain(`@platformatic/node@${WATT_VERSION}`); + expect(cmd).toContain('pnpm add $pkgs'); + expect(cmd).toContain('cd "/srv/app/current"'); + }); + + it('only checks for packages the app has not already installed', () => { + const cmd = wattEnsureInstalledCommand('/srv/app/current', watt, 'npm'); + expect(cmd).toContain('[ -x "/srv/app/current/node_modules/.bin/wattpm" ] || pkgs='); + expect(cmd).toContain('[ -d "/srv/app/current/node_modules/@platformatic/node" ] || pkgs='); + expect(cmd).toContain('npm install --no-audit --no-fund $pkgs'); + }); + + it('never auto-installs a custom capability module', () => { + const cmd = wattEnsureInstalledCommand('/srv/app/current', { ...watt, module: '@platformatic/next' }, 'npm'); + expect(cmd).not.toContain('@platformatic/next'); + expect(cmd).not.toContain('@platformatic/node@'); + }); + + it('keeps the rendered schema version in step with the installed version', () => { + expect(renderRuntimeConfig({ name: 'api', port: 3000 } as never, watt)).toContain(`wattpm/${WATT_VERSION}.json`); + expect(renderAppConfig(watt)).toContain(`@platformatic/node/${WATT_VERSION}.json`); + }); +}); + +describe('portFreeGuard', () => { + function runGuard(listening: string): number { + const dir = mkdtempSync(join(tmpdir(), 'ss-')); + writeFileSync(join(dir, 'ss'), `#!/bin/sh\necho "${listening}"\n`, { mode: 0o755 }); + return spawnSync('bash', ['-c', portFreeGuard(13125)], { env: { ...process.env, PATH: `${dir}:${process.env.PATH}` } }).status ?? -1; + } + + it('fails when something already listens on the port', () => { + expect(runGuard('LISTEN 0 511 *:13125 *:* users:((\\"node\\",pid=1,fd=18))')).not.toBe(0); + }); + + it('passes when the port is free', () => { + expect(runGuard('LISTEN 0 511 *:3125 *:*')).toBe(0); + }); +}); + +describe('pm2DeleteCommand', () => { + it('deletes by exact name and never fails the deploy', () => { + const cmd = pm2DeleteCommand('json-green'); + expect(cmd).toContain('pm2 delete "json-green"'); + expect(cmd).toContain('|| true'); + }); +}); + +describe('renderUnit hardening', () => { + const base = { description: 'shipnode api', user: 'deploy', workingDirectory: '/var/www/app', script: '/var/www/app/run.sh' }; + + it('keeps a script path with spaces as one ExecStart argument', () => { + const unit = renderUnit({ ...base, script: '/var/www/my app/run.sh' }); + expect(unit).toContain('ExecStart=/usr/bin/env bash "/var/www/my app/run.sh"'); + }); + + it('escapes characters systemd would interpret inside the quoted path', () => { + const unit = renderUnit({ ...base, script: '/srv/a"b\\c%d/run.sh' }); + expect(unit).toContain('ExecStart=/usr/bin/env bash "/srv/a\\"b\\\\c%%d/run.sh"'); + }); + + it.each(['\n', '\r', '\0'])('refuses a control character in any rendered value (%j)', (ch) => { + expect(() => renderUnit({ ...base, workingDirectory: `/var/www/app${ch}ExecStartPre=/bin/false` })).toThrow(/control character/); + expect(() => renderUnit({ ...base, user: `deploy${ch}` })).toThrow(/control character/); + }); +}); diff --git a/website/astro.config.mjs b/website/astro.config.mjs index 32f4cbb..c204b83 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -32,6 +32,7 @@ export default defineConfig({ { label: 'shipnode.config.ts', slug: 'docs/configuration' }, { label: 'Multi-environment', slug: 'docs/environments' }, { label: 'Workers', slug: 'docs/workers' }, + { label: 'wattpm runtime', slug: 'docs/watt' }, ], }, { diff --git a/website/src/content/docs/docs/commands/eject.md b/website/src/content/docs/docs/commands/eject.md index 1ac1fa8..b56b270 100644 --- a/website/src/content/docs/docs/commands/eject.md +++ b/website/src/content/docs/docs/commands/eject.md @@ -13,4 +13,6 @@ npx shipnode eject caddy # only Caddyfile template Once ejected, ShipNode uses the local templates on every subsequent deploy. +`eject pm2` applies to the PM2 runtime only; the [wattpm runtime](/docs/watt/) renders its own files per release. + Templates land under `.shipnode/templates/` in your project. diff --git a/website/src/content/docs/docs/configuration.md b/website/src/content/docs/docs/configuration.md index 036baed..493f7b9 100644 --- a/website/src/content/docs/docs/configuration.md +++ b/website/src/content/docs/docs/configuration.md @@ -127,6 +127,7 @@ Registry passwords are read from environment variables on the remote host. If `R | `.accessories({ name: config })` | Workspace-level Docker containers shared by apps. | | `.caddy({ append })` | Append raw Caddy directives inside the generated site block. | | `.pm2(name, opts?)` | PM2 app name + options (`{ instances, exec_mode }`). Backend only. | +| `.runtime('watt', { main })` | Opt in to the [wattpm runtime](/docs/watt/): the web app runs as worker threads sharing one port. PM2 is the default. | | `.port(n)` | App's listening port. Caddy reverse-proxies to it. | | `.domain(host)` | Public hostname. Caddy issues + renews certs. | | `.healthCheck(path, opts?)` | GET path the deploy must hit after reload. | diff --git a/website/src/content/docs/docs/watt.md b/website/src/content/docs/docs/watt.md new file mode 100644 index 0000000..080de59 --- /dev/null +++ b/website/src/content/docs/docs/watt.md @@ -0,0 +1,51 @@ +--- +title: wattpm runtime +description: Opt in to running the web app as worker threads sharing one port, supervised by systemd instead of PM2. +--- + +By default ShipNode supervises your app with PM2. The `watt` runtime is an opt-in alternative: the web app runs under [wattpm](https://docs.platformatic.dev/) as **worker threads in one process**, each accepting connections straight from the kernel via `SO_REUSEPORT`. There is no supervisor process handing connections to workers, and V8's startup state is not duplicated per worker. + +```ts +export default shipnode + .backend() + .ssh({ host: '203.0.113.10', user: 'deploy' }) + .deployTo('/var/www/api') + .pm2('api', { instances: 4, maxMemory: '512M' }) // instances = worker threads + .port(3000) + .domain('api.example.com') + .runtime('watt', { main: 'dist/server.js' }) + .worker({ name: 'mailer', command: 'node dist/worker.js' }) + .build(); +``` + +## Requirements + +- `wattpm` and `@platformatic/node` are installed for you at the version shipnode targets when your app doesn't list them. Add them to your own dependencies to pin the versions yourself; your versions are then used and nothing is installed. +- `main` is the file each worker thread loads. It must start an HTTP server on `process.env.PORT`. +- Scaling past one thread needs **Linux**. Elsewhere wattpm forces a single worker for the web app. + +## What changes + +| | PM2 (default) | `watt` | +|---|---|---| +| Web app | PM2 process(es) | wattpm worker threads, `instances` of them | +| Workers | PM2 processes | one systemd unit each | +| Supervision | PM2 + `pm2 startup` | systemd units named `shipnode-[-]` | +| Zero-downtime | blue-green | blue-green (each colour is its own unit) | +| `shipnode restart` | rolling `pm2 reload` | in-place `systemctl restart` | + +Everything else — `deploy`, `rollback`, `logs`, `stop`, `env`, `status`, `deploy --watch`, the monitor — works as with PM2. `metrics` shows a refreshing `systemctl status` because there is no `pm2 monit`. + +`shipnode restart` drops in-flight requests on a watt app. For a zero-drop roll, run `shipnode deploy` (blue-green). + +## Switching an existing app + +Add `.runtime('watt', { main })` and deploy. After the new release passes its health check and Caddy flips, the old PM2 process is removed. To go back, remove the `.runtime(...)` line and deploy again. + +## Trade-offs + +- **Weaker isolation.** Threads share a process, so a native crash or out-of-memory error takes down every web worker. `maxMemory` is passed to wattpm as its per-worker heap health threshold (`health.maxHeapUsed`); `watt.maxHeapUsed` overrides it when both are set. +- **Uneven spread is possible.** The kernel hashes each connection to a worker. Clients that share few source ports, such as a local proxy over loopback, can land unevenly. Validate with a real traffic split before rolling out widely. +- **Linux only** for more than one thread. + +See the [design notes](https://github.com/devalade/shipnode/blob/main/docs/adr/0009-watt-runtime.md) for the reasoning.