feat(blue-green): stop the old colour by default and let rollback start it again

CHANGELOG.md

@@ -4,6 +4,16 @@ All notable changes to `@devalade/shipnode` will be documented here.
 
 ## [Unreleased]
 
+### Changed
+- **Blue-green now stops the old colour by default (`blueGreenRetention: 'warm'`).** After Caddy switches traffic, the previous colour is stopped following a 10-second drain instead of staying in memory, so each app holds one copy. Its release stays on disk. This changes behaviour for configs that never set the option, which used to keep both colours running; set `'rollback'` to keep that. See [ADR-0010](docs/adr/0010-warm-blue-green-retention.md).
+
+### Added
+- **`shipnode rollback` can start a stopped colour.** When the previous colour is not running it points `current` at the release that colour ran, starts it from there, waits for its health check, flips traffic, and stops the colour that was serving. A failed start leaves the app as it was. `deploy-state.json` now records `blueRelease` / `greenRelease` to make this possible; an app deployed before this change can roll back this way after its second deploy with it, because the colour it would go back to has no recorded release until then.
+- **A drain before the old colour stops**, so a request already in flight when Caddy flips can finish.
+
+### Fixed
+- **A reaped `watt` colour came back after a reboot.** Its unit stayed enabled, so it restarted on whatever `current` pointed at and held memory. A stopped colour's unit is now disabled as well, and re-enabled when it is started again.
+
 ## [3.2.0-beta.2] - 2026-10-01
 
 ### Added

README.md

@@ -327,7 +327,7 @@ See [ADR-0007](docs/adr/0007-fleet-replication.md) and [ADR-0008](docs/adr/0008-
 | `.installCommand(cmd)` | derived from pkg manager | Override the install command run on the server (e.g. `'npm ci --legacy-peer-deps'`). Equivalent to `pkgManager(pm, { installCommand: cmd })` |
 | `.buildDir(dir)` | auto-detected | Frontend build output dir |
 | `.zeroDowntime(altPort?)` | automatic for Caddy backends | Force blue-green releases and optionally choose the green port |
-| `.blueGreenRetention('rollback' \| 'none')` | `'rollback'` | Keep the old web process for instant rollback, or reclaim it after the switch |
+| `.blueGreenRetention('warm' \| 'rollback' \| 'none')` | `'warm'` | What happens to the old colour after the switch: stop it but keep its release so `rollback` can start it again (`warm`), keep it running for an instant flip (`rollback`), or stop it with no rollback (`none`) |
 | `.noZeroDowntime()` | — | Opt out and recreate PM2 processes during deploy |
 | `.healthCheck(path, opts?)` | `/health`, 30s, 3 retries | Post-deploy health check |
 | `.noHealthCheck()` | — | Skip health check |
@@ -575,16 +575,16 @@ export default shipnode
 
 Backends with a domain and a PM2 web port use blue-green releases automatically. Shipnode starts the new web process on the idle port, checks it, and reloads Caddy only after it is healthy. Apps without a domain and worker-only apps keep the PM2 recreate path. Use `.noZeroDowntime()` to opt out explicitly; raw configuration may set `zeroDowntime: false`.
 
-Blue-green keeps the previous and current web processes resident, so budget about **2× the web process memory**. On an eligible Caddy backend, `.zeroDowntime(altPort?)` remains available to force the mode and choose the alternate port. By default, Shipnode uses a less commonly occupied port offset by 10,000 (`3000 → 13000`); near the top of the TCP range it subtracts 10,000 instead.
+By default (`warm`) the old colour is stopped a few seconds after Caddy switches traffic, so only one copy of the web app is in memory. Its release stays on disk, and `shipnode rollback` starts it again, waits for its health check, then flips traffic — seconds, not a redeploy. This is the model Kamal uses. On an eligible Caddy backend, `.zeroDowntime(altPort?)` remains available to force the mode and choose the alternate port. By default, Shipnode uses a less commonly occupied port offset by 10,000 (`3000 → 13000`); near the top of the TCP range it subtracts 10,000 instead.
 
-For memory-constrained apps, retain both processes only while the new release starts and passes its health check:
+To trade memory for an instant rollback, keep the old colour running:
 
 ```ts
   .zeroDowntime()
-  .blueGreenRetention('none')
+  .blueGreenRetention('rollback')
 ```
 
-After Caddy switches traffic, Shipnode stops the old colour immediately. This disables instant `shipnode rollback`; redeploy the desired release instead.
+Budget about **2× the web process memory** with `rollback`, since both colours stay resident. `'none'` behaves like `warm` but disables `shipnode rollback` entirely. See [ADR-0010](docs/adr/0010-warm-blue-green-retention.md).
 
 Every app still uses a Capistrano-style release structure:
 

docs/adr/0005-blue-green-zero-downtime.md

@@ -25,7 +25,7 @@ Because the previous colour is still running, rollback is an instant Caddy flip
 
 ## Trade-offs
 
-- **~2× memory for the web app** — both colours are resident between deploys. Documented; the price of instant rollback.
+- **~2× memory for the web app** — with `blueGreenRetention: 'rollback'` both colours are resident between deploys. Documented; the price of instant rollback. The default is now `warm`, which stops the old colour and avoids this cost ([ADR-0010](0010-warm-blue-green-retention.md)).
 - **The port pair is fixed at the first deploy** and persisted. Later changes to the web port or `altPort` in config are ignored until the state file is cleared, so a running colour is never silently re-homed.
 - **The first migration targets green.** A pre-existing uncoloured process can continue serving on the configured blue port through health and the Caddy reload; it is cleaned up only after the flip succeeds.
 - **Default on for Caddy backends.** Backends without a domain and worker-only apps keep recreate semantics. `.noZeroDowntime()` is the explicit builder opt-out.

docs/adr/0010-warm-blue-green-retention.md

@@ -0,0 +1,28 @@
+# Warm blue-green retention is the default
+
+[ADR-0005](0005-blue-green-zero-downtime.md) kept the previous colour running after every flip so a rollback was an instant Caddy flip. The cost is a second copy of every web app in memory, permanently. On a shared server with no swap that is the difference between fitting and being killed: running 40 small apps this way held roughly 3–4 GB that did nothing until someone rolled back.
+
+`blueGreenRetention: 'none'` removed the cost but also removed rollback — `shipnode rollback` refused, and the only way back was a full redeploy. Nothing sat between the two.
+
+## Decision
+
+`warm` is the default, modelled on how Kamal handles old containers: stop the old version after a short drain, keep it on disk, and start it again on rollback.
+
+- **After the flip.** Caddy has stopped sending the old colour new requests. Shipnode waits `DRAIN_SECONDS` (10) so requests already in flight can finish, then stops it: a PM2 process is deleted by exact name, a watt unit is stopped **and disabled** so it does not return on a reboot and hold memory for nothing. Its release directory is untouched (`keepReleases` still applies).
+- **Recording what ran.** `deploy-state.json` gains `blueRelease` / `greenRelease`, the release each colour was last booted from. State written before this change has neither field, and rollback says so instead of guessing.
+- **Rollback.** If the previous colour is running (`rollback` retention) it is still an instant flip. Otherwise shipnode points `current` at that colour's release — its launcher files resolve through `current` (ADR-0001), so starting it from anywhere else would run the wrong code — starts it from that release's own ecosystem file or unit, waits for its health check, flips Caddy, and then stops the colour that was serving. If it cannot start, the half-started colour is removed and `current` is restored, so a failed rollback leaves the app as it was.
+
+## Modes
+
+| | After the flip | `shipnode rollback` | Memory |
+|---|---|---|---|
+| `warm` (default) | stopped after a drain | boots it again, seconds | 1× |
+| `rollback` | left running | instant flip | 2× |
+| `none` | stopped after a drain | refused; redeploy | 1× |
+
+## Trade-offs
+
+- **Rollback is no longer instant by default.** It takes the colour's boot time plus its health check. An app that needs the instant flip sets `'rollback'`.
+- **Changing the default changes behaviour for existing configs** that never set the option: their previous colour is now stopped after a flip. That is the intent, but it is a behaviour change and is recorded in the changelog.
+- **Workers are not rolled back.** There is a single worker set, restarted against the new release in `afterHealthy`; a rollback moves web traffic only, as the instant flip always did.
+- **The drain is a fixed wait,** not a check that connections have closed. It covers quick requests; a long-lived connection (WebSocket, streaming) can still be cut when the old colour stops.

src/cli/commands/rollback.ts

@@ -14,12 +14,15 @@ import {
   otherColor,
   portFor,
   coloredWebName,
+  releaseFor,
+  type DeployColor,
 } from '../../domain/deploy/blue-green.js';
+import { reapColourCommand } from '../../domain/deploy/retention.js';
 import { rollFleet, type FleetEvent } from '../../domain/deploy/fleet.js';
 import { isFleet } from '../../domain/servers.js';
 import type { RemoteExecutor } from '../../domain/remote/executor.js';
-import type { ShipnodeConfig, ShipnodeApp } from '../../shared/types.js';
-import { isActiveCommand, isWatt, resolveWattUnits, restartUnitCommand, wattUnitName } from '../../domain/runtime/watt.js';
+import type { ShipnodeConfig, ShipnodeApp, Pm2App } from '../../shared/types.js';
+import { enableAndStartUnitCommand, isActiveCommand, isWatt, resolveWattUnits, restartUnitCommand, wattUnitName } from '../../domain/runtime/watt.js';
 import { configForAppResult, configForServer, getServerTargets } from '../../domain/servers.js';
 
 /**
@@ -224,10 +227,17 @@ async function rollbackReplica(
   ui.success(`Rolled back to ${target.timestamp}`);
 }
 
+const MISE = 'export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"';
+
 /**
- * Instant blue-green rollback: the previous colour is still resident, so we
- * flip Caddy's upstream back to it and swap the persisted active colour. No
- * process restart, no dropped requests.
+ * Blue-green rollback: send traffic back to the previous colour.
+ *
+ * With retention `rollback` that colour is still running, so this is an instant
+ * flip. With `warm` (the default) it was stopped after the last flip, so it is
+ * booted again from the release it ran — `current` is pointed back at that
+ * release first, because the colour's launcher files resolve through `current` —
+ * health-checked, and only then does traffic move. The colour that was serving
+ * is stopped afterwards, so memory stays at one copy.
  *
  * Only one step back is possible — older colours were reaped by later deploys.
  * For anything deeper, redeploy the desired release instead.
@@ -242,14 +252,14 @@ async function rollbackBlueGreen(
 ): Promise<void> {
   if (app.blueGreenRetention === 'none') {
     throw new Error(
-      'Instant blue-green rollback is disabled because blueGreenRetention is "none". ' +
-      'Redeploy the desired release instead.',
+      'Blue-green rollback is disabled because blueGreenRetention is "none". ' +
+      'Use "warm" to keep rollback without holding the old colour in memory, or redeploy the desired release.',
     );
   }
 
   if (stepsBack !== 1) {
     throw new Error(
-      `Blue-green rollback only supports one step (the live previous colour). ` +
+      `Blue-green rollback only supports one step (the previous colour). ` +
       `To go further back, redeploy the desired release.`,
     );
   }
@@ -268,43 +278,140 @@ async function rollbackBlueGreen(
   const previousPort = portFor(previous, state);
   const previousName = coloredWebName(namespace, webApp.name, previous);
 
-  // The previous colour must still be online to serve traffic after the flip.
-  // Parse pm2's JSON in-process rather than relying on `node` being on the
-  // remote PATH at rollback time.
-  let online = false;
-  if (isWatt(app)) {
-    const unit = wattUnitName(namespace, webApp.name, previous);
-    online = (await executor.exec(isActiveCommand(unit))).exitCode === 0;
-  } else {
-    const mise = `export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"`;
-    const jlist = await executor.exec(`${mise} && mise exec -- pm2 jlist`);
-    try {
-      const entries = JSON.parse(jlist.stdout.trim()) as Array<{ name: string; pm2_env?: { status?: string } }>;
-      online = entries.some((e) => e.name === previousName && e.pm2_env?.status === 'online');
-    } catch {
-      online = false;
-    }
-  }
-  if (!online) {
-    throw new Error(
-      `Previous colour "${previousName}" is not running — cannot instant-rollback. ` +
-      `Redeploy the desired release instead.`,
-    );
-  }
+  const online = await isColourOnline(executor, app, namespace, webApp.name, previous);
 
   ui.warn(`Active colour:   ${state.activeColor} (port ${portFor(state.activeColor, state)})`);
   ui.warn(`Rollback target: ${previous} (port ${previousPort})`);
 
-  const ok = await ask('Flip traffic back to the previous colour?');
-  if (!ok) {
-    ui.info('Rollback cancelled.');
-    return;
+  let bootFrom: string | undefined;
+  if (online) {
+    if (!(await ask('Flip traffic back to the previous colour?'))) {
+      ui.info('Rollback cancelled.');
+      return;
+    }
+  } else {
+    const release = releaseFor(state, previous);
+    if (release === undefined) {
+      throw new Error(
+        `Previous colour "${previousName}" is stopped and the server did not record which release it ran ` +
+        `(its state predates warm rollback). Redeploy the desired release instead.`,
+      );
+    }
+    const onDisk = await executor.exec(`test -d "${appPath}/releases/${release}"`);
+    if (onDisk.exitCode !== 0) {
+      throw new Error(
+        `Release ${release} is no longer on the server (older releases are cleaned up after keepReleases). ` +
+        `Redeploy the desired release instead.`,
+      );
+    }
+    ui.warn(`"${previousName}" is stopped; it will be started again from release ${release}.`);
+    if (!(await ask(`Start ${previous} from release ${release} and flip traffic to it?`))) {
+      ui.info('Rollback cancelled.');
+      return;
+    }
+    bootFrom = release;
+  }
+
+  if (bootFrom !== undefined) {
+    await bootColourFromRelease({
+      executor, config, app, appPath, namespace, webApp, color: previous, port: previousPort, release: bootFrom,
+    });
   }
 
   const caddy = new CaddyService(executor, config);
   await caddy.configureBackend(app, previousPort);
   await caddy.reload();
   await writeDeployState(executor, appPath, { ...state, activeColor: previous });
 
+  // The colour that was serving is no longer needed: keep one copy in memory.
+  if (app.blueGreenRetention === 'warm') {
+    await executor.execOrThrow(reapColourCommand(app, namespace, webApp.name, state.activeColor));
+  }
+
   ui.success(`Rolled back — traffic now on ${previous} (port ${previousPort})`);
 }
+
+/** Whether the web process of one colour is running right now. */
+async function isColourOnline(
+  executor: RemoteExecutor,
+  app: ShipnodeApp,
+  namespace: string,
+  webName: string,
+  color: DeployColor,
+): Promise<boolean> {
+  if (isWatt(app)) {
+    return (await executor.exec(isActiveCommand(wattUnitName(namespace, webName, color)))).exitCode === 0;
+  }
+  // Parse pm2's JSON in-process rather than relying on `node` being on the
+  // remote PATH at rollback time.
+  const name = coloredWebName(namespace, webName, color);
+  const jlist = await executor.exec(`${MISE} && mise exec -- pm2 jlist`);
+  try {
+    const entries = JSON.parse(jlist.stdout.trim()) as Array<{ name: string; pm2_env?: { status?: string } }>;
+    return entries.some((e) => e.name === name && e.pm2_env?.status === 'online');
+  } catch {
+    return false;
+  }
+}
+
+interface BootColourInput {
+  executor: RemoteExecutor;
+  config: ShipnodeConfig;
+  app: ShipnodeApp;
+  appPath: string;
+  namespace: string;
+  webApp: Pm2App;
+  color: DeployColor;
+  port: number;
+  release: string;
+}
+
+/**
+ * Start a stopped colour from the release it ran and wait until it is healthy.
+ *
+ * `current` is pointed at that release for the start and left there on success,
+ * since the colour's launcher resolves its files through `current`. On failure
+ * the colour is stopped again and `current` is put back, so a rollback that
+ * cannot start leaves the app exactly as it was.
+ */
+async function bootColourFromRelease(input: BootColourInput): Promise<void> {
+  const { executor, config, app, appPath, namespace, webApp, color, port, release } = input;
+  const releases = new ReleaseManager(executor, appPath, app.keepReleases);
+  const before = (await executor.exec(`readlink "${appPath}/current"`)).stdout.trim();
+
+  await releases.switchSymlink(`${appPath}/releases/${release}`);
+  try {
+    if (isWatt(app)) {
+      await executor.execOrThrow(enableAndStartUnitCommand(wattUnitName(namespace, webApp.name, color)));
+    } else {
+      await executor.execOrThrow(
+        `cd "${appPath}/current" && ${MISE} && ` +
+        `mise exec -- pm2 start "${appPath}/current/ecosystem.web.config.cjs" --update-env && ` +
+        `mise exec -- pm2 save`,
+      );
+    }
+    if (app.healthCheck.enabled) {
+      ui.info(`Waiting for ${color} to pass its health check...`);
+      await new HealthCheckService(executor, config).perform(app, {
+        httpPort: port,
+        pm2Apps: [webApp],
+        resolvePm2Name: (a: Pm2App) => coloredWebName(namespace, a.name, color),
+      });
+    }
+  } catch (error) {
+    await executor.exec(reapColourCommand(app, namespace, webApp.name, color));
+    if (before !== '') {
+      try {
+        await releases.switchSymlink(before);
+      } catch {
+        // Don't let a failed restore hide why the start failed: say where
+        // `current` is left and how to put it back, then rethrow the original.
+        ui.warn(
+          `Could not point current back at its previous release. It still points at ${appPath}/releases/${release}; ` +
+          `restore it with: ln -sfn "${before}" "${appPath}/current"`,
+        );
+      }
+    }
+    throw error;
+  }
+}

src/config/builder.ts

@@ -182,7 +182,11 @@ export class ShipnodeBuilder {
     return this;
   }
 
-  /** Reclaim the inactive web process after a successful blue-green switch. */
+  /**
+   * What happens to the old colour after a successful blue-green switch: `warm`
+   * (default) stops it but keeps its release so `rollback` can start it again,
+   * `rollback` keeps it running for an instant flip, `none` stops it with no rollback.
+   */
   blueGreenRetention(retention: ShipnodeApp['blueGreenRetention']): this {
     this.config.blueGreenRetention = retention;
     return this;
@@ -443,7 +447,11 @@ export class ShipnodeAppBuilder {
     return this;
   }
 
-  /** Reclaim the inactive web process after a successful blue-green switch. */
+  /**
+   * What happens to the old colour after a successful blue-green switch: `warm`
+   * (default) stops it but keeps its release so `rollback` can start it again,
+   * `rollback` keeps it running for an instant flip, `none` stops it with no rollback.
+   */
   blueGreenRetention(retention: ShipnodeApp['blueGreenRetention']): this {
     this.state.blueGreenRetention = retention;
     return this;

src/config/schema.ts

@@ -276,7 +276,7 @@ export const ShipnodeAppSchema = z.object({
   envFile: z.string().default('.env'),
   keepReleases: z.number().int().min(1).default(5),
   zeroDowntime: z.boolean().optional(),
-  blueGreenRetention: z.enum(['rollback', 'none']).default('rollback'),
+  blueGreenRetention: z.enum(['warm', 'rollback', 'none']).default('warm'),
   altPort: z.number().int().positive().optional(),
   sharedDirs: z.array(z.string()).optional(),
   sharedFiles: z.array(z.string()).optional(),