Skip to content

fix: migrate prod db off neon free-tier to railway postgres + resilience hardening - #104

Merged
distroinfinity merged 6 commits into
mainfrom
fix/prod-db-railway-migration
Jun 23, 2026
Merged

distroinfinity merged 6 commits into
mainfrom
fix/prod-db-railway-migration

Conversation

@distroinfinity

Copy link
Copy Markdown
Owner

Why

All CLI users saw Distro TV — offline demo (backend not reachable) instead of channel content. Root cause: the prod Postgres (Neon free-tier) exhausted its account compute-time quota (HTTP 402) → compute force-suspended (Jun 19) + branch archived (Jun 20) → API DB connections timed out → /me/content/next 500'd → CLI fell back to the demo fixture. Contributing factors: the uncached /health SELECT 1 kept the Neon compute pinned active 24/7 (burning the quota), and no alerting meant the outage was silent for ~4 days.

What

Prod Postgres moved to Railway Postgres (co-located in the API's Railway project, private networking) — already restored out-of-band (Phase 0). This PR is the supporting code + docs (Phase 1):

  • railway DB target — env.dbTarget accepts local | neon | railway; the dev-safety guard assertEnvSafe now refuses any non-local target in dev (was neon-only). neon/local keep working.
  • dbUrlForTarget + pool options — extracted, testable URL resolver; explicit postgres() options (max: 10, connect_timeout: 10, idle_timeout: 30) so idle connections close.
  • migrate runner — resolves any remote target and guards its direct-run so test imports don't trigger a live migration.
  • cached probeDb — 10s success cache mirroring probeRedis, killing the per-poll SELECT 1 storm (only success is cached; failures re-probe and surface fast).
  • docs — gitbook + CLAUDE.md now say Railway Postgres; incident recorded; alerting follow-up tracked.

Spec + plan: docs/superpowers/specs|plans/2026-06-23-prod-db-railway-migration-*.md

Validation

  • 40/40 API unit tests, typecheck, build all green.
  • New migrate.js with DB_TARGET=railway run against the real Railway Postgres → [migrate] up to date (target=remote).
  • Branch server booted with DB_TARGET=railway → /health 200 ok, and /me/content/next served real news + markets (no demo fixture).
  • Live prod user flow (device bearer → /me/content/next) returns 5 news + 5 ticker, real headlines + live quotes.

After merge (manual, gated on this deploying first)

Flip Railway DB_TARGET=railway, then verify /health is ok and deploy logs show [migrate] up to date (target=remote). neon still resolves identically, so the deploy is safe before the flip.

Follow-ups (non-blocking)

  • /health degraded alerting (the actual "never silent again" fix).
  • Align drizzle.config.ts remote fallback to DATABASE_URL_UNPOOLED ?? DATABASE_URL (pre-existing, off the prod path).

@vercel

vercel Bot commented Jun 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
devdrip Ready Ready Preview, Comment Jun 23, 2026 12:25pm

@distroinfinity
distroinfinity merged commit 7909d45 into main Jun 23, 2026
5 checks passed
@distroinfinity
distroinfinity deleted the fix/prod-db-railway-migration branch June 23, 2026 12:26

This branch was successfully deployed

1 active deployment
Preview — db5f8f68 Deployed Jun 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant