fix(health): limit each dependency check to 3 seconds and log the failed check

[posthog-eu]

Problem

• Operators cannot see why production failed its health check. On Oct 15–16 /api/health returned 503 for almost 3 hours, but no log names the failed dependency.
• The route waited on Postgres and object storage with no time limit. Neither the pg pool nor the S3 client sets a timeout.
• The burst shows two failure types:

Failure type	Approx. duration	Client disconnected
Fast 503 (one each Docker interval)	0.1–0.3 s	no
Hung 503	60 s, 120 s or 180 s	yes

• A hung check stays open for minutes after Docker stops at its 5-second timeout. The server must keep each hung request and its socket open.
• The route caught each dependency error and discarded it. Because of this, the logs cannot show if Postgres or S3 failed.

Origin

• Scout: signals-scout-general
• First signal: 2026-10-16
• Inbox report: open
• Task started by: auto-start, after the report was rated P2 and ready to fix

Changes

• Each dependency check now has a 3-second limit (runHealthCheck in src/server/health.ts). The limit is shorter than the 5-second Docker/Coolify healthcheck timeout.
• The two checks run in parallel, so the response comes back in approximately 3 seconds or less.
• When the limit expires, the route aborts the S3 HeadBucket request with an AbortSignal.
• The route logs a warn record Health check failed with check (database or objectStore) and the redacted error. This record goes to PostHog logs.
• The response body and status codes do not change. production-smoke.spec.ts still matches.

Note

This change does not repair the external cause of the burst. It makes the next burst short and identifies the failed dependency. Requests other than health checks did not occur in the window, so the logs do not show user impact.

Checks

• vitest --project unit src/server: new health.test.ts passes (ok, fast failure, hang plus abort)
• bun run typecheck, bun run lint, oxfmt --check on the changed files
• 20 PDF/raster/export unit tests fail in the sandbox, with and without this change. They need the ICC profile from bun run setup:icc, and the sandbox cannot get it.
• Not run: integration tests and e2e. They need Docker services.

Agent context

• The hung durations are near exact multiples of 60 s, each plus one fast failure. I could not find a 60 s timeout in the app, so the source is possibly the network or the storage provider. The new log record will confirm this.
• I did not change the 503 contract. A container marked unhealthy can cause Coolify/Traefik to stop routing to it, so an S3 outage could make the whole site unavailable. Decide if object storage must stay in the health check that Docker uses.

---

Created with PostHog Desktop from this inbox report.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 254f6cf1-7e21-47c4-b6ea-7a266bac46ca

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}