Fix Home Assistant connection errors: dropped calls and an unreachable HA at startup

CHANGELOG.md

@@ -12,6 +12,8 @@ Until 1.0.0, releases stay on 0.x: `Added` bumps minor, `Fixed`/`Changed` (and t
 ### Fixed
 
 - `/spacestate`: the Sentry context for a failed call now says why the network failed (`errorCause`, such as `ENOTFOUND`, `ECONNREFUSED` or `UNEXPECTED_REDIRECT`), and shows the address without its query. Sentry was replacing the old redacted address with `[Filtered]`.
+- Home Assistant: a call cut off by a dropped connection now says it can't reach Home Assistant, instead of being logged as an unexpected failure and sent to Sentry as "Object captured as exception".
+- Home Assistant: if Pixel can't reach Home Assistant at startup, it now says so within about 15 seconds (`home.connect_failed` in the logs, once to Sentry) and keeps trying in the background, waiting longer each time up to a minute. Before, it retried every second forever without a word, and the login and devices checks never ran. When it does connect, it logs `home.ready` and runs those checks.
 - `/admin level set` and capability changes no longer write `members.yaml` when the file was hand-edited into a state that fails the startup cross-checks (an id in two entries, or an admin with no entry). Before, the file was saved but the in-memory view and the `access.changed` audit record were not, and the admin got a generic error. Now the change is refused with "The members file is invalid", and nothing is written.
 - `/schedule`: a schedule deleted, paused or edited while another one due at the same time was posting no longer goes out as it was. Pixel reads each schedule again right before posting it.
 - `/schedule`: a message or poll that `{{tokens}}` would push past Discord's length limits (2000 characters for a message, 300 for a question, 55 for an answer) is now refused when it's created or edited, instead of failing at every occurrence. One that still comes out too long when it's posted is cut to fit and logged as `schedule.clipped`.

docs/home-assistant.md

@@ -153,7 +153,7 @@ Each row is a threat and what Pixel actually does about it. Identification of Di
 | **Compromised Discord account** of someone without HA capabilities | They can still run every command their tier allows (`/ha list` and `/ha status` on devices at or below their floor). They cannot change a device. | Same as any other takeover: Discord moderation, then `/admin level set` / revoke. `/ha` refusals say `HOME_DENIED` and do not confirm that a device exists. |
 | **Someone who can edit `devices.yaml` or `members.yaml`** | Widen the allow-list (add a door, add `unlock`) or grant `ha-doors` / `ha-admin`. | Files live on a protected volume (Azure Files + snapshots) or a laptop you control. `admins.yaml` is not writable from Discord. Capability grants are audited and the person is DMed. |
 | **Compromised Home Assistant admin account, or a wide-open remote-access route** (Nabu Casa, exposed port, leaked sidecar) | Full control of the building, bypassing Pixel. | Pixel cannot fix a broken HA. Use a strong unique password and MFA on HA admins; do not expose HA to the internet except via Nabu Casa or Tailscale; keep the Pixel user non-admin so a *Pixel* leak is not an *HA admin* leak. Rotate HA passwords and the Pixel token. |
-| **Home Assistant down or slow** | `/ha` says it cannot reach HA. Discord, SpaceAPI and the rest keep working. | Connect in the background, reconnect forever, fail at once, no queue, no retry, time limit on every call. HA is not on `/healthz`. After a refused token the connection stays off until `/admin reload`. A hung call cannot hold a command forever. |
+| **Home Assistant down or slow** | `/ha` says it cannot reach HA. Discord, SpaceAPI and the rest keep working. | Connect in the background, reconnect forever, fail at once, no queue, no retry, time limit on every call. HA is not on `/healthz`. If HA can't be reached at startup, Pixel says so once (`home.connect_failed`, Sentry) and keeps trying, waiting up to a minute between attempts. After a refused token the connection stays off until `/admin reload`. A hung call cannot hold a command forever. |
 | **Typo or hostile text from Home Assistant** (friendly names, states, attributes) | Could try to inject mentions or formatting into Discord. | Device names come from `devices.yaml`, not HA. States and attributes are shown as code spans, control characters stripped, length limited. HA's own error text is never shown. |
 | **Two Pixels, or `dev` pointed at prod HA** | A test command unlocks the real door. | Separate bots, tokens and (when hosted) vaults. `dev` uses a test HA or none. Never put the prod URL and token in the Pixel Dev app. |
 

docs/runbook.md

@@ -560,7 +560,7 @@ Discord and the rest of Pixel keep working. Check the instance is up, the token
 **Home Assistant is unreachable** (`/admin status` shows not connected or off; `/ha` says it can't reach it)
 
 1. `/admin status` shows why. "Off: the token was refused" means the token was revoked or replaced: make a new one (see [Secrets](#secrets)). Not connected means it can't reach the HA URL: check HA is up and the network or tunnel between them.
-2. Pixel keeps reconnecting by itself, never queues an action and never retries one, so nothing will fire late when it's back.
+2. Pixel keeps reconnecting by itself, never queues an action and never retries one, so nothing will fire late when it's back. If it couldn't reach HA at startup, the logs say `home.connect_failed` once and it keeps trying (waiting up to a minute between attempts); `home.ready` means it got through.
 3. After fixing it, `/admin reload` re-checks the connection.
 
 **Someone has the wrong access**

src/adapters/home-assistant/backend.test.ts

@@ -1,11 +1,13 @@
+import type { AddressInfo } from "node:net";
 import { afterEach, describe, expect, it } from "vitest";
+import { WebSocketServer } from "ws";
 import { HomeRequestError, HomeUnavailableError } from "../../core/home.ts";
 import type { Logger } from "../../core/logger.ts";
 import {
 	type FakeHomeAssistant,
 	startFakeHomeAssistant,
 } from "../../testing/fake-home-assistant.ts";
-import { HomeAssistantBackend, translateError } from "./backend.ts";
+import { HomeAssistantBackend, type HomeAssistantOptions, translateError } from "./backend.ts";
 
 const TOKEN = "a-very-secret-long-lived-token-value";
 
@@ -29,8 +31,13 @@ async function fake(options: Parameters<typeof startFakeHomeAssistant>[0] = {})
 	open.push(server);
 	return server;
 }
-function backendFor(server: FakeHomeAssistant, token = TOKEN, url = server.url) {
-	const backend = new HomeAssistantBackend({ url, token, logger: logger() });
+function backendFor(
+	server: FakeHomeAssistant,
+	token = TOKEN,
+	url = server.url,
+	more: Partial<HomeAssistantOptions> = {},
+) {
+	const backend = new HomeAssistantBackend({ url, token, logger: logger(), ...more });
 	open.push({ stop: () => backend.close() });
 	return backend;
 }
@@ -108,25 +115,101 @@ describe("connecting", () => {
 		expect(await backend.check()).toMatchObject({ kind: "connected" });
 	});
 
-	it("keeps trying while Home Assistant is unreachable, without ever blocking, and connects when it appears", async () => {
+	it("settles and says so when Home Assistant is unreachable at first, then keeps trying and connects when it appears", async () => {
 		const server = await fake();
 		const { port } = server;
 		await server.stop();
-		const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`);
-		let settled = false;
-		void backend.settled.then(() => {
-			settled = true;
+		const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, {
+			retryMs: { first: 100, max: 200 },
 		});
-		await new Promise((r) => setTimeout(r, 600));
-		expect(settled).toBe(false);
+		let connected = false;
+		void backend.connected.then(() => {
+			connected = true;
+		});
+		await backend.settled;
 		expect(backend.status()).toEqual({ kind: "connecting" });
+		const failed = logs.filter((l) => l.obj.event === "home.connect_failed");
+		expect(failed).toEqual([
+			expect.objectContaining({
+				level: "warn",
+				obj: expect.objectContaining({ why: "connection error 1" }),
+			}),
+		]);
 		await expect(backend.getStates(["light.x"])).rejects.toBeInstanceOf(HomeUnavailableError);
 
+		// Still trying, quietly: one warning, not one per attempt.
+		await waitFor(() => logs.some((l) => l.obj.event === "home.connect_retry"));
+		expect(connected).toBe(false);
+		expect(logs.filter((l) => l.obj.event === "home.connect_failed")).toHaveLength(1);
+
 		await server.start();
-		await backend.settled;
-		expect(backend.status().kind).toBe("connected");
+		await backend.connected;
+		expect(backend.status()).toMatchObject({ kind: "connected", adminToken: false });
+		expect(logs.some((l) => l.obj.event === "home.ready")).toBe(true);
+	}, 20_000);
+
+	it("waits longer between each attempt, up to a limit", async () => {
+		const server = await fake();
+		const { port } = server;
+		await server.stop();
+		const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, {
+			retryMs: { first: 10, max: 40 },
+		});
+		await waitFor(() => logs.filter((l) => l.obj.event === "home.connect_retry").length >= 4);
+		backend.close();
+		const waits = logs
+			.filter((l) => l.obj.event === "home.connect_failed" || l.obj.event === "home.connect_retry")
+			.map((l) => l.obj.retryInMs);
+		expect(waits.slice(0, 5)).toEqual([10, 20, 40, 40, 40]);
 	}, 20_000);
 
+	it("gives up on an attempt that takes too long, such as a login that is never answered", async () => {
+		const silent = new WebSocketServer({ port: 0, path: "/api/websocket" });
+		open.push({
+			stop: () =>
+				new Promise<void>((resolve) => {
+					for (const client of silent.clients) client.terminate();
+					silent.close(() => resolve());
+				}),
+		});
+		await new Promise<void>((resolve) => silent.once("listening", () => resolve()));
+		const { port } = silent.address() as AddressInfo;
+		const backend = new HomeAssistantBackend({
+			url: `http://127.0.0.1:${port}`,
+			token: TOKEN,
+			logger: logger(),
+			attemptTimeoutMs: 200,
+			retryMs: { first: 60_000, max: 60_000 },
+		});
+		open.push({ stop: () => backend.close() });
+		await backend.settled;
+		expect(backend.status()).toEqual({ kind: "connecting" });
+		expect(logs.find((l) => l.obj.event === "home.connect_failed")?.obj.why).toBe("ConnectTimeout");
+	});
+
+	it("stops trying when closed", async () => {
+		const server = await fake();
+		const { port } = server;
+		await server.stop();
+		const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, {
+			retryMs: { first: 50, max: 50 },
+		});
+		await backend.settled;
+		backend.close();
+		await server.start();
+		await new Promise((r) => setTimeout(r, 300));
+		expect(server.connectionCount()).toBe(0);
+		expect(backend.status()).toEqual({ kind: "connecting" });
+	});
+
+	it("goes off, rather than retrying, when the address can't work", async () => {
+		const server = await fake();
+		const backend = backendFor(server, TOKEN, "not a url");
+		await backend.settled;
+		expect(backend.status()).toMatchObject({ kind: "off" });
+		expect(logs.find((l) => l.obj.event === "home.connect_failed")?.level).toBe("error");
+	});
+
 	it("re-checking while connected asks again who the token belongs to", async () => {
 		const server = await fake({ isAdmin: false });
 		const backend = backendFor(server);
@@ -322,6 +405,21 @@ describe("when the connection drops", () => {
 		expect((await backend.getStates(["light.a"])).size).toBe(1);
 	}, 20_000);
 
+	it("fails a call that was in flight as unreachable, not as an unknown failure", async () => {
+		const server = await fake();
+		server.hangService("lock.unlock");
+		const backend = backendFor(server);
+		await backend.settled;
+		const call = backend
+			.callService({ domain: "lock", service: "unlock", entityId: "lock.front_door" })
+			.catch((e: unknown) => e);
+		await waitFor(() => server.calls.length === 1);
+		server.dropConnections();
+		const error = await call;
+		expect(error).toBeInstanceOf(HomeUnavailableError);
+		expect((error as Error).message).toBe("I can't reach Home Assistant right now.");
+	}, 20_000);
+
 	it("goes off if the token is refused when it comes back", async () => {
 		const server = await fake({ token: "first-token-value" });
 		const backend = backendFor(server, "first-token-value");
@@ -378,6 +476,36 @@ describe("translateError", () => {
 		expect((error as Error).message).not.toContain("explode");
 	});
 
+	it("treats a call cut off by a dropped connection as unreachable", () => {
+		// What the library rejects calls in flight with when the socket closes.
+		const lost = {
+			type: "result",
+			success: false,
+			error: { code: 3, message: "Connection lost" },
+		};
+		const error = translateError(lost);
+		expect(error).toBeInstanceOf(HomeUnavailableError);
+		expect((error as Error).message).toBe("I can't reach Home Assistant right now.");
+	});
+
+	it("treats Home Assistant's own error inside a failed result as a refusal, keeping the code", () => {
+		const error = translateError({
+			type: "result",
+			success: false,
+			error: { code: "unauthorized", message: "token abc123 can't do that" },
+		});
+		expect(error).toBeInstanceOf(HomeRequestError);
+		expect((error as HomeRequestError).code).toBe("unauthorized");
+		expect((error as Error).message).not.toContain("abc123");
+	});
+
+	it("lets a failed result it doesn't understand through as it is", () => {
+		const odd = { type: "result", success: false, error: "nope" };
+		expect(translateError(odd)).toBe(odd);
+		const ok = { type: "result", success: true, error: { code: 3 } };
+		expect(translateError(ok)).toBe(ok);
+	});
+
 	it("lets a real bug through as it is", () => {
 		const bug = new TypeError("oops");
 		expect(translateError(bug)).toBe(bug);