diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b334fa..c46ddc0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,8 @@ Until 1.0.0, releases stay on 0.x: `Added` bumps minor, `Fixed`/`Changed` (and t ### Fixed - `/spacestate`: the Sentry context for a failed call now says why the network failed (`errorCause`, such as `ENOTFOUND`, `ECONNREFUSED` or `UNEXPECTED_REDIRECT`), and shows the address without its query. Sentry was replacing the old redacted address with `[Filtered]`. +- Home Assistant: a call cut off by a dropped connection now says it can't reach Home Assistant, instead of being logged as an unexpected failure and sent to Sentry as "Object captured as exception". +- Home Assistant: if Pixel can't reach Home Assistant at startup, it now says so within about 15 seconds (`home.connect_failed` in the logs, once to Sentry) and keeps trying in the background, waiting longer each time up to a minute. Before, it retried every second forever without a word, and the login and devices checks never ran. When it does connect, it logs `home.ready` and runs those checks. - `/admin level set` and capability changes no longer write `members.yaml` when the file was hand-edited into a state that fails the startup cross-checks (an id in two entries, or an admin with no entry). Before, the file was saved but the in-memory view and the `access.changed` audit record were not, and the admin got a generic error. Now the change is refused with "The members file is invalid", and nothing is written. - `/schedule`: a schedule deleted, paused or edited while another one due at the same time was posting no longer goes out as it was. Pixel reads each schedule again right before posting it. - `/schedule`: a message or poll that `{{tokens}}` would push past Discord's length limits (2000 characters for a message, 300 for a question, 55 for an answer) is now refused when it's created or edited, instead of failing at every occurrence. One that still comes out too long when it's posted is cut to fit and logged as `schedule.clipped`. diff --git a/docs/home-assistant.md b/docs/home-assistant.md index e5fd974..5ca12a1 100644 --- a/docs/home-assistant.md +++ b/docs/home-assistant.md @@ -153,7 +153,7 @@ Each row is a threat and what Pixel actually does about it. Identification of Di | **Compromised Discord account** of someone without HA capabilities | They can still run every command their tier allows (`/ha list` and `/ha status` on devices at or below their floor). They cannot change a device. | Same as any other takeover: Discord moderation, then `/admin level set` / revoke. `/ha` refusals say `HOME_DENIED` and do not confirm that a device exists. | | **Someone who can edit `devices.yaml` or `members.yaml`** | Widen the allow-list (add a door, add `unlock`) or grant `ha-doors` / `ha-admin`. | Files live on a protected volume (Azure Files + snapshots) or a laptop you control. `admins.yaml` is not writable from Discord. Capability grants are audited and the person is DMed. | | **Compromised Home Assistant admin account, or a wide-open remote-access route** (Nabu Casa, exposed port, leaked sidecar) | Full control of the building, bypassing Pixel. | Pixel cannot fix a broken HA. Use a strong unique password and MFA on HA admins; do not expose HA to the internet except via Nabu Casa or Tailscale; keep the Pixel user non-admin so a *Pixel* leak is not an *HA admin* leak. Rotate HA passwords and the Pixel token. | -| **Home Assistant down or slow** | `/ha` says it cannot reach HA. Discord, SpaceAPI and the rest keep working. | Connect in the background, reconnect forever, fail at once, no queue, no retry, time limit on every call. HA is not on `/healthz`. After a refused token the connection stays off until `/admin reload`. A hung call cannot hold a command forever. | +| **Home Assistant down or slow** | `/ha` says it cannot reach HA. Discord, SpaceAPI and the rest keep working. | Connect in the background, reconnect forever, fail at once, no queue, no retry, time limit on every call. HA is not on `/healthz`. If HA can't be reached at startup, Pixel says so once (`home.connect_failed`, Sentry) and keeps trying, waiting up to a minute between attempts. After a refused token the connection stays off until `/admin reload`. A hung call cannot hold a command forever. | | **Typo or hostile text from Home Assistant** (friendly names, states, attributes) | Could try to inject mentions or formatting into Discord. | Device names come from `devices.yaml`, not HA. States and attributes are shown as code spans, control characters stripped, length limited. HA's own error text is never shown. | | **Two Pixels, or `dev` pointed at prod HA** | A test command unlocks the real door. | Separate bots, tokens and (when hosted) vaults. `dev` uses a test HA or none. Never put the prod URL and token in the Pixel Dev app. | diff --git a/docs/runbook.md b/docs/runbook.md index 8aec9d8..090d209 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -560,7 +560,7 @@ Discord and the rest of Pixel keep working. Check the instance is up, the token **Home Assistant is unreachable** (`/admin status` shows not connected or off; `/ha` says it can't reach it) 1. `/admin status` shows why. "Off: the token was refused" means the token was revoked or replaced: make a new one (see [Secrets](#secrets)). Not connected means it can't reach the HA URL: check HA is up and the network or tunnel between them. -2. Pixel keeps reconnecting by itself, never queues an action and never retries one, so nothing will fire late when it's back. +2. Pixel keeps reconnecting by itself, never queues an action and never retries one, so nothing will fire late when it's back. If it couldn't reach HA at startup, the logs say `home.connect_failed` once and it keeps trying (waiting up to a minute between attempts); `home.ready` means it got through. 3. After fixing it, `/admin reload` re-checks the connection. **Someone has the wrong access** diff --git a/src/adapters/home-assistant/backend.test.ts b/src/adapters/home-assistant/backend.test.ts index c4c5c50..5660acf 100644 --- a/src/adapters/home-assistant/backend.test.ts +++ b/src/adapters/home-assistant/backend.test.ts @@ -1,11 +1,13 @@ +import type { AddressInfo } from "node:net"; import { afterEach, describe, expect, it } from "vitest"; +import { WebSocketServer } from "ws"; import { HomeRequestError, HomeUnavailableError } from "../../core/home.ts"; import type { Logger } from "../../core/logger.ts"; import { type FakeHomeAssistant, startFakeHomeAssistant, } from "../../testing/fake-home-assistant.ts"; -import { HomeAssistantBackend, translateError } from "./backend.ts"; +import { HomeAssistantBackend, type HomeAssistantOptions, translateError } from "./backend.ts"; const TOKEN = "a-very-secret-long-lived-token-value"; @@ -29,8 +31,13 @@ async function fake(options: Parameters[0] = {}) open.push(server); return server; } -function backendFor(server: FakeHomeAssistant, token = TOKEN, url = server.url) { - const backend = new HomeAssistantBackend({ url, token, logger: logger() }); +function backendFor( + server: FakeHomeAssistant, + token = TOKEN, + url = server.url, + more: Partial = {}, +) { + const backend = new HomeAssistantBackend({ url, token, logger: logger(), ...more }); open.push({ stop: () => backend.close() }); return backend; } @@ -108,25 +115,101 @@ describe("connecting", () => { expect(await backend.check()).toMatchObject({ kind: "connected" }); }); - it("keeps trying while Home Assistant is unreachable, without ever blocking, and connects when it appears", async () => { + it("settles and says so when Home Assistant is unreachable at first, then keeps trying and connects when it appears", async () => { const server = await fake(); const { port } = server; await server.stop(); - const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`); - let settled = false; - void backend.settled.then(() => { - settled = true; + const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, { + retryMs: { first: 100, max: 200 }, }); - await new Promise((r) => setTimeout(r, 600)); - expect(settled).toBe(false); + let connected = false; + void backend.connected.then(() => { + connected = true; + }); + await backend.settled; expect(backend.status()).toEqual({ kind: "connecting" }); + const failed = logs.filter((l) => l.obj.event === "home.connect_failed"); + expect(failed).toEqual([ + expect.objectContaining({ + level: "warn", + obj: expect.objectContaining({ why: "connection error 1" }), + }), + ]); await expect(backend.getStates(["light.x"])).rejects.toBeInstanceOf(HomeUnavailableError); + // Still trying, quietly: one warning, not one per attempt. + await waitFor(() => logs.some((l) => l.obj.event === "home.connect_retry")); + expect(connected).toBe(false); + expect(logs.filter((l) => l.obj.event === "home.connect_failed")).toHaveLength(1); + await server.start(); - await backend.settled; - expect(backend.status().kind).toBe("connected"); + await backend.connected; + expect(backend.status()).toMatchObject({ kind: "connected", adminToken: false }); + expect(logs.some((l) => l.obj.event === "home.ready")).toBe(true); + }, 20_000); + + it("waits longer between each attempt, up to a limit", async () => { + const server = await fake(); + const { port } = server; + await server.stop(); + const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, { + retryMs: { first: 10, max: 40 }, + }); + await waitFor(() => logs.filter((l) => l.obj.event === "home.connect_retry").length >= 4); + backend.close(); + const waits = logs + .filter((l) => l.obj.event === "home.connect_failed" || l.obj.event === "home.connect_retry") + .map((l) => l.obj.retryInMs); + expect(waits.slice(0, 5)).toEqual([10, 20, 40, 40, 40]); }, 20_000); + it("gives up on an attempt that takes too long, such as a login that is never answered", async () => { + const silent = new WebSocketServer({ port: 0, path: "/api/websocket" }); + open.push({ + stop: () => + new Promise((resolve) => { + for (const client of silent.clients) client.terminate(); + silent.close(() => resolve()); + }), + }); + await new Promise((resolve) => silent.once("listening", () => resolve())); + const { port } = silent.address() as AddressInfo; + const backend = new HomeAssistantBackend({ + url: `http://127.0.0.1:${port}`, + token: TOKEN, + logger: logger(), + attemptTimeoutMs: 200, + retryMs: { first: 60_000, max: 60_000 }, + }); + open.push({ stop: () => backend.close() }); + await backend.settled; + expect(backend.status()).toEqual({ kind: "connecting" }); + expect(logs.find((l) => l.obj.event === "home.connect_failed")?.obj.why).toBe("ConnectTimeout"); + }); + + it("stops trying when closed", async () => { + const server = await fake(); + const { port } = server; + await server.stop(); + const backend = backendFor(server, TOKEN, `http://127.0.0.1:${port}`, { + retryMs: { first: 50, max: 50 }, + }); + await backend.settled; + backend.close(); + await server.start(); + await new Promise((r) => setTimeout(r, 300)); + expect(server.connectionCount()).toBe(0); + expect(backend.status()).toEqual({ kind: "connecting" }); + }); + + it("goes off, rather than retrying, when the address can't work", async () => { + const server = await fake(); + const backend = backendFor(server, TOKEN, "not a url"); + await backend.settled; + expect(backend.status()).toMatchObject({ kind: "off" }); + expect(logs.find((l) => l.obj.event === "home.connect_failed")?.level).toBe("error"); + }); + it("re-checking while connected asks again who the token belongs to", async () => { const server = await fake({ isAdmin: false }); const backend = backendFor(server); @@ -322,6 +405,21 @@ describe("when the connection drops", () => { expect((await backend.getStates(["light.a"])).size).toBe(1); }, 20_000); + it("fails a call that was in flight as unreachable, not as an unknown failure", async () => { + const server = await fake(); + server.hangService("lock.unlock"); + const backend = backendFor(server); + await backend.settled; + const call = backend + .callService({ domain: "lock", service: "unlock", entityId: "lock.front_door" }) + .catch((e: unknown) => e); + await waitFor(() => server.calls.length === 1); + server.dropConnections(); + const error = await call; + expect(error).toBeInstanceOf(HomeUnavailableError); + expect((error as Error).message).toBe("I can't reach Home Assistant right now."); + }, 20_000); + it("goes off if the token is refused when it comes back", async () => { const server = await fake({ token: "first-token-value" }); const backend = backendFor(server, "first-token-value"); @@ -378,6 +476,36 @@ describe("translateError", () => { expect((error as Error).message).not.toContain("explode"); }); + it("treats a call cut off by a dropped connection as unreachable", () => { + // What the library rejects calls in flight with when the socket closes. + const lost = { + type: "result", + success: false, + error: { code: 3, message: "Connection lost" }, + }; + const error = translateError(lost); + expect(error).toBeInstanceOf(HomeUnavailableError); + expect((error as Error).message).toBe("I can't reach Home Assistant right now."); + }); + + it("treats Home Assistant's own error inside a failed result as a refusal, keeping the code", () => { + const error = translateError({ + type: "result", + success: false, + error: { code: "unauthorized", message: "token abc123 can't do that" }, + }); + expect(error).toBeInstanceOf(HomeRequestError); + expect((error as HomeRequestError).code).toBe("unauthorized"); + expect((error as Error).message).not.toContain("abc123"); + }); + + it("lets a failed result it doesn't understand through as it is", () => { + const odd = { type: "result", success: false, error: "nope" }; + expect(translateError(odd)).toBe(odd); + const ok = { type: "result", success: true, error: { code: 3 } }; + expect(translateError(ok)).toBe(ok); + }); + it("lets a real bug through as it is", () => { const bug = new TypeError("oops"); expect(translateError(bug)).toBe(bug); diff --git a/src/adapters/home-assistant/backend.ts b/src/adapters/home-assistant/backend.ts index 75fc0a2..b0b29c1 100644 --- a/src/adapters/home-assistant/backend.ts +++ b/src/adapters/home-assistant/backend.ts @@ -29,9 +29,16 @@ import type { Logger } from "../../core/logger.ts"; * a slow or absent Home Assistant never holds up the bot, and every call is a * single attempt: if the connection is down the call fails at once. * + * Until the first connection, Pixel retries itself, waiting longer each time: the + * library's own endless retry says nothing and never gives up, so a Home Assistant + * that is down at startup would go unnoticed. Once connected, the library + * reconnects by itself after a drop. + * * The library throws numbers (connection errors) and plain `{ code, message }` - * objects (Home Assistant's own errors), not `Error`s. They are translated here - * into safe messages, and never include the token or the address. + * objects (Home Assistant's own errors), not `Error`s, and rejects calls in flight + * when the socket drops with a whole failed result (`{ type: "result", success: + * false, error: { code: 3, ... } }`). They are translated here into safe messages, + * and never include the token or the address. */ export type HomeAssistantOptions = { @@ -40,6 +47,10 @@ export type HomeAssistantOptions = { /** A long-lived access token. A secret: never logged or put in a message. */ token: string; logger: Logger; + /** How long one connection attempt may take, the login included. For tests. */ + attemptTimeoutMs?: number; + /** The wait before the first retry, doubled each time up to `max`. For tests. */ + retryMs?: { first: number; max: number }; }; type Phase = "connecting" | "connected" | "reconnecting" | "off"; @@ -70,6 +81,15 @@ type AreaNames = { /** Why the login can't work, in words that name no secret. */ const REFUSED = "Home Assistant refused the token, so it needs replacing"; +/** The library opens the socket within 10 s; this also bounds the login after it. */ +const DEFAULT_ATTEMPT_TIMEOUT_MS = 15_000; +const DEFAULT_RETRY_MS = { first: 1_000, max: 60_000 }; + +/** A connection attempt that took too long, such as a server that never answers the login. */ +class ConnectTimeout extends Error { + override name = "ConnectTimeout"; +} + export class HomeAssistantBackend implements HomeBackend { readonly #url: string; readonly #token: string; @@ -79,13 +99,30 @@ export class HomeAssistantBackend implements HomeBackend { #offReason = ""; #adminToken: boolean | undefined; #attempt: Promise | undefined; - /** Settles once the first attempt has either connected or given up. */ + readonly #attemptTimeoutMs: number; + readonly #retryMs: { first: number; max: number }; + /** Failed attempts since the last connection, which sets the next wait. */ + #failures = 0; + #retryTimer: ReturnType | undefined; + #closed = false; + #markConnected: () => void = () => {}; + /** + * Settles once the first attempt has connected, been refused or failed, within + * about `attemptTimeoutMs`. After a failure it keeps trying in the background. + */ readonly settled: Promise; + /** Settles the first time it connects. Never rejects, and stays pending if it never connects. */ + readonly connected: Promise; - constructor({ url, token, logger }: HomeAssistantOptions) { + constructor({ url, token, logger, attemptTimeoutMs, retryMs }: HomeAssistantOptions) { this.#url = url.replace(/\/+$/, ""); this.#token = token; this.#logger = logger.child({ component: "home-assistant" }); + this.#attemptTimeoutMs = attemptTimeoutMs ?? DEFAULT_ATTEMPT_TIMEOUT_MS; + this.#retryMs = retryMs ?? DEFAULT_RETRY_MS; + this.connected = new Promise((resolve) => { + this.#markConnected = resolve; + }); this.settled = this.#connect(); } @@ -201,6 +238,9 @@ export class HomeAssistantBackend implements HomeBackend { /** Stops reconnecting and closes the socket. For shutdown. */ close(): void { + this.#closed = true; + clearTimeout(this.#retryTimer); + this.#retryTimer = undefined; this.#connection?.close(); } @@ -215,11 +255,15 @@ export class HomeAssistantBackend implements HomeBackend { async #tryConnect(): Promise { this.#phase = "connecting"; try { - // With infinite retries this only ever rejects when the token is refused. - const connection = await createConnection({ - auth: createLongLivedTokenAuth(this.#url, this.#token), - setupRetry: -1, - }); + const connection = await this.#open(); + if (this.#closed) { + connection.close(); + return; + } + if (this.#failures > 0) { + this.#logger.info({ event: "home.ready" }, "connected to Home Assistant"); + } + this.#failures = 0; this.#connection?.close(); this.#connection = connection; connection.addEventListener("ready", () => { @@ -236,7 +280,13 @@ export class HomeAssistantBackend implements HomeBackend { }); this.#phase = "connected"; await this.#identify(); + this.#markConnected(); } catch (error) { + if (this.#closed) return; + if (isUnreachable(error)) { + this.#retryLater(error); + return; + } this.#logger.error( { event: "home.connect_failed", why: describeFailure(error) }, "couldn't connect to Home Assistant", @@ -245,6 +295,57 @@ export class HomeAssistantBackend implements HomeBackend { } } + /** + * One attempt, bounded in time: the library retries nothing here (`setupRetry: 0`) + * and only times out opening the socket, not the login after it. A connection that + * turns up after the time limit is closed. + */ + async #open(): Promise { + const pending = createConnection({ + auth: createLongLivedTokenAuth(this.#url, this.#token), + setupRetry: 0, + }); + let timer: ReturnType | undefined; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => reject(new ConnectTimeout()), this.#attemptTimeoutMs); + }); + try { + return await Promise.race([pending, timeout]); + } catch (error) { + if (error instanceof ConnectTimeout) { + pending.then( + (late) => late.close(), + () => {}, + ); + } + throw error; + } finally { + clearTimeout(timer); + } + } + + /** Home Assistant can't be reached yet: say so once, then try again, waiting longer each time. */ + #retryLater(error: unknown): void { + const waitMs = Math.min(this.#retryMs.first * 2 ** this.#failures, this.#retryMs.max); + if (this.#failures === 0) { + this.#logger.warn( + { event: "home.connect_failed", why: describeFailure(error), retryInMs: waitMs }, + "couldn't connect to Home Assistant, will keep trying", + ); + } else { + this.#logger.debug( + { event: "home.connect_retry", why: describeFailure(error), retryInMs: waitMs }, + "still can't connect to Home Assistant", + ); + } + this.#failures += 1; + this.#retryTimer = setTimeout(() => { + this.#retryTimer = undefined; + void this.#connect(); + }, waitMs); + this.#retryTimer.unref(); + } + /** Asks who the token belongs to, so an admin's token can be flagged. Failure isn't fatal. */ async #identify(): Promise { try { @@ -279,7 +380,7 @@ export class HomeAssistantBackend implements HomeBackend { /** Turns what the library throws into something safe, or lets a real bug through. */ #translate(error: unknown): unknown { const translated = translateError(error); - if (translated instanceof HomeUnavailableError && error === ERR_INVALID_AUTH) { + if (translated instanceof HomeUnavailableError && unwrap(error) === ERR_INVALID_AUTH) { this.#turnOff(REFUSED); } return translated; @@ -288,19 +389,41 @@ export class HomeAssistantBackend implements HomeBackend { /** * The library throws a number for a connection problem and `{ code, message }` for - * Home Assistant's own errors. Anything else is a bug, and is returned as it is. + * Home Assistant's own errors, either bare or wrapped in a failed result. Anything + * else is a bug, and is returned as it is. */ export function translateError(error: unknown): unknown { - if (typeof error === "number") return new HomeUnavailableError(HOME_MESSAGES.unreachable); - if (isHaError(error)) return new HomeRequestError(HOME_MESSAGES.refused, error.code); + const failure = unwrap(error); + if (typeof failure === "number") return new HomeUnavailableError(HOME_MESSAGES.unreachable); + if (isHaError(failure)) return new HomeRequestError(HOME_MESSAGES.refused, failure.code); return error; } +/** + * When the socket drops, the library rejects calls in flight with a whole failed + * result, `{ type: "result", success: false, error: { code: 3, message } }`. Its own + * codes are numbers (a connection problem); Home Assistant's are strings. This + * returns what is inside, so it reads like the bare forms. + */ +function unwrap(error: unknown): unknown { + if (!isRecord(error) || error.type !== "result" || error.success !== false) return error; + const inner = error.error; + if (isRecord(inner) && typeof inner.code === "number") return inner.code; + return inner; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null; +} + function isHaError(error: unknown): error is { code: string } { + return isRecord(error) && typeof error.code === "string"; +} + +/** Worth trying again: Home Assistant didn't answer, as opposed to refusing or a bad address. */ +function isUnreachable(error: unknown): boolean { return ( - typeof error === "object" && - error !== null && - typeof (error as { code?: unknown }).code === "string" + error === ERR_CANNOT_CONNECT || error === ERR_CONNECTION_LOST || error instanceof ConnectTimeout ); } @@ -322,8 +445,9 @@ function reasonFor(error: unknown): string { /** A short, safe description of a failure for logs: a code or a type, never a message that could hold a secret. */ function describeFailure(error: unknown): string { - if (typeof error === "number") return `connection error ${error}`; - if (isHaError(error)) return error.code; + const failure = unwrap(error); + if (typeof failure === "number") return `connection error ${failure}`; + if (isHaError(failure)) return failure.code; return error instanceof Error ? error.name : "unknown"; } diff --git a/src/index.ts b/src/index.ts index b00e5c9..cd75f3b 100644 --- a/src/index.ts +++ b/src/index.ts @@ -93,11 +93,21 @@ async function main(): Promise { let homeAssistant: HomeAssistantBackend | undefined; const homeAssistantSettled = () => homeAssistant?.settled ?? Promise.resolve(); if (config.homeAssistant) { - homeAssistant = new HomeAssistantBackend({ ...config.homeAssistant, logger }); - home.attach(homeAssistant); - homeAssistant.settled + const backend = new HomeAssistantBackend({ ...config.homeAssistant, logger }); + homeAssistant = backend; + home.attach(backend); + backend.settled .then(async () => { - await home.check(); + if ((await home.check()).kind === "connecting") { + // The first attempt couldn't reach it. It keeps trying in the background: say so + // once, then check the login and the devices when it does connect. + reporter.captureBackground( + new Error("Couldn't connect to Home Assistant at startup. Still trying."), + "home-assistant", + ); + await backend.connected; + await home.check(); + } // A typo in the devices file shows up as a warning, never a failure. const missing = await findMissingDevices(homeDevices.view, home); if (missing && missing.length > 0) { diff --git a/src/testing/fake-home-assistant.ts b/src/testing/fake-home-assistant.ts index 0f0af09..71ea7df 100644 --- a/src/testing/fake-home-assistant.ts +++ b/src/testing/fake-home-assistant.ts @@ -64,6 +64,8 @@ export type FakeHomeAssistant = { failIdentity(fail: boolean): void; /** Answer this service with an error (domain.service → HA error code). */ failService(key: string, code: string): void; + /** Never answer this service (domain.service), so the call is still in flight when the connection drops. */ + hangService(key: string): void; /** Cut every connection but keep serving, so the client has to reconnect. */ dropConnections(): void; connectionCount(): number; @@ -85,6 +87,7 @@ export async function startFakeHomeAssistant( let entityRegistryFails: string | undefined; let areaRegistryFails: string | undefined; const failing = new Map(); + const hanging = new Set(); const calls: FakeServiceCall[] = []; const sockets = new Set(); let wss: WebSocketServer | undefined; @@ -174,6 +177,7 @@ export async function startFakeHomeAssistant( target: msg.target, serviceData: msg.service_data, }); + if (hanging.has(`${msg.domain}.${msg.service}`)) return; const code = failing.get(`${msg.domain}.${msg.service}`); return code ? fail(code) : reply({ context: { id: "ctx" } }); } @@ -241,6 +245,9 @@ export async function startFakeHomeAssistant( failService: (key, code) => { failing.set(key, code); }, + hangService: (key) => { + hanging.add(key); + }, dropConnections, connectionCount: () => sockets.size, stop,